Aula 5 - Selenium e BeautifulSoup

12 Mar, 2022

Webdrivers:

Chromedriver:
- https://chromedriver.chromium.org/downloads
Geckodriver:
- https://github.com/mozilla/geckodriver/releases
IEDriver:
- https://www.selenium.dev/downloads/
  - x64: https://github.com/SeleniumHQ/selenium/releases/download/selenium-3.150.0/IEDriverServer_x64_3.150.2.zip
  - x32: https://github.com/SeleniumHQ/selenium/releases/download/selenium-3.150.0/IEDriverServer_Win32_3.150.2.zip
Operadriver:
- https://github.com/operasoftware/operachromiumdriver/releases
Edgedriver:
- https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/

Selenium:

Instalação do Selenium:
```
pip install selenium
```
Importação da biblioteca:
```
from selenium import webdriver
```

Chamar navegadores com o(s) webdriver(s):

#Google Chrome
driver = webdriver.Chrome('chromedriver.exe')

#Mozilla Firefox
driver = webdriver.Firefox('geckodriver.exe')

#Internet Explorer
driver = webdriver.Ie('IEDriverServer.exe')

Maximizar a janela do navegador:
```
driver.maximize_window()
```
Colocar um tempo para carregamento - IMPLÍCITO:
```
driver.implicitly_wait(30)
```

Colocar um tempo para carregamento - EXPLÍCITO:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.ID, 'twotabsearchtextbox')))

Acessar um site:

driver.get('https://www.amazon.com.br/')

Buscar elementos na página:

#Buscar pela class do elemento
driver.find_element_by_class_name('nav-input.nav-progressive-attribute')

#Buscar pelo id do elemento
driver.find_element_by_id('twotabsearchtextbox')

#Buscar pela tag do elemento
driver.find_element_by_tag_name('h1')

#Buscar pelo xpath do elemento
driver.find_element_by_xpath('//input[@id="twotabsearchtextbox"]')

#Buscar pelo texto completo do elemento:
driver.find_element_by_link_text('Todos')

#Buscar pelo texto parcial do elemento:
driver.find_element_by_partial_link_text('endereço').click()

#Buscar pelo seletor CSS:
driver.find_element_by_css_selector('#searchDropdownBox')
driver.find_element_by_css_selector('[name=url]')
driver.find_element_by_css_selector('.nav-search-dropdown.searchSelect.nav-progressive-attrubute.nav-progressive-search-dropdown')

Digita um valor no site:

driver.find_element_by_id('twotabsearchtextbox').send_keys('Python')

Apagar um campo escrito:

driver.find_element_by_id('twotabsearchtextbox').clear()

Clicar em um elemento:

driver.find_element_by_id('nav-search-submit-button').click()

Movimentar a tela para um elemento (scroll):

driver.find_element_by_class_name('navFooterBackToTopText').location_once_scrolled_into_view

Extrair texto:

driver.find_element_by_id('glow-ingress-line2').text

Coletar atributos do HTML:

driver.find_element_by_id('nav-logo-sprites').get_attribute('aria-label')

Tirar print de um elemento ou do site:

#Print do elemento
driver.find_element_by_id('navbar').screenshot('menu.png')

#Print do site
driver.save_screenshot("screenshot.png")

Selecionar uma opção no Dropdown:

from selenium.webdriver.support.ui import Select

dropdown = Select(driver.find_element_by_id('searchDropdownBox'))

#Selecionar pelo texto presente
dropdown.select_by_visible_text('Computadores e Informática')

#Selecionar pelo atributo value do HTML
dropdown.select_by_value('search-alias=computers')

#Selecionar pelo número dele na lista
dropdown.select_by_index(11)

Fechar aba do navegador:
```
driver.close()
```
Fechar navegador e seu webdriver:
```
driver.quit()
```

BeautifulSoup:

Importar biblioteca para "baixar" o HTML da página:
```
from urllib.request import urlopen
```
Efetuar requisição da página HTML:
```
html = urlopen('https://quotes.toscrape.com/')
```
- Resolvendo problema de certificado SSL (se houver):
```
import ssl

ssl._create_default_https_context = ssl._create_unverified_context
```
  - Outras formas de resolver o erro de certificado SSL: https://exerror.com/urlopen-error-ssl-certificate_verify_failed-certificate-verify-failed/
Instalar o “parser” do HTML:
```
pip install beautifulsoup4
```
Importar biblioteca do “parser” de HTML:
```
from bs4 import BeautifulSoup
```

Formatar o código de HTML para “parsear”:

soup = BeautifulSoup(html.read(), 'html.parser')

ALTERNATIVA de biblioteca para "baixar" o html da página:
- Instalação da "lib":
```
pip install requests
```
- Importação da “lib”:
```
import requests
```
- Efetuar requisição da página HTML:
```
html = requests.get('https://quotes.toscrape.com/')
```
- Formatar o código de HTML para “parsear”:
```
soup = BeautifulSoup(html.text, 'html.parser')
```

Acessar uma TAG diretamente:

soup.h1

# output:
# <h1>
# <a href="/" style="text-decoration: none">Quotes to Scrape</a>
# </h1>

Acessar o PRIMEIRO elemento através dos seus atributos básicos (ID ou CLASS):

soup.find(class_='text')

# output:
# <span class="text" itemprop="text">“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”</span>

Buscar algum elemento através do seu texto (entre as TAGS):

soup.find(text="deep-thoughts")

# output: 'deep-thoughts'

Buscar mais de um elemento na página:

soup.find_all(class_='text')

# ou

soup.find_all('span', {'class': 'text'})

# output: [<span class="text" itemprop="text">“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”</span>, <span class="text" itemprop="text">“It is our choices, Harry, that show what we truly are, far more than our abilities.”</span>, ... ]

Ver atributos da TAG de HTML:

div = soup.find('div', {'class': 'quote'})
div.attrs

# output: {'class': ['quote'], 'itemscope': '', 'itemtype': 'http://schema.org/CreativeWork'}

Pegar atributos da TAG de HTML:

soup.find('div', {'class': 'quote'}).get('itemtype')

# output: 'http://schema.org/CreativeWork'

Xpath:

Guia de criação de Xpath: https://www.guru99.com/xpath-selenium.html
Extra: https://devhints.io/xpath

CSS Selector:

Guia: https://www.w3schools.com/cssref/css_selectors.asp
Cheat Sheet: https://devhints.io/css
Extra: https://www.freecodecamp.org/news/css-selectors-cheat-sheet/

Wilberhg's blog

Aula 5 - Selenium e BeautifulSoup

Webdrivers:

Selenium:

BeautifulSoup:

Xpath:

Extras: