使用 BeautifulSoup 和 Selenium 抓取一个网站的多个网页的内容

问题描述

我想废弃的网站是:

我的代码:

 from bs4 import BeautifulSoup从 urllib.request 导入 urlopen 作为 uReqfrom selenium import webdriver;导入时间from selenium.webdriver.common.by import By从 selenium.webdriver.support.ui 导入 WebDriverWait从 selenium.webdriver.support 导入 expected_conditions 作为 EC从 selenium.webdriver.common.desired_capabilities 导入 DesiredCapabilitiesfirefox_capabilities = DesiredCapabilities.FIREFOXfirefox_capabilities['marionette'] = Truefirefox_capabilities['binary'] = '/etc/firefox'驱动程序 = webdriver.Firefox(capabilities=firefox_capabilities)url = "http://www.mouthshut.com/mobile-operators/Reliance-Jio-reviews-925812061"driver.get(url)等待 = WebDriverWait(驱动程序，10)汤=BeautifulSoup(driver.page_source,"lxml")容器 = 汤.findAll("ul",{"class":"pages table"})容器 [0] = 汤.findAll("li")li_len = len(容器[0])对于soup.find("ul",{"class":"pages table"}) 中的项目:li_text = item.select("li")[li_len].text打印(li_text:{}
".格式(li_text))驱动程序退出()

我需要帮助找出获取最后页码的代码中的错误.另外，如果有人提供相同的替代解决方案并建议实现我的意图的方法，我将不胜感激.

解决方案

如果你想得到上面链接的最后一个页码继续进行，就是499 您可以使用 Selenium 或 Beautifulsoup 如下:

硒:

from selenium import webdriverdriver = webdriver.Firefox(executable_path=r'C:UtilityBrowserDriversgeckodriver.exe')url = "http://www.mouthshut.com/mobile-operators/Reliance-Jio-reviews-925812061"driver.get(url)element = driver.find_element_by_xpath("//div[@class='row pagination']//p/span[contains(.,'Reviews on Reliance Jio')]")driver.execute_script("返回参数[0].scrollIntoView(true);", element)print(driver.find_element_by_xpath("//ul[@class='分页表']/li/ul[@class='pages table']//li[last()]/a").get_attribute("innerHTML"))驱动程序退出()

控制台输出:

499

美汤:

导入 bs4从 bs4 导入 BeautifulSoup 作为汤从 urllib.request 导入 urlopen 作为 uRequrl = "http://www.mouthshut.com/mobile-operators/Reliance-Jio-reviews-925812061"uClient = uReq(url)page_html = uClient.read()uClient.close()page_soup = 汤(page_html，html.parser")container = page_soup.find("ul",{"class":"pages table"})all_li = container.findAll("li")last_div = 无对于 all_li:pass 中的 last_div如果 last_div:content = last_div.getText()打印(内容)

控制台输出:

499

The website I want to scrap is :

http://www.mouthshut.com/mobile-operators/Reliance-Jio-reviews-925812061

I want to get the last page number of the above the link for proceeding, which is 499 while taking the screenshot.

My code :

   from bs4 import BeautifulSoup
   from urllib.request import urlopen as uReq
   from selenium import webdriver;import time
   from selenium.webdriver.common.by import By
   from selenium.webdriver.support.ui import WebDriverWait
   from selenium.webdriver.support import expected_conditions as EC
   from selenium.webdriver.common.desired_capabilities import         DesiredCapabilities

   firefox_capabilities = DesiredCapabilities.FIREFOX
   firefox_capabilities['marionette'] = True
   firefox_capabilities['binary'] = '/etc/firefox'

   driver = webdriver.Firefox(capabilities=firefox_capabilities)
   url = "http://www.mouthshut.com/mobile-operators/Reliance-Jio-reviews-925812061"

   driver.get(url)
   wait = WebDriverWait(driver, 10)
   soup=BeautifulSoup(driver.page_source,"lxml")
   containers = soup.findAll("ul",{"class":"pages table"})
   containers[0] = soup.findAll("li")
   li_len = len(containers[0])
   for item in soup.find("ul",{"class":"pages table"}) :
   li_text = item.select("li")[li_len].text
   print("li_text : {}
".format(li_text))
   driver.quit()

I need help to figure out the error in my code for getting the last page number. Also, I would be grateful if someone give the alternate solution for the same and suggest ways to achieve my intention.

解决方案

If you want to get the last page number of the above the link for proceeding, which is 499 you can use either Selenium or Beautifulsoup as follows :

---

Selenium :

from selenium import webdriver

driver = webdriver.Firefox(executable_path=r'C:UtilityBrowserDriversgeckodriver.exe')
url = "http://www.mouthshut.com/mobile-operators/Reliance-Jio-reviews-925812061"
driver.get(url)
element = driver.find_element_by_xpath("//div[@class='row pagination']//p/span[contains(.,'Reviews on Reliance Jio')]")
driver.execute_script("return arguments[0].scrollIntoView(true);", element)
print(driver.find_element_by_xpath("//ul[@class='pagination table']/li/ul[@class='pages table']//li[last()]/a").get_attribute("innerHTML"))
driver.quit()

Console Output :

499

---

Beautifulsoup :

import bs4
from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq

url = "http://www.mouthshut.com/mobile-operators/Reliance-Jio-reviews-925812061"
uClient = uReq(url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")
container = page_soup.find("ul",{"class":"pages table"})
all_li = container.findAll("li")
last_div = None
for last_div in all_li:pass
if last_div:
    content = last_div.getText()
    print(content)

Console Output :

499

这篇关于使用 BeautifulSoup 和 Selenium 抓取一个网站的多个网页的内容的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！