如何从selenium webdriver中的网页获取数据

问题描述

我想从这个获取公司名称，电子邮件，电话号码，并将这些内容在一个excel文件。我想为网站的所有页面做同样的事情。我有了在浏览器中获取链接并在它们之间切换的逻辑。我无法从网站获取数据。任何人都可以提供对我写的代码的增强。

I want to fetch company name, email, phone number from this Link and put these contents in an excel file. I want to do the same for the all pages of the website. I have got the logic to fetch the the links in the browser and switch in between them. I'm unable to fetch the data from the website. Can anybody provide me an enhancement to the code i have written.

以下是我写的代码：

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.keys import Keys
import time
from lxml import html
import requests
import xlwt

browser = webdriver.Firefox() # Get local session of firefox

# 0 wait until the pages are loaded
browser.implicitly_wait(3) # 3 secs should be enough. if not, increase it

browser.get("http://ae.bizdirlib.com/taxonomy/term/1493") # Load page
links = browser.find_elements_by_css_selector("h2 > a")

#print link
for link in links:
    link.send_keys(Keys.CONTROL + Keys.RETURN)
    link.send_keys(Keys.CONTROL + Keys.PAGE_UP)
#tree = html.fromstring(link.text)
    time.sleep(5)

推荐答案

您尝试从这些链接中获取数据？我可以给你一个例子。如果你想得到S.S.D Middle East - F.Z.E，点击链接后，它有很多属性，从公司名称到行业。如果您想查找并检索其公司名称，您需要首先找到它并获取其文本：

What have you tried to fetch data from those links? I can show you an example. If you want to get "S.S.D Middle East - F.Z.E", after you click on its link, it has a number of attributes ranging from Company name to Industry. If you want to locate and retrieve its company name, you need to locate it first and get its text:

companyNameElement = browser.find_element_by_css_selector("div[class="region region-content"] span[itemprop="name"]");
companyName = companyNameElement.getText()

您应该获得SSD Middle East - FZE在这个companyName变量中。

You should get "S.S.D Middle East - F.Z.E" in this companyName variable.

这篇关于如何从selenium webdriver中的网页获取数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！