问题描述
我很难用我的网页抓取代码找出正确的路径.
I have a hard time figuring out a correct path with my web scraping code.
我试图从 http://financials 中抓取不同的信息.Morningstar.com/company-profile/c.action?t=AAPL.我尝试了几种路径,有些似乎有效,有些则无效.我对操作详情下的 CIK 感兴趣
I am trying to scrape different info from http://financials.morningstar.com/company-profile/c.action?t=AAPL.I have tried several paths, and some seem to work and some not.I am interested in CIK under Operation Details
page = requests.get('http://financials.morningstar.com/company-profile/c.action?t=AAPL')
tree=html.fromstring(page.text)
#desc = tree.xpath('//div[@class="r_title"]/span[@class="gry"]/text()') #works
#desc = tree.xpath('//div[@class="wrapper"]//div[@class="headerwrap"]//div[@class="h_Logo"]//div[@class="h_Logo_row1"]//div[@class="greeter"]/text()') #works
#desc = tree.xpath('//div[@id="OAS_TopLeft"]//script[@type="text/javascript"]/text()') #works
desc = tree.xpath('//div[@class="col2"]//div[@id="OperationDetails"]//table[@class="r_table1 r_txt2"]//tbody//tr//th[@class="row_lbl"]/text()')
我想不出最后的路径.似乎我正确地遵循了路径,但我得到了空列表.
I can't figure the last path. It seems like I am following the path correctly, but I get empty list.
推荐答案
问题在于操作细节是通过额外的 GET 请求单独加载的.在维护网络抓取会话的代码中模拟它:
The problem is that Operational Details are loaded separately with an additional GET request. Simulate it in your code maintaining a web-scrapin session:
import requests
from lxml import html
with requests.Session() as session:
page = session.get('http://financials.morningstar.com/company-profile/c.action?t=AAPL')
tree = html.fromstring(page.text)
# get the operational details
response = session.get("http://financials.morningstar.com/company-profile/component.action", params={
"component": "OperationDetails",
"t": "XNAS:AAPL",
"region": "usa",
"culture": "en-US",
"cur": "",
"_": "1444848178406"
})
tree_details = html.fromstring(response.content)
print tree_details.xpath('.//th[@class="row_lbl"]//text()')
旧答案:
只是你应该从表达式中删除 tbody
:
It's just that you should remove tbody
from the expression:
//div[@class="col2"]//div[@id="OperationDetails"]//table[@class="r_table1 r_txt2"]//tr//th[@class="row_lbl"]/text()
tbody
浏览器插入的元素,用于定义表格中的数据行.
tbody
is an element that is inserted by the browser to define the data rows in a table.
这篇关于网页抓取,获取空列表的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!