问题描述
我在 Windows Vista 64 位上使用 Python.org 版本 2.7 64 位.我有一些 Scrapy 代码试图解析此链接中的表格，标题为韦恩鲁尼的比赛历史:"...
I am using Python.org version 2.7 64 bit on Windows Vista 64 bit. I have some Scrapy code that is attempting to parse the table at this link headed 'Wayne Rooney's Match History:"...
http://www.whoscored.com/Players/3859/Fixtures/韦恩-鲁尼我到目前为止的代码是这样的:
http://www.whoscored.com/Players/3859/Fixtures/Wayne-RooneyThe code I have so far is this:
from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.utils.markup import remove_tags
from scrapy.cmdline import execute
import re
class MySpider(Spider):
name = "wiki"
allowed_domains = ["whoscored.com"]
start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]
def parse(self, response):
for row in response.selector.xpath('//table[@id="player-fixture"]//tr[td[@class="tournament"]]'):
# Is this row contains goal symbols?
list_of_goals = row.xpath('//span[@title="Goal"]')
if list_of_goals:
list = str(list_of_goals)
print remove_tags(list).encode('utf-8')
execute(['scrapy','crawl','wiki'])
这将返回表中除目标数据之外的所有数据(它也不返回助攻，但我还没有为此添加任何逻辑.这段代码是我拥有的原始代码的开发既没有进球也没有助攻:
This returns all the data from the table apart from the goal data (it doesn't return assists either, but I haven't added any logic for that yet. This code is a development of the original piece of code I had which did not return goals or assists either:
from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.utils.markup import remove_tags
from scrapy.cmdline import execute
import re
class MySpider(Spider):
name = "goal"
allowed_domains = ["whoscored.com"]
start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]
def parse(self, response):
titles = response.selector.xpath("normalize-space(//title)")
for titles in titles:
body = response.xpath('//table[@id="player-fixture"]//tr[td[@class="tournament"]]').extract()
body2 = "".join(body)
print remove_tags(body2).encode('utf-8')
execute(['scrapy','crawl','goal'])
HTML 源代码中指示目标的语句是这样的:
The statement in the HTML source that indicates a goal is this:
<span class="incident-wrapper"><span class="incidents-icon ui-icon goal" title="Goal"></span></span>
谁能告诉我为什么我在顶部列出的代码不返回以该逻辑得分的目标?这是否与球图标用于表示目标而不是单词的事实有关?
Can anyone tell me why my code listed at the top does not return goals scored with that logic? is it something to do with the fact that a ball icon is used to denote goals rather than a word?
谢谢
推荐答案
在第一个版本中你只会得到 <span class="incidents-icon ui-icon goal" title="Goal"></span> 并且没有文本，因此您会得到空字符串 - 因为您 remove_tags().
In first version you get only <span class="incidents-icon ui-icon goal" title="Goal"></span> and there is no text so you get empty string as result - because you remove_tags().
为带有目标图标"的行添加字符串GOAL":
Adding string "GOAL" for rows with "goal icon":
list_of_goals = row.xpath('//span[@title="Goal"]')
if list_of_goals:
list = str(list_of_goals)
print remove_tags(list).encode('utf-8') + "GOAL" # <-- string
或(因为 中没有文字)
or (becasue there is no text in <span title="Goal"> )
list_of_goals = row.xpath('//span[@title="Goal"]')
if list_of_goals:
print "GOAL" # <-- string
我使用 Scrapy 0.22.2 制作了我的版本.
I made my version with Scrapy 0.22.2.
可能我们使用了不同版本的 Scrapy，因为您的某些功能对我不起作用.
Probably we use differrent version of Scrapy because some your functions did't work for me.
我更喜欢 css 选择器而不是 xpath - 它们对我来说更简单.
I prefer css selectors then xpath - they are simpler for me.
from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.cmdline import execute
class MySpider(Spider):
name = "goal"
allowed_domains = ["whoscored.com"]
start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]
def parse(self, response):
sel = Selector(response)
#titles = sel.xpath("normalize-space(//title)")
#print 'titles:', titles.extract()[0]
print
print 'titles:', "".join( sel.css("title::text").extract() ).strip()
print
#rows = sel.xpath('//table[@id="player-fixture"]//tbody//tr')
rows = sel.css('table#player-fixture tbody tr')
for row in rows:
#print 'date:', row.xpath('.//td[@class="date"]/text()').extract()
#print 'result:', row.xpath('.//td[@class="result"]/a/text()').extract()
print 'date:', "".join( row.css('.date::text').extract() ).strip()
print 'result:', "".join( row.css('.result a::text').extract() ).strip()
print 'team_home:', "".join( row.css('.team.home a::text').extract() ).strip()
print 'team_away:', "".join( row.css('.team.away a::text').extract() ).strip()
print 'info:', "".join( row.css('.info::text').extract() ).strip(), "".join( row.css('.info::attr(title)').extract() ).strip()
print 'rating:', "".join( row.css('.rating::text').extract() ).strip()
print 'incidents:', ", ".join( row.css('.incidents-icon::attr(title)').extract() ).strip()
print '-'*40
#execute(['scrapy','crawl','goal'])
execute(['scrapy','runspider','main.py'])
和部分结果
titles: Wayne Rooney Match History | WhoScored.com
date: 17-08-2013
result: 1 : 4
team_home: Swansea
team_away: Manchester United
info: 28' Minutes played in this match
rating: 7.26
incidents: Assist, Assist
----------------------------------------
date: 26-08-2013
result: 0 : 0
team_home: Manchester United
team_away: Chelsea
info: 90' Minutes played in this match
rating: 7.03
incidents:
----------------------------------------
date: 14-09-2013
result: 2 : 0
team_home: Manchester United
team_away: Crystal Palace
info: 90' Minutes played in this match
rating: 8.44
incidents: Man of the Match, Goal
----------------------------------------
date: 17-09-2013
result: 4 : 2
team_home: Manchester United
team_away: Bayer Leverkusen
info: 84' Minutes played in this match
rating: 9.18
incidents: Goal, Goal, Assist
----------------------------------------
date: 22-09-2013
result: 4 : 1
team_home: Manchester City
team_away: Manchester United
info: 90' Minutes played in this match
rating: 7.17
incidents: Goal, Yellow Card
----------------------------------------
date: 25-09-2013
result: 1 : 0
team_home: Manchester United
team_away: Liverpool
info: 90' Minutes played in this match
rating:
incidents: Man of the Match, Assist
----------------------------------------
这篇关于Xpath() 方法不使用 Scrapy 返回结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！