我正在使用scrapy来爬行页面,我能够获得所有可见文本的简单内容。但是,有些文本对于搜寻器不可见,最终显示为空格。

例如,查看页面资源使我可以查看以下字段:

https://www.dropbox.com/s/f056mffmuah6uu4/Screenshot%202015-07-23%2018.23.32.png?dl=0

我已经尝试了无数次通过xpath和CSS访问此字段,但每次尝试后都无法获取这些字段。

当我尝试类似的东西:

response.xpath('//text()').extract()


我根本看不到文本转储中的这些字段。

谁会想到为什么这些字段不可见?该网站是:https://www.buzzbuzzhome.com/uc/units/houses/sapphire

最佳答案

在您的Spider中,您需要向https://www.buzzbuzzhome.com/bbhAjax/Development/UnitPriceHistory端点发出附加的XHR POST请求,以获取提供必要标头和POST参数的价格历史记录:

import json
import scrapy


class BuzzSpider(scrapy.Spider):
    name = 'buzzbuzzhome'
    allowed_domains = ['buzzbuzzhome.com']
    start_urls = ['https://www.buzzbuzzhome.com/uc/units/houses/sapphire']

    def parse(self, response):
        unit_id = response.xpath("//div[@id = 'unitDetails']/@data-unit-id").extract()[0]
        development_url = "uc"
        new_relic_id = response.xpath("//script[contains(., 'xpid')]").re(r'xpid:"(.*?)"')

        params = {"developmentUrl": development_url, "unitID": unit_id}
        yield scrapy.Request("https://www.buzzbuzzhome.com/bbhAjax/Development/UnitPriceHistory",
                             method="POST",
                             body=json.dumps(params),
                             callback=self.parse_history,
                             headers={
                                 "Accept": "*/*",
                                 "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36",
                                 "X-Requested-With": "XMLHttpRequest",
                                 "X-NewRelic-ID": new_relic_id,
                                 "Origin": "https://www.buzzbuzzhome.com",
                                 "Host": "www.buzzbuzzhome.com",
                                 'Content-Type': 'application/json; charset=UTF-8'
                             })

    def parse_history(self, response):
        for row in response.css("div.row"):
            title = row.xpath(".//div[@class='content-title']/text()").extract()[0].strip()
            text = row.xpath(".//div[@class='content-text']/text()").extract()[0].strip()

            print title, text


印刷品:

05/25/2015 Unit listed as Sold
12/18/2014 Unit listed as For Sale
11/24/2014 Unit price increased  by 1.54% to $461,990
11/04/2014 Unit price increased  by 6.81% to $454,990
10/02/2014 Unit price increased  by 4.67% to $425,990
01/22/2014 Unit price increased  by 2.52% to $406,990
12/06/2013 Unit listed as For Sale at $396,990

关于python - scrapy无法从网站提取一些数据,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/31599124/

10-14 19:21
查看更多