我正在使用Scrapy从该页面抓取数据
https://www.bricoetloisirs.ch/magasins/gardena
产品列表会动态显示。
查找网址以获取产品
https://www.bricoetloisirs.ch/coop/ajax/nextPage/(cpgnum=1&layout=7.01-14_180_69_164_182&uiarea=2&carea=%24ROOT&fwrd=frwd0&cpgsize=12)/.do?page=2&_=1473841539272
但是当我用Scrapy刮它时，它给了我空白页
<span class="pageSizeInformation" id="page0" data-page="0" data-pagesize="12">Page: 0 / Size: 12</span>
这是我的代码
# -*- coding: utf-8 -*-
import scrapy
from v4.items import Product
class GardenaCoopBricoLoisirsSpider(scrapy.Spider):
name = "Gardena_Coop_Brico_Loisirs_py"
start_urls = [
'https://www.bricoetloisirs.ch/coop/ajax/nextPage/(cpgnum=1&layout=7.01-14_180_69_164_182&uiarea=2&carea=%24ROOT&fwrd=frwd0&cpgsize=12)/.do?page=2&_=1473841539272'
]
def parse(self, response):
print response.body
最佳答案
我解决了。
# -*- coding: utf-8 -*-
import scrapy
from v4.items import Product
class GardenaCoopBricoLoisirsSpider(scrapy.Spider):
name = "Gardena_Coop_Brico_Loisirs_py"
start_urls = [
'https://www.bricoetloisirs.ch/magasins/gardena'
]
def parse(self, response):
for page in xrange(1, 50):
url = response.url + '/.do?page=%s&_=1473841539272' % page
yield scrapy.Request(url, callback=self.parse_page)
def parse_page(self, response):
print response.body