python - Scrapy:在scrapyd运行多个蜘蛛-python逻辑错误

Scrapy 1.4

我正在使用此脚本（Run multiple scrapy spiders at once using scrapyd）在Scrapyd安排多个蜘蛛。在我使用Scrapy 0.19并运行良好之前。

我收到错误：TypeError: create_crawler() takes exactly 2 arguments (1 given)

所以现在我不知道问题是在Scrapy版本还是简单的python逻辑问题（我是python的新手）

我做了一些修改，以检查Spider是否在数据库上处于活动状态。

class AllCrawlCommand(ScrapyCommand):

    requires_project = True
    default_settings = {'LOG_ENABLED': False}

    def short_desc(self):
        return "Schedule a run for all available spiders"

    def run(self, args, opts):

        cursor = get_db_connection()
        cursor.execute("SELECT * FROM lojas WHERE disponivel = 'S'")
        rows = cursor.fetchall()

        # Coloco todos os dominios dos sites em uma lista
        # La embaixo faco uma verificacao para rodar somente os
        # que estao disponiveis e somente os que batem o dominio do site
        sites = []
        for row in rows:
            site = row[2]
            print site

            # adiciono cada site na lista
            sites.append(site)

        url = 'http://localhost:6800/schedule.json'
        crawler = self.crawler_process.create_crawler()
        crawler.spiders.list()
        for s in crawler.spiders.list():
            #print s
            if s in sites:

                values = {'project' : 'esportifique', 'spider' : s}
                r = requests.post(url, data=values)
                print(r.text)

最佳答案

根据parik建议的链接，这是我所做的：

from scrapy.utils.project import get_project_settings
from scrapy.crawler import CrawlerProcess
import requests

setting = get_project_settings()
process = CrawlerProcess(setting)

url = 'http://localhost:6800/schedule.json'

cursor = get_db_connection()
cursor.execute("SELECT * FROM lojas WHERE disponivel = 'S'")
rows = cursor.fetchall()

# Coloco todos os dominios dos sites em uma lista
# La embaixo faco uma verificacao para rodar somente os
# que estao disponiveis e somente os que batem o dominio do site
sites = []
for row in rows:
    site = row[2]
    print site

    # adiciono cada site na lista
    sites.append(site)

for spider_name in process.spiders.list():
    print ("Running spider %s" % (spider_name))
    #process.crawl(spider_name,query="dvh") #query dvh is custom argument used in your scrapy
    if spider_name in sites:
        values = {'project' : 'esportifique', 'spider' : spider_name}
        r = requests.post(url, data=values)