我一次运行200个以上Spider的解决方案是为该项目创建一个自定义命令。有关实现自定义命令的更多信息，请参见http://doc.scrapy.org/en/latest/topics/commands.html#custom- project-commands。
：
from scrapy.command import ScrapyCommand
import urllib
import urllib2
from scrapy import log
class AllCrawlCommand(ScrapyCommand):
requires_project = True
default_settings = {'LOG_ENABLED': False}
def short_desc(self):
return "Schedule a run for all available spiders"
def run(self, args, opts):
url = 'http://localhost:6800/schedule.json'
for s in self.crawler.spiders.list():
values = {'project' : 'YOUR_PROJECT_NAME', 'spider' : s}
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
log.msg(response)
确保在您的settings.py中包含以下内容
COMMANDS_MODULE = 'YOURPROJECTNAME.commands'
然后，从命令行（在项目目录中），您只需键入
scrapy allcrawl
602392714
清零编程群