Python爬虫：scrapy框架Spider类参数设置

Spider设置 参数说明示例name爬虫名称，必须唯一name = “myspider”handle_httpstatus_list需要处理的状态码handle_httpstatus_list = [404]download_delay下载延时（单位：s秒）download_delay = 5allowed_domains域名限制allowed_domains = [“baidu.com”]custom_settings用户设置，单独设置爬虫参数custom_settings ={‘RETRY_TIMES’: 3}start_urls起始链接设置start_urls = [“http://www.baidu.com“] settings.py或者custom_settings 参数说明USER_AGENT请求头ROBOTSTXT_OBEY = False是否遵守robots协议DOWNLOAD_DELAY = 2同一个站点抓取延迟（秒s）CONCURRENT_REQUESTS_PER_DOMAIN = 1对同一个站点并发线程CONCURRENT_REQUESTS_PER_IP = 1对同一个ip并发线程