pyspider 初次使用

一 安装

pip  install pyspider

请安装PhantomJS：http：//phantomjs.org/build.html

二 检验是否启动成功

cmd中输入: pyspider

安装问题解决

python版本：3.6

1、启动报错

    raise ValueError("Invalid configuration:\n  - " + "\n  - ".join(errors))

ValueError: Invalid configuration:

  - Deprecated option 'domaincontroller': use 'http_authenticator.domain_control

ler' instead.

解决方法：

将wsgidav替换为2.4.1

•# python -m pip install wsgidav==2.4.1

 

2、PhantomJS配置

windows:在官网http://phantomjs.org/download.html下载对应版本的程序，然后放到python安装目录的python.exe同级目录下。

pyspider Web预览界面太小的解决方法

完美css代码：

body{margin:0;padding:0;height:100%;overflow:hidden}.warning{color:#f0ad4e}.error{color:#d9534f}#control{z-index:9999;min-width:760px;width:100%;height:35px;position:fixed;left:0;right:0;box-shadow:0 1px 2px #999}#control div{line-height:35px;margin-left:10px;margin-right:10px}#control .webdav-btn{position:relative;float:right;padding:1px 7px 0;line-height:21px;border-radius:5px;border:1px solid #428bca;background:#fff;color:#428bca;cursor:pointer;margin:6px 0 0 10px}#control .webdav-btn:hover{background:#6aa3d5;color:#fff}#control .webdav-btn.active{background:#428bca;color:#fff}#editarea{width:100%;position:fixed;top:37px}#editarea,.debug-panel{left:0;right:0;bottom:0}.debug-panel{position:absolute;top:0}.resize{ padding: 0px; color: rgb(128, 0, 128); line-height: 1.5 !important;">555;cursor:ew-resize}.resize:hover+.debug-panel{border-left:1px dashed #555!important}.overlay{position:absolute;top:0;bottom:0;left:0;right:0;z-index:9999;background:rgba(0,0,0,.4)}.focus .CodeMirror-activeline-background{background:#e8f2ff!important}.CodeMirror-activeline-background{background:transparent!important}#task-panel{height:100%;overflow-x:auto}#run-task-btn{z-index:99;position:absolute;top:0;right:0;background:#5cb85c;border-radius:0 0 0 5px;color:#fff;margin:0;padding:3px 7px 5px 10px;cursor:pointer;font-weight:700;line-height:15px}#run-task-btn:hover{background:#449d44}#undo-redo-btn-group{z-index:99;position:absolute;top:0;right:0;background:#91cf91;border-radius:0 0 0 5px;color:#fff;margin:0;padding:3px 7px 5px 10px;cursor:pointer;font-weight:700;line-height:15px;top:auto;bottom:0;border-radius:5px 0 0 0;padding:5px 0 3px;overflow:hidden}#undo-redo-btn-group:hover{background:#6ec06e;background:#91cf91}#undo-redo-btn-group a{color:#fff;text-decoration:none;padding:5px 7px 3px 10px}#undo-redo-btn-group a:hover{background:#6ec06e}#save-task-btn{z-index:99;position:absolute;top:0;right:0;background:#428bca;border-radius:0 0 0 5px;color:#fff;margin:0;padding:3px 7px 5px 10px;cursor:pointer;font-weight:700;line-height:15px}#save-task-btn:hover{background:#3071a9}#task-editor{position:relative}#task-editor .CodeMirror{height:auto;padding-bottom:3px;background:#c7e6c7}#task-editor .CodeMirror-scroll{overflow-x:auto;overflow-y:hidden}#task-editor.focus .CodeMirror-activeline-background{background:#eaf6ea!important}#tab-control{list-style-type:none;position:absolute;bottom:0;right:0;margin:8px 20px;padding:0}#tab-control li{position:relative;float:right;padding:1px 7px 0;line-height:21px;margin-left:10px;border-radius:5px;border:1px solid #428bca;background:#fff;color:#428bca;cursor:pointer}#tab-control li:hover{background:#6aa3d5;color:#fff}#tab-control li.active{background:#428bca;color:#fff}#tab-control li span{position:absolute;top:-5px;right:-10px;background:#d9534f;color:#fff;font-size:80%;font-weight:700;padding:2px 5px 0;border-radius:10px}#debug-tabs{margin-bottom:45px}#tab-web.fixed{padding-top:24px}#tab-web iframe{border-width:0;width:100%;height:900px !important}#tab-html{margin:0;padding:7px 5px}#tab-html pre{margin:0;padding:0}#tab-follows .newtask{position:relative;height:30px;line-height:30px;background:#fceedb;border-bottom:1px solid #f0ad4e;border-top:1px solid #f0ad4e;margin-top:-1px;padding-left:5px;padding-right:70px;overflow:hidden;white-space:nowrap;text-overflow:ellipsis;cursor:pointer}#tab-follows .newtask:hover,#tab-follows .newtask:hover .task-more{background:#f8d9ac}#tab-follows .newtask .task-callback{color:#ec971f}#tab-follows .newtask .task-url{font-size:95%;text-decoration:underline;font-weight:lighter;color:#428bca}#tab-follows .newtask .task-more{position:absolute;right:33px;top:0;float:right;color:#f0ad4e;padding:0 10px;background:#fceedb;border-radius:10px}#tab-follows .newtask .task-run{position:absolute;right:0;top:0;font-size:80%;padding:0 10px 0 30px;float:right;border-bottom:1px solid #a3d7a3;border-top:1px solid #a3d7a3;background:#80c780;color:#fff;text-shadow:0 0 10px #fff;font-weight:700}#tab-follows .newtask .task-run:hover{background:#5cb85c}#tab-follows .task-show pre{margin:5px 5px 10px}#python-editor{position:absolute;top:0;width:100%;bottom:0}#python-editor .CodeMirror{height:100%;padding-bottom:20px}#python-log{width:100%;min-height:10px;max-height:40%;background:rgba(0,0,0,.6);overflow:auto}#python-log #python-log-show{z-index:89;width:auto;padding-top:5px;background:#d9534f;box-shadow:0 2px 20px #d9534f;cursor:pointer}#python-log pre{margin:0;padding:10px;color:#fff}#css-selector-helper{padding:0;width:100%;height:24px;text-align:right;white-space:nowrap}#css-selector-helper.fixed{position:absolute;top:0}#css-selector-helper button{line-height:16px;vertical-align:2px}span.element{position:relative;height:24px;display:inline-block;padding:0 .2em;cursor:pointer;color:#afafaf;z-index:99999}span.element.invalid{display:none}span.element.selected{color:#000}span.element:hover{ul{display:block}span.element>ul{display:none;margin:0;padding:0;position:absolute;top:24px;left:0;border:1px solid #000;border-top-width:0;color:#afafaf}span.element>ul>li{display:block;text-align:left;white-space:nowrap;padding:0 4px}span.element>ul>li.selected{color:#000}span.element>ul>li:hover{padding:0;border:0;margin:0;padding-right:.2em;font-size:1em;text-align:right;width:100%;margin-left:-100px;background:#eee}

替换/pyspider/webui/static/debug.min.css文件中所有内容

三 入门简介

你的第一个脚本

from pyspider.libs.base_handler import *

class Handler(BaseHandler):

    crawl_config = {

        #配置信息

    }

    #全局变量,获取数据库句柄...

 client=pymongo.MongoClient(host='106.12.108.236',port=27017)

    db=client['trip']

    @every(minutes=24 * 60)

    def on_start(self):

    #程序请求的入口,start_url

        self.crawl('http://scrapy.org/', callback=self.index_page)

    @config(age=10 * 24 * 60 * 60)

    def index_page(self, response):

    #解析start_url获取详情页的url

        for each in response.doc('a[href^="http"]').items():

            self.crawl(each.attr.href, callback=self.detail_page)

    @config(priority=2)

    def detail_page(self, response):

    #获取详情页的信息

        return {

            "url": response.url,

            "title": response.doc('title').text(),

        }

四 页面简介

点击save按钮

接下来执行run

显示当前待解析页面的页面

项目启动和删除

五 页面爬取代码

#!/usr/bin/env python
# -*- encoding: utf-8 -*-

# Created on 2019-04-10 20:27:19

# Project: tripadvisor

from pyspider.libs.base_handler import *

import pymongo

class Handler(BaseHandler):

    crawl_config = {

    }

　　#连接数据库

    client=pymongo.MongoClient(host='106.12.108.236',port=27017)

    db=client['trip']

    @every(minutes=24 * 60)

    def on_start(self):

        self.crawl('https://www.tripadvisor.cn/Attractions-g186338-Activities-c47-t163-London_England.html', callback=self.index_page,validate_cert=False)

    @config(age=10 * 24 * 60 * 60)

    def index_page(self, response):

        for each in response.doc('.listing_title > a').items():

            self.crawl(each.attr.href, callback=self.detail_page,validate_cert=False)

        next=response.doc('.pagination .nav.next').attr.href

        self.crawl(next,callback=self.index_page,validate_cert=False)

    @config(priority=2)

    def detail_page(self, response):

        url=response.url

        name=response.doc('.h1').text()

        rate=response.doc('a > .reviewCount').text()

        address=response.doc('.contactInfo > .address').text()

        phone=response.doc('.contact > .is-hidden-mobile > div').text()

        decs=response.doc('#component_5 > div > div:nth-child(2)').text()

        return {

            'url':url,

            'name':name,

            'rate':rate,

            'address':address,

            'phone':phone,

            'decs':decs,

        }

    def on_result(self,result):

        '如果有详情页有返回值，就调用此函数'

        if result:

            self.save_to_mongo(result)

    def save_to_mongo(self,result):
　　　　 # 保存到数据库

        if self.db['london'].insert(result):

            print('save to mongo',result)

    

六 查看 pyspider all 执行后产生的本地文件

七 设置启动配置