scrapy+selenium之中国裁判文书网文书爬取

scrapy+selenium之中国裁判⽂书⽹⽂书爬取浅尝python⽹络爬⾍，略有⼼得。有不⾜之处，请多指正爬取内容：裁判⽂书爬取框架：scrapy框架+selenium模拟浏览器访问开始想暴⼒分析⽹页结构获取数据，哈哈哈哈哈，天真了。看来⾃⼰什么⽔平还真不知道。之后锁定pyspider框架，搞了四五天。该框架对于页⾯超链接的连续访问问题，可以⼿动点击单个链接测试，但是通过外部“run”操作，会获取不到数据。其实最后发现很多博客说pyspider的官⽹⽂档已经很久没有更新了，企业、项⽬般都会⽤到scrapy。scrapy框架结构如下图：中间为scrapy引擎，左侧item为爬取内容实体，相关pipeline作⽤是在yielditem语句返回之后，先通过pipelines.py些类进⾏处理操作，⽐如：存⼊MongoDB中；下⾯为spiders实现原始页⾯的请求、对响应页⾯的解析以及将后续访问页⾯传输到上⾯Scheduler访问序列中，然后每次从该序列选取⼀个链接通过右边Downloader访问Internet。⽽其中有两个中间件，SpiderMiddlewares和DownloaderMiddlewares。前者使⽤频率不如后者，⽽下载中间件是当Scheduler发来请求时，可以在访问Internet之前，对链接添加⼀些参数信息；或者是在返回响应页⾯之前，通过selenium进⾏动态js渲染以及模拟浏览器可以实现些点击翻页操作等等。上⾯说到⼀个js渲染问题，这⾥使⽤的是selenium，其实还有种渲染⽅法是splash渲染。但是通过splash需要在Spiders代码中编写Lua脚本，这种⽅法实现页⾯访问，翻页，点击等的不如selenium来的⽅便；另⽅⾯，对于⼤型爬取任务来说，毕竟selenium为阻塞式访问界⾯，这将会给我们的项⽬平添时耗，⽽splash可以实现分布式访问。先选择的是selenium操作，但是由于⽹站服务器响应问题，需要设置对于些⽹页标签元素显⽰与否的等待时间以及对于整个⽹页的模拟访问timeout时间。期间真的是各种调参，进⾏单元测试，其实最后页⾯访问成功率基本为100%。代码为爬取前两页数据，单页⽂书数量为5，通过selenium设置单页⽂书数量为最⼤值15，这样可以省去很多主页⾯的跳转、访问、解析时间。由于单页15个访问链接，外加2个⽂书列表界⾯，总共32次响应返回，响应码全部为"200"（成功响应）。以下为代码执⾏结果图：此时数据库中显⽰插⼊的30条⽂书实体数据，以下为MongoDB中相应collection更新的数据：踩坑合集（bugs）：1.详情页解析函数测试，通过xpath或css函数得到标签元素⽽返回的对象为"list"类型。⽽之后使⽤extract()，返回的也是"list"类型，但是不可以继续调⽤xpath或css函数；2.通过selenium模拟浏览器操作，对⼀些标签元素的显⽰设置等待时间以及整个访问页⾯响应的超时时间。对时间多少，标签元素的选择等进⾏测试；3.将⽂书主体内容不同部分：当事⼈、法院原由、判决结果进⾏拆分。通过div个数以及⼀些特殊情况的解决，可以解决不同的div标签个数情况下的数据清洗。以及对于⼀些不授权展⽰的特殊情况的处理。4.开始时，为了通过⼀个下载中间件响应⽂书列表和⽂书详情两种页⾯。解决办法：通过meta字段，设置哨兵区别两种不同链接。5.由于不论是第⼏页发送的请求，链接总是不变的。实际操作中，点击后续页⾯只是内容发⽣变化⽽链接⾃始⾄终不会改变。所以在spiders代码中需要设置访问链接的"dont_filter"参数为true，即不对重复链接过滤。6.由于5造成的影响，如果单纯模拟点击下⼀页链接，到了第⼆页还是原来链接，再点击下⼀页还会回到第⼀页。所以需要直接点击模拟点击对应的页码，但是开始最多只显⽰6个可选页码按钮点击6页⾯才会显⽰后续页⾯按钮。所以通过规律，编写函数获取点击的按钮序列。7.由于6中对页码数较⼤的页⾯访问需要模拟多次点击按钮跳转，所以可以⾸先模拟选择下⾯单页⽂书列表可以展⽰的⽂书数量为最⼤15，是原始的3倍，因此翻页的次数⼤⼤降低，以提⾼爬取效率。好了，话不多说。上代码：spiders:#-*-coding:utf-8-*-importtimeimportscrapyfromscrapyimportRequestfromurllib.parseimporturljoinfromscrapyimportSelectorfromwenshu.itemsimportWenshuItemimportrequestsfromscrapyimportSpiderfromscrapy_splashimportSplashRequestfromscrapy_splashimportSplashRequestfromurllib.parseimportquotescript_one="""functionmain(splash,args)splash.html5_media_enabled=truesplash.plugins_enabled=truesplash.response_body_enabled=truesplash.request_body_enabled=truesplash.js_enabled=truesplash.resource_timeout=30splash.images_enabled=falseassert(splash:go(args.url))assert(splash:wait(args.wait))return{html=splash:html(),har=splash:har(),png=splash:png()}end"""script="""functionmain(splash,args)splash.resource_timeout=40splash.images_enabled=trueassert(splash:go(args.url))assert(splash:wait(args.wait))js=string.format("isnext=function(page=%s){if(page==1){returndocument.html;}document.querySelector('a.pageButton:nth-child(8)').click();returndocsplash:runjs(js)splash:evaljs(js)assert(splash:wait(args.wait))returnsplash:html()end"""detail='''functionmain(splash,args)splash.resource_timeout=20splash.images_enable=falseassert(splash:go(args.url))assert(splash:wait(args.wait))returnsplash:html()end'''classWsSpider(scrapy.Spider):name='ws'allowed_domains=['']base_url='/website/wenshu'#start_urls='/website/wenshu/181217BMTKHNT2W0/index.html?pageId=11c90f39fb379fbf4ab85bb180682ce0&s38=300&fymc=%start_urls='/website/wenshu/181217BMTKHNT2W0/index.html?s38=300&fymc=%E6%B2%B3%E5%8C%97%E7%9C%81%E9%Adefstart_requests(self):forpageinrange(1,self.settings.get('MAX_PAGE')+1):self.logger.debug(str(page))#ifpage==1:##self.logger.debug(self.settings.get('MAX_PAGE'))self.logger.debug('=====================================')#elifpage==2:#self.logger.debug('+++++++++++++++++++++++++++++++++++++')yieldRequest(url=self.start_urls,callback=self.parse_origin,meta={'page':page,'tag':0},dont_filter=True)#yieldSplashRequest(url=self.start_urls,callback=self.parse_origin,endpoint='execute',args={'lua_source':script_one,#'wait':5,'page':page#})defparse_origin(self,response):#self.logger.debug(response.text)urls=response.xpath('//a[@class="caseName"]/@href').extract()self.logger.debug(str(response.status))self.logger.debug(str(response.status))self.logger.debug(str(response.status))forurlinurls:target_url=self.base_url+url[2:]self.logger.debug('url:'+url)self.logger.debug('target_url:'+target_url)forurlinurls:#each_url=url.xpath('./@href').extract_first()#self.logger.debug(str(url))target_url=self.base_url+url[2:]yieldRequest(url=target_url,callback=self.parse_detail,meta={'tag':1},dont_filter=False)#defparse_detail()items：#-*-coding:utf-8-*-#Defineherethemodelsforyourscrapeditems##Seedocumentationin:#/en/latest/topics/items.htmlfromscrapyimportItem,FieldclassWenshuItem(Item):#definethefieldsforyouritemherelike:#name=scrapy.Field()collection='wenshu'title=Field()#标题release=Field()#发布时间views=Field()#访问量court=Field()#审判法院type=Field()#⽂书类型prelude=Field()#⾸部parties=Field()#当事⼈justification=Field()#理由end=Field()#结果chief=Field()#审判长judge=Field()#审判员time=Field()#审判时间assistant=Field()#法官助理clerk=Field()#书记员pipelines:#-*-coding:utf-8-*-#Defineyouritempipelineshere##Don'tforgettoaddyourpipelinetotheITEM_PIPELINESsetting#See:/en/latest/topics/item-pipeline.htmlimportpymongofromwenshu.itemsimportWenshuItemclassWsMongoPipeline(object):def__init__(self,mongo_uri,mongo_db):self.mongo_uri=mongo_uriself.mongo_db=mongo_db@classmethoddeffrom_crawler(cls,crawler):returncls(mongo_uri=crawler.settings.get('MONGO_URI'),mongo_db=crawler.settings.get('MONGO_DB'))defopen_spider(self,spider):self.client=pymongo.MongoClient(self.mongo_uri)self.db=self.client[self.mongo_db]defclose_spider(self,spider):self.client.close()defprocess_item(self,item,spider):ifisinstance(item,WenshuItem):self.db[item.collection].insert(dict(item))returnitemclassWenshuPipeline(object):defprocess_item(self,item,spider):returnitemmiddlewares:#-*-coding:utf-8-*-#Defineherethemodelsforyourspidermiddleware##Seedocumentationin:#/en/latest/topics/spider-middleware.htmlimporttimefromscrapyimportsignalsfromseleniumimportwebdriverfrommon.exceptionsimportTimeoutExceptionfrommon.byimportByfromselenium.webdriver.support.uiimportWebDriverWaitfromselenium.webdriver.supportimportexpected_conditionsasECfromscrapy.httpimportHtmlResponsefromloggingimportgetLoggerfromselenium.webdriver.support.selectimportSelectfromselenium.webdriver.support.selectimportSelectclassWenshuSeleniumMiddleware(object):def__init__(self,timeout=None,service_args=[]):self.logger=getLogger(__name__)self.timeout=timeoutself.browser=webdriver.Chrome()self.browser.set_window_size(1200,600)self.browser.set_page_load_timeout(self.timeout)self.wait=WebDriverWait(self.browser,self.timeout)def__del__(self):self.browser.close()#defprocess_request()classWenshuSpiderMiddleware(object):#Notallmethodsneedtobedefined.Ifamethodisnotdefined,#scrapyactsasifthespidermiddlewaredoesnotmodifythe#passedobjects.@classmethoddeffrom_crawler(cls,crawler):#ThismethodisusedbyScrapytocreateyourspiders.s=cls()crawler.signals.connect(s.spider_opened,signal=signals.spider_opened)returnsdefprocess_spider_input(self,response,spider):#Calledforeachresponsethatgoesthroughthespider#middlewareandintothespider.#ShouldreturnNoneorraiseanexception.returnNonedefprocess_spider_output(self,response,result,spider):#CalledwiththeresultsreturnedfromtheSpider,after#ithasprocessedtheresponse.#MustreturnaniterableofRequest,dictorItemobjects.foriinresult:yieldidefprocess_spider_exception(self,response,exception,spider):#Calledwhenaspiderorprocess_spider_input()method#(fromotherspidermiddleware)raisesanexception.#ShouldreturneitherNoneoraniterableofRequest,dict#orItemobjects.passdefprocess_start_requests(self,start_requests,spider):#Calledwiththestartrequestsofthespider,andworks#similarlytotheprocess_spider_output()method,except#thatitdoesn’thavearesponseassociated.#Mustreturnonlyrequests(notitems).forrinstart_requests:yieldrdefspider_opened(self,spider):defspider_opened(self,spider):('Spideropened:%s'%)classWenshuDownloaderMiddleware(object):#Notallmethodsneedtobedefined.Ifamethodisnotdefined,#scrapyactsasifthedownloadermiddlewaredoesnotmodifythe#passedobjects.@classmethoddeffrom_crawler(cls,crawler):#ThismethodisusedbyScrapytocreateyourspiders.s=cls()crawler.signals.connect(s.spider_opened,signal=signals.spider_opened)returnsdefprocess_request(self,request,spider):#Calledforeachrequestthatgoesthroughthedownloader#middleware.#Musteither:#-returnNone:continueprocessingthisrequest#-orreturnaResponseobject#-orreturnaRequestobject#-orraiseIgnoreRequest:process_exception()methodsof#installeddownloadermiddlewarewillbecalledreturnNonedefprocess_response(self,request,response,spider):#Calledwiththeresponsereturnedfromthedownloader.#Musteither;#-returnaResponseobject#-returnaRequestobject#-orraiseIgnoreRequestreturnresponsedefprocess_exception(self,request,exception,spider):#Calledwhenadownloadhandleroraproces