利用开源工具构建小型搜索引擎项目报告.

1、深圳大学考试答题纸（以论文、报告等形式考核专用）二O二O 学年度第学期课程编号1501320002 课程名称 搜索引擎技术主讲教师王旭评分学 号姓名专业年级教师评语：题目:利用开源工具构建小型搜索引擎一、项目操作环境及基本操作工具：操作系统：Win10。基本操作环境：基于 Heritrix+ Lucene ; java。基本操作工具：JDK 1.8 , MyEclipse Professional 2014 , Tomcat 8.0.27, Heritrix 1.14.4 , Lucene 2.1.0, JE-analysis-1.5.3Htmlparser 1.5。基本操作工具基本功能介绍：

2、JDK 1.8:JDK （Java SE Development Kit）包括 Java 开发包和 Java开发工具，是一个写 Java 的applet和应用程序的程序开发环境。它由一个处于操作系统层之上的运行环境还有 开发者编译，调试和运行用Java语言写的applet和应用程序所需的工具组成。MyEclipse Professional 2014 : Eclipse 是一种可扩展的开放源代码IDE。2001 年 11 月，IBM 公司捐出价值4,000万美元的源代码组建了Eclipse联盟，并由该联盟负责这种工具的后续开发。集成开发环境（IDE ）经常将其应用范围限定在“开发、构建和调试”

3、的周期 之中。为了帮助集成开发环境（IDE ）克服目前的局限性，业界厂商合作创建了Eclipse平台。MyEclipse ,是在eclipse基础上加上自己的插件开发而成的功能强大的企业级集 成开发环境。Tomcat 8.0.27: Tom服#器是一个免费的开放源代码的Web应用服务器，它是Apache软件基金会（Apache Software Foundation ）的Jakarta项目中的一个核心项目，由Apache、Sun和其他一些公司及个人共同开发而成。一Tomcat 8支持最新的Servlet 3.1和JSP2.3规范。因为Tomcat技术先进、性能稳定，而且免费，因而深受Java爱

4、好者的喜爱并得到了部分软件开发商的认可，成为目前比较流行的 Web应用服务器。Heritrix 1.14.4:Heritrix 是一个开源、可扩展的 Web爬虫项目。Heritrix设计成严格按照 robots.txt 文件的排除指示和META robots标签。Lucene 2.1.0: LuclbeApache软件基金会jakarta项目组的一个子项目，是一个开放源代码的全文检索引擎工具包，即它不是一个完整的全文检索引擎，而是一个全文检索引擎的架构，提供了完整的查询引擎和索引引擎，部分文本分析引擎（英文与德文两种西方JE-analysis-1.5.3:是一个免费的 Lucene中文分词组件

5、。Htmlparser 1.5:是一个纯的java写的html解析的库，Htmlparser不依赖于其它的java库,Htmlparser主要用于改造或提取html。第1页共23页、该搜索引擎系统的基本构架：in dexs;最后先通过爬虫爬取原始网页并进行网页搜集；然后进行网页预处理；接着建立索引 进行查询服务。三、具体搭建和配置过程及问题分析:（1）基本工具的安装：a、JDK1.8的安装：并配置环境变量：JAVA_HOME:C:Program FilesJavajdk1.8.0_66JRE_HOME:C:Program FilesJavajre1.8.0_66CLASSPATH:%JA VA

6、_HOME%jrelibrt.jar;.;PATH:%JAVA_HOME%jrelibrt.jar;.;b、MyEclipse的安装：运行.exe文件c、Tomcat的安装：解压安装；设置环境变量：CATALINE_HOME:D:big worka n zhua ngapache-tomcat-8.0.30; 设置好之后，测试一下Tomcat。打开Bin文件夹，运行startup.bat;hjfl Tomc-at一 口 X27-Dec20151&:11:54. 940INFOr localhost-startStop-J org. apache, catalina. startu pr Has

7、tConfig. deployDireottyry Deploying web application directory D：big workan zhuangapache-tomcat-8, 0. 30webappsROOT27-Dee-2015 18:11:54.972 INFO Llocalhost-startStop-1J org. apaehe. catalina. startu pL HostConf igL dsployDirectory Deployment o web appl icat ion di rectory D： bi g work, an zhuangXpach

8、e-tomcat-S. 0, 30webappsRQ0T has finished in 32 ms27-Dec-2015 18:11: 54, 985 INFO Elocalhost-startStop-l orB，apache, catalina. startu, p. HostConfig. deployDireetory Deploying 谢eb application directory D:big workan zhilangapachc-1 omca t-8 0. 30webappsscar ch eng inc27Deo2015 18:11： 55* 1 09 INFO I

9、ocal host-starts ton 1apache* jasper* sfFfvIetTldScanner. scanJars At least one JAR was scanned for TLDs yet contained no TLDs”Enab1e debug logging for this logger for a. complete list of JARs that were scan ned but no TLDs were f ound in t hem. Skipping tlnnceded JAR a during scanni ng can i mprovE

10、? startup time and J$P CQmpi lation tim早”27-Deo-2015 18:11:119 INFO E local host-star t Stop-1 ore，apache, ca tsil sthrtup. HostConfig. deployDirectory Deployment of web application directory D:big work an zh Llang apache-tome a t-8. 0. 30webapp35earch-cngine has f iniahed in 134 m 出 27-JcTc-201t) 1

11、8： 11：1 26 INFO rnainl oig. apache, coyote. AbstractProtocoL. start Starting Protooo1Handler http-nio-808027-Dec-2015 18:11: 55, 134 INFO main org. apache, coyote. AbstractProtocol. start St 且rtlng ProtocolHand1er 且jp-nio一$00927-Dcc2015 1:1丄：137 INFO main orgx apache, catai ina、占上日亍七口口亠 C呂tai in日-占

12、Server startup in 1022 ms打开浏览器，输入http:/localhost:8080 (或者 :8080 )，如果看到下图所示的界第2页共23页面,表明Tomcat运行正常：flW _ T成工 怎-.-K JHix-h-t Q bM 苒*严.W-U * - -T Q FZ：馆 JH肝E 壽.林 百*呂kUKh* ICtru UCJa JOFkKl HlpApache Tomcat/8；0.30Hom# O&cufrintAllsn Cannguratlon Examples Wlkl MaOlmig LlsltWWiliiTi frApach

13、e Software Foundation h 11 P ! /7 W W W . a pa e h e . fl r g /Silver SlaEusRecoinrnended Reading:Security CQnsiderBtlons HOVj（-TQ Mana & r A ppi iffffon HO WTO sClumt出inq/g&apion 艮EpliwptiDn HDWTQMandigor AppDeveloper Quick StartTOfnW 寻GtipWnb ApiriitafLiDriIR自 Int白 & 3MkJIDBU DailaSruiTi,rlt Spiwj

14、FiWiCH沽Tomcii Wi牙immManaging TomcatFor security, access to thm m白门郎疳 阳eba嗣l& re&lrided Uaer& are dehnedl InOocumentationTom匚E.D DocunnEintaijonGetting HeipFAQ and Mailin口 1.宙怙H備* Mn |1爭第3页共23页第#页共23页d、其他工具解压即可。(2) Heritrix工程的建立:a、解压 Heritrix-1.14.4 和 Heritrix-1.14.4-src 文件，big wortPubfcWindiview* Oi

15、eDfhrf!corrfdesignjabslibrewjurces-y 匸 ripMsrctargetshapip5projfrrthe 曲 src aKRE 系统库JavaSE-1.7 M引用的库c、配置该工程：1、将 Heritrix-1.14.4 和 Heritrix-1.14.4-src 文件从上述网站中下载。2、 将文件中的 lib 放入工程中， 右键工程-build path-c on figure build path-add external jars , 将lib中所有jar包添加到library中。3、 将文件中的org和st两个文件夹放到工程文件的 src目录下。4、

16、将下的所有文件及其文件夹拖至工程文件下，修改perties中的heritrix.versio n=1.14.4; heritrix.cmdli ne.admi n=admi n:admin ，这是分另 U是版本号和登录所 需要的用户(左)和密码(右)。5、将src下除了 conf和java以外的所有文件夹，拖至工程文件目录下。v 出 Heritrix-1.14.4A 佛 src URE 珈# JavaSE-1.7 引用的库； & articles 凸 conf；& design & jobs & lib-& resources 凸 scripts ;凸 target?

17、0 webapps (& xsd heritrixdmesgog Qi heritrix_out.log6、运行run，成功时会显示版本：H 已ritrix Jaw曰应用程序匚：、mycl ipEEXfai nm rytcani=un javR.jci k7.wi n 32.x86_S4_127：0LLi45binjavi 13:29:18.020 EVENT Started WebApplicationContext/Heritrix Console 13:29:18.371 EVENT 5七arted 5ocketListener on :30E0 13:29:18.37

18、2 EVENT Started org.flion” jEtty”S自Heritrix version: 1.14.47、创建好的Heritrix目录结构如下：第#页共23页HFiilrra-1.14.-4mtfr * !wni$-*F MH呂-即烟也怙峙郃* *mP*出 3 沿| 】big wills Hmilrn l u4.prw-?0l 5l JOfi Hmilra- L1J|.4 ：R*ri 阳jhkMsdw*g*iscnpis9：CLAEF4TH.轴: 那KBWffl?日褂201虫1如 口価d、网页搜集:1 扩展 Heritrix新建一个类,叫 FrontierSchedulerFo

19、rScu ,派生于 org.archive.crawler.postprocessor.第5页共23页Fron tierScheduler，它的代码如下：package org.arthix.匸rawler.postp。“酊。!； import org,archive,crawler.datamodel.CandidateURI;public class FrontierSchedjlerForSzu extends FrontierScheduler private static final long seriaLVersionUID = -122468389435661S296L;publ

20、ic FrontierScbedulerForSzu(String nme) super(nrne);protected void schedule(CndidateURI callri) String uri二“Ui if(uri r indexOf(dns;H!=-l)getController(. getF rQrrtiurO.wuhudiilCuBUri)： else if(uri.indexOFCsaj)l-l&(uiri. indexOFf. html) I 1 uri. indexOf.htm)J=-l uri.indexOfCl)1=-1 ur i. indexOf (u -

21、doc11) (-1 uri,indexOf(.xls)I-1) if(uri,indexOf(=)1=-l) uri=uri.replaceAll(,? getControllerO.getFrontierO.scheduletcaUri);屏蔽zip、rar、exe文件，只抓取网页，并且抓取的网页中必须包含scu(即四川大学)打开文件 con fmodulesProcessor.optio ns添加新建的module选项：org.archive.crawler.postprocessor.Fr on tierSchedulerForSzu|Fr on tierSchedulerForScu

22、: or自.a广亡hive.t广awl亡r.writer.Mirrd广MriterPro亡亡r|Mi广广。广Zrit亡r org archive.crawler.postprocessor.CrmbdEtmteUpdat己r|Updater org.archive亠 crawler.postprocescr * L in ksScoper|LinksScoper orgarchive c rawler postprocessor. Supplementary Link sSccper |Supplementa ry Lin lew 5c ope r crg.archive.-crawler.p

23、ostprtce5scr .FrontierStheduler |FrcntierScheduler orgnchivf x rawler. pos tprH：ESEo r.吕eu| FrorrtiwrthmdulmrForEtul o re，arc hi vex rawler .post processor. LowDiskPau 5 e Processor | LowD is kPa use crE.arehive.crawler.postprocessor.WaitEvaluator|NaitEvaluator取消 robots.txt 的限制：找至U org.archive.crawl

24、er.prefetch 中的 PreconditionEnforcer 类: 找到considerRobotsPreconditions函数，将该函数内的全部代码注释掉或者直接删除 添上 return false:*zprivate boolconsiderRobotsPreconditions(CrawlURI curi) / treat /robots七*七 fetches speciaillyUURI uur.i 三 cui-getUURI()itry ) / cl low processIniE； to ccntipuc curi. &et Pre requisite (true);

25、return false;毗爭 tch ( UR I IE k ceite) logger. severe Failed get of path for * curj/ require /robots.txt if not presentif ( i sRobot s Expi red C mi) iff(Boolean)getUnchcckedAttribute(cyr.i,ATTR_(ALCUiJlTEJftOeOTS_ONL).booleanValue() = true / ainncitate UR.I as excludedj but continue to process- nor

26、mallycuri addAnnotation(RrobotExcluded);rettirn false;/ mark as pretluded; in FetchHTTP. this will/ prevent fetching and cause a skip to the end/ of proc-essing (unless an intervening processorretijrn false;H mark as precluded; in FetchHTTP this 计订1 / prevent fetchine and cause d skip to the end/ of

27、 processing (unless an interv亡nin百 prqc亡mmcjr/ overru丄亡w)curi. setFetch5ttLi(5_Fl00TE_PRECILUDED);curi.putString(errorjrcbots.txt exclusion);logger.fine(Hrobots.txt precluded +return true;return false;/ No valid robots ftxjnd = Attempt to et robots,txt failed curi.skipToProcessorChainCgetController(

28、).get Pos t proces s a)curl. setFetchStatLi5(S_Ft0B0TS_PREREQllISITE_FAlLURE) j curi.piitStringf亡rrarrobotstxt prerequisite failed);if (logger,i&LoggableLevel.FINE) logger .fine (robot s, txt prerequisite failed M + 驱1);return tru;*freturn false;f、网页抓取:1、登录 Heritrix:运行 Tomcat 后登录 http:/localhost:808

29、0 (或者 :8080)孑60安全浏览器B.1 | C O 0 http /local host8080/l og i n.js p;jsession id=2 byjxl4 q 88pa* 手机收蔵夹 G 稣 总网址丈全 d好搜 芦游盼心 Links 廉圳大学 园鉗）血 O 口跨屏浏崟 因 Heritrix: Loginx +LoginUsername:伽自Password: Login2、Heritrix的控制台如下:O Qr科阳呱心iMmwksum?眄 of I 二月.2L 2015 13：翻；】o rkl U血LEHHG JOfiAdiin CodecHp

30、 0 jobs 妙ndi谒 5 旦邮2遇_ Jubs Pir3il 自#Lu 呂嚣Re|kIs LiJp Hdp-Crawler Status: MOLDING JOBS I Star!Alerts; tq alertsCuniaol bJobsKone running0 pending, 5 coinpletdAlerts: 0 Id 血诃)Memory11547 KB ustd6U4G KB current hmp90S28B KB max h*ap；拉仃 doiTj l【Eritix 吕ofTwag-RLqdutdentifirr; wgr archive cjraaer, Writri

31、s3、设置seed:设置为四川大学的 URL第7页共23页ConsoleJobs卩rofilesLogs ReportsSetup HelpCreate rgw crawl job based on job sichuan-universityJName of new job:Description:Seeds:sichuan-universitysi chuan-uni verity,Fill in gee日匚RTm b已 1用 one pef 1 ig. Ccmti年rut : ines begin 时 th http:/, wwwm scu- edu. cnModules Submodu

32、les Settings Overrides Submit jab4、点击Modules:设置各个处理模块：Select Crawl Scope 选择 org.archive.crawler.scope.BroadScope：Select Crawl ScopeCurrent selection:archive, crawler, scope, BroadScopdBroadScope: A scope for broad crawls. Crawls made ith this scope will not be limited to lhe hosts or domains of iis

33、seeds. NOTE: BioadScoped crawls will eventually run out of seniorv (See Release srarAo res).Available alternatives: org.archive.crawler,scope.BroadScope* ChangeSelect Writers 选择 org.archive.crawler.writer.MirrorWriterProcessor 并将另一个移除：然后抓取网页后会生成镜像文件便于查看：Select Writers Processors that write documents

34、 to archive filesot氐 0?chiE+writgr” Mirror艸Rejtioye Infoorg.archives rawler. pre fete h PreselectorAddSelect Post Processors 第三项修改为 org.archive.crawler.postprocessor.Fr on tierSchedulerForScuSelect Post Processors Processors ihar do 芒 1 eanup and f&d ih& Frontier with /i&.org. archive, crawler, post

35、processor. CrawlStareUpdaterDown Rqukhq Infoorg archive, crawler. postprocesscrr LinksScoper紅rchivw. ciEi常PDsLprocessor* FronliErSchmdulEi-Fcn-Sgiil尚Do笳nInfoRemove rnfoAddorg.anchive.crawler.prefetch Preselector其他设置如图:第#页共23页Select URI FrontierCurrent selection:org archive, crawler frontier. Bdbl ro

36、ntierBdbFronrier. A Frontier using BerkeleyDB Java Edi lion databases for persistence to disk.Available alternatives: org.archive.crawler.frontier.BdbFrontier ChangeSelect. Pre Processors Processors thm should run before unv fe t chi ngorg. archive, crawler, prefetch. PreselectorRemseorg. archive, c

37、rawler, prefetch, PreconditionEnforcer氓 Infoong .arch ive.c rawler. fete her.F etchDN SAddusing tarious protocolsDown 陰iiiAF InfoRemoye InfoSelect Fctchcrs Processors that fetch documentsAddorg. archive, crawler, fetcher, FerchDNS org, archive, crawler, fetcher. FetchHTTPorg .archive.c rawl er. pref

38、etch. P reselec torDgn Remove Info 血叱芒Tnfo Doi订i Removm Info Down Reihove InfoRemove InfoSelect Extractors Processors that extracts links from UR Isorg. archive, crawler, extractor. ExtractorHTTP org, archive, crawler, extractor. ExrractorHTML org. archive, crawler, extractor, ExtractorCSS or咅.archi

39、ve, crawler, extractor. ExtractorJS org. archiv e. crawler, extractor. Ex tract or SWForg archive crawler.prefetch.PreselectorSelect Statistics Trackingorg. archive, craw 1ei admin* StatisticsTracker Reitkne Info5、参数设置：点击Setti ngs,这里只需要修改 user-age nt和from : VERSION表示版本，后面的 URL 输入email。http-headers?H

40、TTP headersuser_agent:?Mdzil5.0 (ddmpatiblB； h&ritriKj1.14. +httpJAvww. ecu. Bdu.tiV)from：?1156276H35robots-hcnor ing_pal icy?Robots honor inc policy设置好之后，点击 Submit Job，返回Con sole界面，刷新一下(start和refresh)，就开始 运 行：如图可得到一下的进度条：JobsProfilewLog 呂 Reports SetupCrawler Status: CRAWLING JOBSJobsRunning： sichu

41、an-universiry 0 pending, 5 completedHoLdMemory23483 KB used&L440 KB current heap908288 KB max heap厂Job Status: RiwiNGPauseCMckMintTgnninatgRatesLoad0 UBIs/sec (0 avg)0 KB/sec (0 avg)D active of 30 threads1 congestion rat ioTimeelapsedlffl3s remaining (estimated)33 deepest queue S average depthTotals

42、downloaded 856 queued64 total downloaded and queued18 KB crawled (18 KB noveL)Refmh左边表示已经下载了得数目，右边表示在队列中等待下载的数目。6、最终抓取的到的结果在工作区间的 Heritrix下的jobs文件夹中:-liefii,ix_LuC!n-S)1512DS HeriVii-1.14.4 jobi yclWudn-uHive rYlirfor3Q2.115-M 20idowfflHSarchmrM.&cui.edlLLC门xmib5；&c ULedM.cn20211222.1.111

43、30222.1B15J35理Qiiwt 乩 LL&dlKtnatem.wiAccfip.cuuxnCea5iJLdu8Cfl林chenYMusedu.cA1 ert s: 0 O n歸共抓取约13万个文件。(3) 进行网页预处理：a、添加 htmlparser解析网页：将下载的htmlparser相关的压缩包解压，找到htmlparser.jar文件，将至导入library中。3砲9项目(P)咏库(L)齢排序和导出(O)构建路径上的JAR和类文件夹(T):恳 htmlparser.jar - Pre proc ess/libb、添加解析网页文件：Page.java代码如下:第9页共

44、23页D Preconditio.D FrontieiScheQuota EnforceindexBuildepackage cn.edj.szu,search.paee；Ipublic class Page private String urlj private ring title; private Strine summary; private String context；private int scorespublic Page() url = null; title - null;uirmary = null context - null; score =public Stirin

45、g getLrlO return this.url;public void setUrl(String url) this.url = url;public String getTitle() return this.title;public void 5etTitle(String title) this.title = titie;public String getSummary() return this,summary;public void setSummary(String summary) this-summary = summary;public String getConte

46、xt( return this.context;public void setContext(St ring context) this-context = context; publi亡 int getScore() return this.score;public void setScorefint score) this.score = score;对 Cperties 进行配置：namevdlueminror.pathD:big workWenitrixucene-ZOI 512G8Herkiri)c-1!14-4job5isichuan-jniver5iiy5122

47、-files.pathD:big wcrkIHeritriK Lucene-201 51 ZDBXPreprocessXfilesiridex.pithiD:big workHeritriK_Lucent 201512DSPreprotei5iindi&x4mirror.path是得到的网页文件的地址：D:bigworkHeritrix_Luce ne-20151208Heritrix-1.14.4jobssichua n-u ni versity-20151223130135337files.path是将网页文件解析后存放的文件地址；第11页共23页Index.path是网页解析文件建立得到

48、的索引的存放地址。d、添加文件 Extractor.java代码如下：package cn. edu.szu.search.extractor;import org.htmlparser.*;import org.htmlparser.util.*;import org.htmlparser.visitors.*;import org.htmlparser. no des.*;import org.htmlparser.tags.*;import cn. edu.szu.search.page.*;import cn. edu.szu.search.util.*;public class Ex

49、tractor impleme nts Runn ableprivate String file name;private Parser parser;private Page page;private String encode;public void setE ncode(Stri ng en code) this.e ncode = en code;private String comb in eNodeText(Node no des) Stri ngBuffer buffer = new Stri ngBuffer();for (int i = 0; i no des .len gth; i+) Node anode = (Node )no desi;String line = n ull;if (anode in sta