ElasticSearch与Solr搜索引擎特性对比-new

ElasticSearch与Solr搜索引擎特性对比1、关键特性对比对比项Solr5.xElasticSearch2.x开发人员开发人员少，官方提供功能比ElasticSearch多开发人员多，官方功能少，但是第三方插件很丰富，扩展能力更强建立索引和查询效率建立索引的速度和ElasticSearch差不多，当索引完全建立好了之后搜索会很快，但是边建立索引边搜索就会非常慢。经过详细对比测试后发现，Solr在建索引时的查询性能非常之差，因为Solr在建索引时会产生io的阻塞，造成搜索性能的下降。建立索引的速度和Solr差不多，第一次搜索速度比Solr慢些，但之后会缓存结果，第一次搜索就会很快。另外ElasticSearch边建立索引边搜索速度没有太大变化，因为它是先把索引的内容保存到内存之中，当内存不够时再把索引持久化到硬盘中，冋时它还有一个队列，是在系统空闲时自动把索引写到硬盘中。Sharding没有自动shardrebalancing的功能shard必须一次设置好，之后不能修改，如果要修改则需要重新建立索引Schema模式需要Schema（但是4.3版本之后有schemaless模式）无需Schema，创建索引更加灵活高级查询没有QueryDSL有QueryDSL，能够支持更加高级和复杂的查询语法，而且还可以以此扩展实现类SQL语法的查询实时性传统搜索应用实时搜索应用（这个是ElasticSearch作者说的）可扩展性不支持插件开发模式支持插件开发模式，提供丰富的开源插件库社区社区活跃社区活跃并有大数据公司支持Hadoop集成支持支持，v2.3.0提供Graph图结构数据存储存储Solr暂不支持ElasticSearch支持多盘存储2、基于ElasticSearch数据交叉模型功能测试2.1、环境情况1•测试环境（市局环境）IP43/slave1~77/slave35CPU16*Intel(R)Xeon(R)CPUE5-2650v2@2.60GHz

OSCentOSrelease6.5(Final)OS2.测试数据数据来源省厅数据：VWSHARESTJJKKXX(交警—卡口信息)时间范围20151001~20160331数据量kettle清洗，数据量为11.1亿测试数据查询地址43:8200/plugin/head/3.测试地址url41:18099/swagger-ui.html2.2、测试结果一个车牌交叉满足N个条件场景:序号交叉条件个数测试用例预期结果实际结果3次测试耗时(s)备注13[{"datestart":"2016-01-05T08:40:59","location":"犍为县"，"dateend":〃2016-01-05T08:53:59〃，"querystr":"犍为外滩"},{"datestart":"2016-02-0309:48:00","location":"乐山市","dateend":"2016-02-0319:48:00","querystr":"月卡口"},{"datestart":"2016-01-0719:47:18","location":"南充市"，"dateend":"2016-01-0719:57:18","querystr":"绿心路"}]川LFC5503川LFC550310在之前的测试时，曾出现结果为川LFC5502的情况，经定位是对“犍为县”的分词不完善，修改代码后，结果正确2223[{"datestart":"2016-01-15T07:08:00","location":"成华区"，"dateend":"2016-01-15T07:18:22","querystr":"双桥子"},{"datestart":"2016-01-05T06:39:44","location":"锦江区"，"dateend":"2016-01-05T06:41:44","querystr":"老成仁路口"},{"datestart":川AW7D163川AW7D1631在之前测试中，曾出现结果为川AW7D160的情况，经定位，是分词不够完善，修改代码后，结果正确1111〃2016-01-25T20:40:46〃，"location":"武侯区"，"dateend":〃2016-01-25T20:44:46〃，"querystr〃：〃科华北路〃}][{"datestart":"2016-02-06T17:48:41","location":"四川省"，"dateend":"2016-02-06T17:49:41","querystr":"海棠路"},{"datestart":"2016-02-04T10:36:43","location":"富顺县"，"dateend":"2016-02-04T10:36:44","querystr":"钟秀街东段"},{"datestart":"2016-02-04T13:10:50","location":"富顺县"，"dateend":"2016-02-04T13:10:53","querystr":"吉安庄路"}][{"datestart":"2016-02-16T15:06:57","location":"泸定县"，"dateend":"2016-02-16T15:06:57","querystr":"石油路"},{"datestart":"2015-12-09T16:10:58","location":"新龙县"，"dateend":"2015-12-09T16:10:58","querystr":"卡口抓拍机"},{"datestart":"2016-01-07T10:46:17","location":"泸定县"，"dateend":"2016-01-07T10:46:17","querystr":"船头"},{"datestart":"2016-02-16T18:10:20","location":"雅安市"，"dateend":"2016-02-16T18:10:20","querystr":"名山红星"}][{"datestart":"2015-ll-05T13:39:46","location":"内江市"，"dateend":"2015-ll-05T13:39:46",〃」〃〃aa4-4-〃iquerystr:二桂},{"datestart":"2015-n-21Tll:24:24",贵JL73213贵JL73213川AE15F84川AE15F84川K0103学2川K0103学2111111""location"":""内江市""，""dateend"":〃2O15T1-21T11:24:24〃，""querystr"":""牌楼路""}][{""datestart"":〃2015-12-01T15:06:39"",""location"":""广安市华蓥市""，"dateend"":""2015-12-01T15:06:39"","querystr"":""古桥职中""},{""datestart"":""2015-09-30T15:13:34"",""location"":""广安市""，""dateend"":""2015-09-30T15:16:34"","querystr"":""会展街""},{""datestart"":""2016-02-12T12:15:10"",渝B18D025渝B18D025""location"":""渝B18D025渝B18D025""2016-02-12T12:15:10"",""querystr"":""高速""},{""datestart"":""2016-02-10T08：n：49"",""location"":""广安市""，""dateend"":""2016-02-10T08：n：59"","querystr"":""前华段出城""},{""datestart"":""2016-02-05T12:36:06"",""location"":""广安市""，""dateend"":""2016-02-05T12:39:06"",""querystr"":""马石梯""}][{""datestart"":""2016-01-01T22:06:37"",""location"":""青羊区""，""dateend"":""2016-01-01T22:06:37"",""querystr"":""苏坡立交""},{""datestart"":""2016-01-22T21:17:18"",""location"":""武侯区""，""dateend"":""2016-01-22T21:17:18"","querystr"":""人民南路三段""},{""datestart"":""2016-02-14T00:01:26"",""location"":""金牛区""，""dateend"":""2016-02-14T00:01:26"",""querystr"":""营门口""},{""datestart"":""2016-02-08Tn：47:08"",""location"":""成华区""，""dateend"":川ATS9586川ATS958620〃2016-02-08Tn：47:08〃，"querystr":"桃溪路"},{"datestart":〃2015-n-02T15:00:22〃，"location":"锦江区"，"dateend":"2015Tl-02T15:00:22","querystr":"琉璃"},{"datestart":"2015T0T3T09:34:12","location":"武侯区"，"dateend":"2015T0T3T09:34:12","querystr":"双楠路内侧"}][{"datestart":"2016-01-01T22:06:37","location":"市青羊"，"dateend":"2016-01-01T22:06:37","querystr":"苏坡立交"},{"datestart":"2016-01-22T21:17:18","location":"武侯区"，"dateend":"2016-01-22T21:17:18","querystr":"人民南路三段"},{"datestart":"2016-02-14T00:01:26","location":"金牛区"，"dateend":"2016-02-14T00:01:26","querystr":"营门口"},{"datestart":"2016-02-08Tll:47:08","location":"成华区"，"dateend":"2016-02-08Tll:47:08","querystr":"桃溪路"},{"datestart":"2015Tl-02T15:00:22","location":"锦江区"，"dateend":"2015Tl-02T15:00:22","querystr":"琉璃"},{"datestart":"2015T0T3T09:34:12","location":"武侯区"，"dateend":"2015T0T3T09:34:12","querystr":"双楠路内侧"}]川ATS9585川ATS9585[{"datestart":"2016-01-05T08:40:59","location":"犍为县"，"dateend":"2016-01-05T08:53:59",川LFC5501川LFC5501—1""querystr":"犍为外滩""}]场景二：关注时间和地点，多个车牌满足其中1~N个条件序号交叉条件个数测试用例预期结果实际结果3次测试耗时(s)备注13[{""datestart":〃2016-02-07T16:44:10〃，""location"":""广安市""，""dateend"":〃2016-02-07T16:44:20〃，"querystr":""人民南路"},{"datestart"":〃2016-02-10Tll:08:08〃，""location"":""广安市""，""dateend"":〃2016-02-10Tll:09:09〃，""querystr"":""民族小学""},{"datestart"":""2016-01T4T19:53:10"",""location"":""武胜县""，""dateend"":""2016-01T4T19:53:12"",""querystr"":""58KM""}]粤YQJ2212川XW62382粤YQJ2212川XW6238184924[{"datestart"":""2016-01-01T00:00:00"",""location"":""德阳市""，""dateend"":""2016-01-04T00:00:00"",""querystr"":""西山""},{"datestart"":""2016-02-22T08:10:55"",""location"":""德阳市""，""dateend"":""2016-02-22T08:10:55"",""querystr"":""108国道""},{"datestart"":"2015-11-16T06:17:58",""location"":""罗江县""，""dateend"":"2015-11-16T06:17:58",""querystr"":""学校""},{"datestart"":""2016-01-03T16:53:16"",""location"":""绵阳市""，""dateend"":""2016-01-03T16:53:16"",""querystr"":""东路""}]川FAH5672川FL19302川A9EB552川FAH5672川FL19302川A9EB5523163、ElasticSearch与Solr的压力测试对比3.1、测试环境1■集群环境此次用于测试Solr的集群包括4个节点：0(master),1~3(slave1~slave3)。用于测试ElasticSearch的集群包括4个节点：0(datanode,不可选举为masternode),1~3(datanode、可选举为masternode)。测试集群信息：IP0~3CPU8*Intel(R)Xeon(R)CPUE5-26090@2.40GHzMemSolr：4G,ElasticSearch：4G(sys32G)OSCentOSrelease6.5测试客户机（临时使用slave3）：IP3CPU8*Intel(R)Xeon(R)CPUE5-26090@2.40GHzMemSolr4GElasticSearch4G(sys32G)OSCentOSrelease6.52•数据环境Solr中单索引库中索引条数为5000W,ElasticSearch中单索引库中索引条数为1亿。3.2、测试工具工具名称http_load输入参数说明http_load-p5-f1000train_hc.txt-parallel简写-p：并发的用户进程数-fetches简写-f：总计的访问次数-rate简写-r:每秒的访问频率-seconds简写-s：总计的访问时间输出参1000fetches,5maxparallel,1.13104e+06bytes,in4.50961seconds运行的数说明请求数、最大的并发进程数、总计传输的数据、运行的时间1131.04meanbytes/connection平均每个请求连接传输的数据量221.748fetches/sec,250807bytes/sec每秒响应请求数、每秒传输数据量msecs/connect:0.485064mean,1.337max,0.384min连接平均响应时间、最大响应时间、最小响应时间msecs/first-response:21.7613mean,436.541max,13.468min连接平均返回时间321badbytecountsHTTPresponsecodes:请求响应码返回个数code200—10003.3、测试结果注：每组测试均重复进行三次。场景一：模糊数字査询查询条件："q":"31997"统计测试结果如下:并发数总访问次数搜索引擎平均每秒响应请求数(fetches/s)总消耗时间(s)连接平均响应时间(ms)连接平均返回时间(ms)平均成功次数平均失败次数1010000Solr239.66441.7250.16340.467100000ElasticSearch552.47118.1010.13017.6411000002510000Solr530.81918.8390.77744.675100000ElasticSearch677.38514.7630.12836.4281000005010000Solr664.68015.0450.20573.096100000ElasticSearch686.34014.5700.13072.28710000010010000Solr718.75113.9130.243136.493100000ElasticSearch699.83814.2890.440141.77310000020010000Solr631.86915.8260.289310.823100000ElasticSearch632.78915.8034.370310.03010000050010000Solr38.925256.9081.9711848.6906479353

ElasticSearch680.43214.6974.941716.01010000080010000Solr166.50360.0592.433278.3741139887ElasticSearch521.34919.18139.6641425.5809782218100010000Solr166.12360.1967.802386.7981489852ElasticSearch692.36114.44381.5321276.0009848152120010000Solr41.609240.33192.813407.1501019899ElasticSearch164.70560.715187.910841.63715528448场景二：精确中文查询查询条件："q"："人民南路"统计测试结果如下：并发数总访问次数搜索引擎平均每秒响应请求数(fetches/s)总消耗时间(s)连接平均响应时间(ms)连接平均返回时间(ms)平均成功次数平均失败次数1010000Solr274.66536.4080.15935.091100000ElasticSearch606.13516.4980.13316.0401000002510000Solr496.52420.1400.17348.379100000ElasticSearch832.06612.0180.12229.5811000005010000Solr528.96718.9050.39291.614100000ElasticSearch822.15112.1630.13260.28210000010010000Solr499.61420.0160.313196.558100000ElasticSearch848.29911.7881.236115.98910000020010000Solr463.91021.5560.358422.760100000ElasticSearch785.50912.7312.071249.89810000050010000Solr155.62664.25610.400183.3117989202ElasticSearch827.33112.0879.174576.56710000080010000Solr166.10660.2037.612343.5382209780ElasticSearch814.86012.27230.071912.044100000100010000Solr166.01860.2342.354451.4892289772ElasticSearch867.41811.52946.7301026.2909859141120010000Solr165.70260.35079.148428.1622789722ElasticSearch165.51660.417185.596734.681159984013.4、总结◊在本次测试中，每次测试总的访问次数为10000,从以上图表中可以看出随着并发数的增加，Solr和ElasticSearch的每秒响应请求数都是先增加，然后稳定在一定数值，在超过某个阈值之后极速下降；从平均每秒响应请求次数来看，ElasticSearch性能指标优于Solr；在不同的并发数下，ElasticSearch的每秒响应请求数在500~800之间，而Solr的每秒响应请求数在300~600之间，ElasticSearch采用默认分词进行中文模糊搜索时速度非常慢，经分析更换ik中文分词后性能有明显提高；◊从连接平均返回时间来看，在低并发时，ElasticSearch和Solr相差不大，但是随着并发数量的增加，ElasticSearch返回时间的增长速度大于Solr；◊从不同的并发数下的成功次数来看，ElasticSearch性能指标优于Solr，Solr在并发数达到500的时候出现大面积错误；ElasticSearch直到并发数为1000的时候才出现了错误；Solr集群不太稳定，12月30日简单测试过一次，当时并发达到1000时才开始出现性能大幅下降，并且平均每秒响应请求数在600左右，而31日则在并发500时表现有较大差距。4、ElasticS