离线计算使用说明

1、离线计算使用说明版本说明版本jar名称特点spark-offline-1.0-SNAPSHOT-jar-with-scala 代码编写，自由度高,不dependencies.jar能配置基于 xml可配置通用计算实现支持离线计算实时计算可配 置版本1部署配置说明修改resourceView-server（资源库后台程序）中的配置文件配置chart库修改jar中配置文件配置表修改jar中mapping目录下的配置文件，如果默认那么不需要修改功能对应名称功能对应名称目录备注计算版 本资源库-电子病历库dataset_count医疗数据集,存chart.his_count 表资源库-健康档案库dat

2、aset_count公卫数据集，存chart.his_count 表资源库-家庭成员关 系family_member存 chart.family_member 表资源库人员统计 资源库-用药统计 综管统计people_count clinic_count hems存 chart.people_count 表 存 chart.clinic_count 表 shanghai 负责221212启动命令cd到相应计算目录中执行以下命令sh submit.sh与资源库联动Agawwmt =【奈 epi abm4MMaaBwaiWMiz，M；MaMaffiajggfiteCTarfTOM8g.皿 toit

3、a.waMmaammM. 如下图有启动，禁用，设置定时执行的功能资源库服务配置资源库后台程序中配置文件perties,配置此文件才能界面上远程执行 spark中的离线及计算resourceView-server/WEB-INF/config/spring/perties# ip地址对应的是部署了离线计算脚本的主机quartz.ssh.port=22quartz.ssh.username=rootquartz.ssh.password=passwordquartz.ssh.basedir=/usr/src/wl/spark参数配置离线脚本所在的目录，各工程组目前目录并不统一资源库后台数据库配置三

4、 f Begin Transaction 国 Memo , Filter 卜 Sort jh Import % Exportjob.idupdate_time1 2021-07-20 20:57:582 2021-07-20 20:58X）73 2021-07-20 20:58204 2021-07-20 2058:265 2020-08-17 16:29:197 2020-08-17 16:29:16job.status cron_expres$ion descriptionbean_da$PeopleCount1048 14 , ?It诊人口312让runClinicCount10011

5、* * ?ftBETlEWttrfrunFamilyMember10 40 14 * * ?runDataSetCount1055 14 , ?Q心座836runPeopleManagement1055 14*?VUQASSjSStvf-run Medkallnstitution1055 14 , ?115床（2汁runMedicdlEquipment10 55 14 , ?runchart.quartzjob_details表中没有数据那么需耍导入配置数据，不然界面上无法操作，数 据文件是 chart .quartz_job_details.sqlsql优化全量计算太慢要2个小时，添加条件只

6、统计7天内的数据 spark-sqlselect date_add(now(),-7);select concat(date_add(now()-7)/ 00:00:001);脚本中可添加如下条件 where hbaseprocessflag = 11 and AUTHORORGANIZATION is not null and upl oadtime concat(date_add(now()-7) 00:00:001)日志查看执行 sh submit.sh 后/NameNodeHA/user/hdfs/.sparkStaging/appHeation_16367O6679749_OO18/

7、spark_conf.zip22/01/0622/01/0622/01/0622/01/0622/01/0611:11:3911:11:39 11:11:39 11:11:3911:11:39INFO SecuntyManager：INFO SecurityManager：INFO SecurityManager：INFO SecurityManager：INFO SecurityManager：Changing view acts to： hdfsChanging modify acls to： hdfsChanging view acls groups to：Changing modify

8、 acls groups to：SecurityManager： authentication disabled； ui acts disabled； usersith view permissions： Set()； users with modify permissions： Set(hdfs)； groups with modify permissions： Set()with view permissions： Set(hdfs); groups w22/01/0622/01/0622/01/0622/01/0611:11:39 INFO Client：11:11:39 INFO Ya

9、rnClientlmpl： ACCEPTED) 11:11:40 INFO Client：client token： N/Adiagnostics： AM container is launched, waiting for AM container to Register with RMApplicationMaster host: N/AApplicationMaster RPC port： -1queue： defaultstart time：final status： UNDEFINEDtracking URL: :/node2:8088/proxy/application_16367

10、O6679749_0O18/ user： hdfs22/01/0622/01/0622/01/0622/01/0622/01/0622/01/0611:11:4111:11:4211:11:4311:11:4411:11:4511:11:45INFO Client:INFO Client：INFO Client：INFO Client:INFO Client:INFO Client:ApplicationApplicationApplicationApplicationApplicationreport for application_16367O6679749_OO18report for

11、application_16367O6679749_OO18(state： ACCEPTED)(state： ACCEPTED)(state： ACCEPTED)(state： ACCEPTED)(state： RUNNING)client token： N/A其中state说明状态，有如下三种(state: ACCEPTED)申请资源状态，如果一直是这个状态那么标识资源不够或者 资源规划不合理(state: RUNNING)运行状态(state: FINISHED)结束状态命令行查看日志找到日志所在目录667974920019 (state：ACCEPTED)22/01/0614:10:23

12、 INFO Client:Application reportfor application二二 0019 (state：ACCEPTED)22/01/0614:10:24 INFO Client:Application report：ACCEPTED)22/01/0614:19:25 INFO Client:Application reportfor application110019 (state：RUNNING)22/01/0614:10:25 INFO Client:可看到应用产生的临时日志rootnode2 log# find ./application_16367O6679749_

13、OO19./application110019/container二101二。二1 0019/container二 e75116367e66797491 0019二。匚。./application_16367O6679749_OO19/container_e75_16367O6679749_O019_Ol_O0O0O5/stdout./叩plication_16367O6679749_OO19/container_e75_16367e6679749_O019_Ol_O0OOO5/stderr使用一下命令查看实时日志或者页面查看日志含 Services YARN SummarySUMMARY H

14、EATMAPS CONFIGS METRICSSummaryAO Quick LinksComponentsO StartedTIMELINESfWVICEVI 5O StartedYARN REGISTRY DNS3/3 StartedNOOEM AN AGERS54dl8h 37mRESOURCEMANAGER UPTIMENOOEMANAGERS STATUS 3O StartedSTANDBYRESOURCEMANAGCRO StartedACTIVERESOURCEMANAGERO StartedTIMRINE SERVICE V2 0READERResxrceMinjgef JMX

15、4 InstalledYARN CUEMTSResxrceManagef logs ReMXMCtKUmQtf jmx Thread Stacks点击连接跳转到 :/node2:8088/ui2/#/cluster-overview这个是新版2代UI,查看日志使用老页面就可以了，访问地址 :node2:8088O 8 node2：8088/dusterso% aAll Applications ClusterAbout Nodes Node Labels ApplicationsNEWNEW SAVING SUBMITTED AUCWPTED RUMNIMG WISHED FAILED KIL

16、LEDSchedulerCluster Metrics内存资源已使用总内存资源, IdolsApps SubmittedApps PendingApps RunningApps CompletedContainers RunningLi 600160Cluster Nodes MetricsActive NodesIsScheduler MetricsDecommissioning NodesDecommissioned NodesScheduler TypeCapacity Scheduler Show 20 v entriesScheduling Resource Typememory-m

17、b (unit=Mi), vcocesMemory Used OBLost NodesQMinimum AllocationMemory Total 27 GBMemory Reserved )BUnhealthy NodesMaximum Allocationmemory9216z vID,U 力Name0Applicationa Application StartTimeType ”叫的 5FinishTime* FinalStatusRunning ContainersAllocated CPUVCores *Allocated MemoryMB 0%VaQHdtion 16367066

18、79749 0018标鼠hdfs conxbsoftspark.coce.sql.CalculateEntry 名称启动时可配置SPARKdefault 0Ibu Jan 611:11:39 .0800 2022Thu Jan 6 11:11:56 .0600 2022RNISHED SUCCEEDEDN/AN/AN/AN,a。Mb conxbsoftoffline.$cAla.GxjntDaUSet2SPARKdefault 0Wed Mn5 17:31:16 .0800 2022Wed Jan 5 17:32:58 .0600 2022ANISHED SUCCEEDEDN/AN/AN/AN

19、,applkation 16S6706679749 0015SPARKdefault 0Wed JanWed Jan 5FINISHED SUCCEEDEDN/AN/AN/AN,5 17:22:54 0800 20221724:59 406002022apSPARKdefault 0TueDec14 19:4106 0800Tue Dec 14 19421 .0600RNISHED SUCCEEDEDN/A N/AN,V点击标识进入到日志界面80% AboutNode，Node Labels ApplicationsHEW“EW SAVING SUBMITTED-epte6RUNNING FI

20、NISHED FAILED KILLEDScheduler ToolsUser:Name:Application Type: Application Tgs: Application Priority:YamApplicationState: Quuc: FinaiStatus Reported by AM: Started: Elapsed: Tracking URL: Log Aggregation Status: Application Timeout (Remaining Time): Diagnostics: Unmanaged AppGcstion: Application Nod

21、e Label expression:hdfc0 (Higher Integer value indicates higher priority) FINISHEDdekubSUCCEEDED星期四一月 06 11:11:39 40800 202217$ecHistorySUCCEEDEDUnlimitedfalseAM container Node Ubd expression: Total Resource Preempted:Total Number of Non-AM Containers Preempted: Total Number of AM Containers Preempt

22、ed: Resource Preempted from Current Attempt Number of Non-AM Containers Preempted from Current Attempt Aggregate Resource Allocation: Aggregate Preempted Resource Allocation:000217546 MB-seconds. 61 vcocc-scconds0 MB-seconds, 0 vcore-secondsShow 20 * entriesAttempt ID, Started ，appattempt Thu Jan 6

23、11:11:39 +0800Nodes blacklisted by the appShoving 1 to 1 of 1 entries点击here可查看详细日志so% Application“ ToolsConfiguration Local logs Server 4配匕 Server metricsLog Type: Log Upload Time:星期四一月 06 11:11:58 0800 2022Log Length: 42535Showing 4096 bytes of 42535 total. Click IffS foe the full log.ywnU41W3162 1

24、141903163 1141903164 1141903168 1141903166 1141903167 1141903168 1141903169 1107749863 1107749866 542083723hadoop84 drwxr-xr-xApplication MetricsNodes blacklisted by the system543807406116 r108 n1464 n3884 n3812 n537415298 4600537415297 64537415296 58001842428 40068 6440 14876441 yarn1 yarnhadoop ha

25、doop25 If199 If弓6 11:11 . /二co/_hadoop con/slaves习6 11:11 . /spr):con_hadoop_con/topologymappings. datA1 yamhadoop1044 If习 6 11:11 . /_spwk_conf / hadoop conf_/ssl-srTr. xnl1 yumhadoop961 If习 6 11 11 ./_印a4_con / hftdoop conf /nftprd-onv. end1 yamhadoop2697 IfH 6 11 11 . f_印4r，_con / hftdoop con_/ss

26、l-srrr. xnl. axanpl1 yamhftdoop945 IfH 6 11 11 . /_sp4rk_conf / had。 conf_/tskcontrollr. c1 y4rnhadoop682 IfH 6 11 11 ./_spwk_con_/_hftdoop_conf_/ls-si te xil1 yarnhdoop4113 If口 6 il 11 . /_spirk_conf_/_hadoop_conf_/itprd-(iuues xid tenplat*1 yarnMdoop194625 If司 6 11 11 ./_spark_con_/_spark_hdoop_co

27、nf_xnl1 yarnhadoop3104 If目 6 11:11 ./_spark_con_/_spark_conf_. properties2 yarnhajoop锄6 8月16 18:20 ./_hejlib3_1 yarnhadoop47950679 6f习 23 2020 . /_hive_libs-/st4ndJ.one-netastore-l. .0 0-1634-hive3. ji1 yrnhadoop6月 23 2020 . / hive libs /phoenix-hiTe6. 0. O-HBase-2.0. jar1 rootroot13 2018 . /_hive_l

28、ib5/kive-hbAze-hanller-3. 1.0. -1654. j at1 rootroot13 2018 . / hive libs_/hbaz-zoo):c4p0r-2. 0.0. 3.0.0. 0-1634. jr13 2018 . /lib: /hba：QrhadQd-protobuf2. 1.0. jarI rootroot1495963 7月I rootroot3973361 7月13 2018 . / hiv libs /hbauQFhajed-kase-Tplication 0 0 3 0 0 0-1634. jw,I rootroot5935M5 7月13 2018 ./_hiveJibs-/hl*se-proto