金融领域大数据管理与数据质量分析

大数据管理与数据质量

-美国金融业中的对策处理速度容量限制数据质量Overview数据<=Data=信息(并非数字集合)数据科学(约)=信息科学为何研究大数据?因为相关产品(如硬盘,

memory,CPU等)价格指数下降因为信息爆炸因为大数据导致许多新问题大数据研究是多学科的综合(IT,DM,BI,BA,…)实业界对大数据问题的对策(见下文)1.数据库策略1.1Database(DB)performance1.2DBspace1.1DBperformanceAuditing–2tables:asmallactive&ahugepassivePartitionIndex(good/bad;Cluster;Global/Local)Locktype(whenapplyrowlock)Transaction:1-phaseor2-phaseNormalizationInternaloptimization

(e.g.ExecutionPlan=>hintinOracle)Constraints(e.g.Check)usagetoreplacetriggerTricks(e.g.Datefunction;Searchsmalltablefirst;…)1.2DBspaceSpacearrangementforevendistribution

(e.g.1hugetableusesafewdatafiles)CleaningprocedurewithdefragmentPartitiondesignwithcleaningplan2.Applications(软件)

(Javaexample)Usingadvancedlanguage(e.g.JavaorC#)2.1Memory（内存）2.2Disk/networkspace2.3Performance2.4Maintainability2.1MemoryMinimizebigobjectscreationandcoexistenceGC(GarbageCollection)ornullbigobjectsonceoutofscopeChooseappropriateGCtypegc()TrytosplitonebigobjecttosmallobjectsUsemutableclassforfrequentlychangedbigobjects(e.g.StringBuilder,insteadofString)2.2Disk/networkspaceSmartcleanandarchiveprocesses

e.g.archivezippedoldornotusedfilestolowspeednetworkspaceanddeleteveryoldfilesfromthatspaceSmartloggingsettingse.g.log4jsizerollinge.g.AvoidduplicatedortriviallogginginfoMonitorforspaces2.3PerformanceAvoidredundanttreatment(inbigloops)

MaximizereuseMulti-threadingDBaccessingLogging--avoidslowoptions(e.g.line#)2.4MaintainabilitySOAprinciplesLosecoupling,reusability,granularity,modularity,composability,componentization,interoperability,…JEEpatterns(DAO,DTO,BizDelegation,…)Designpatterns(23)andMVCCreationStructureBehavior(e.g.Visitor)OOPprinciplesAbstraction,encapsulation,polymorphism,…Open/Close3.数据质量控制3.1Business3.2ProcessA.Failover&DR(DisasterRecovery)B.QA(QualityAssurance)

(see<软件质量管理点滴>

fordetails)C.UAT(UserAcceptanceTest)3.3Technology3.1BusinessReducemanualwork;IncreaseautomationCompleteapprovalsystemformanualwork

E.g.1level=>2levelsor3levelsapprovalExtendviewpointstoconfirmdataqualityReduceredundancysystems(e.g.duetomerge,duetovendors)ScheduleCleansing(seedetails)EnhanceReconciliation(seedetails)BuildTrustlevel(seedetails)Trytocoverallrarecases3.1.ECleansingWhenAtsystemmergeAtmajorchangeHowDevelopdetectionapplicationsDelivermismatchreportstoIT&businessFindsolutionsonbothIT&business3.1.FReconciliationWhere1+subsystemshavedataforsamecontents.1+subsystemshaveindependentdatechangefunctionality.WhatRun&improverecon.app.routinely.Categorizereportsbyurgency.Analyzereports.DebugoradjustbizruleorapplyCleansing.3.1.GTrustlevelWhenAt1+fixeddatainputsInputsareindependentMustdecidefinaldetailsfrominputsHow(basedon)Providerlevel(foradetaileddatagroup)DatahistorySamples:Bloomberg,Reuter,Telekurs,DTCC,…;Moody,S&P,Fitch.3.2.AFailover&DRFailoverDB:2+atdiff.locations;real-timereplicationAppActive-Active:ClusterwithLoadBalancingActive-PassiveAuto(viaSAN)Manual+AutoDRDB:e.g.dailyorhourlyorreal-timereplicationApp:Manualswitch3.3TechnologyDBdesignConstraint‘Check’(forsensitivetablevalues)Normalization(toreduceduplicatio