互联网用户行为日志数据分析1

1、互联网用户行为日志数据分析大数据课程作业吴宁时间: 2017.05.14 15:00原始数据-用户信息USERIDGENDERBIRTHDAYEDUJOBINCOMEPROVINCECITYISCITY13F320E7B5EAD1024AC95C3B208610DB男1981高中/中专/技校企业/公司一般职员15012000元内蒙古呼和浩特unknownEA3502C3594588F0E9D5142F99C66627女1989大学本科学生无收入浙江嘉兴unknown0B5F2CAF4C5DE1E3A8E68F505AE6899B女1975大专专业技术人员50018000元广东珠海城市58594

2、26ED1A0862FAD780BEA63537AB9女1981高中/中专/技校个体户/自由职业者5011000元安徽芜湖城市原始数据-用户操作日志文件Last38638L_Start2012-05-07 08-27-59T153=Pexplorer.exe=I144=W10092=V6.00.2900.5512T173=Pwps.exe=I4528=W30488=V1,0,0,2877=NKingsoft Office=CKingsoft Corp. Ltd.T1389=Pexplorer.exe=I144=W30056=V6.00.2900.5512T1399=Pwps.exe=I4528

3、=W104a6=V1,0,0,2877T1417=Pexplorer.exe=I144=W705d0=V6.00.2900.5512T1419=Pwps.exe=I4528=W104a6=V1,0,0,2877T2971=Pexplorer.exe=I144=W705d0=V6.00.2900.5512T159=PQQ.exe=I4092=W10474=V1.71.2521.290T184=PQPlus.exe=I5056=W2047a=V0, 0, 0, 0=NQ+=CTencentT194=PQQ.exe=I4092=W10556=V1.71.2521.290T222=Ppopup_QQ.

4、exe=I4092=W305d2=V1.71.2521.290T242=Pchrome.exe=I5932=U 200万行左右数据分析目标 分析用户经常使用的应用程序的关联关系 用户年龄与上网页时间的关系 用户年龄与在线时间的关系 学历与上网页时间的关系 学历与在线时间的关系 数据分析过程l 数据解析：解析日志文件,将数据读入预处理程序中,将需要的数据分组存入文件中.l 数据集成：将日志文件(体现同)和用户信息表(excel) 关联在一起 .l 数据选择：从数据文件中提取与分析任务相关的数据l 数据分析：根据课堂所学的大数据知识,分析数据,获得分析结果(知识)l 知识表示：用图形直观表示分析结

5、果.数据解析-用户使用计算机的时间V1V2V3V4V5V60143692D264FD906F10B8ECAB0F139D177712012-05-07 12-29-51106485801EDEB72AE20FE8BD93D126EC0FBAF9118812012-05-07 12-42-49194168260238BF8BE49F536ABC8F6670B766CAA7481912012-05-07 10-11-57113861852002BF8ABFA665EA584964322AEC740E2057432012-05-07 20-20-28360360339FC64528A066AB3EE

6、8E1C90CA0B56193072012-05-07 07-32-0259720439303AF956B4521A1AE096D3F10FB662D23153782012-05-07 11-44-42241024103C3D1E536560A76B66CAC523F25838530552012-05-07 182A06576A04EEF4BE27B87FE84C7D2313502012-05-07 08-05-0660911749203DF84A064F0A0A3FC0EBB8B1A833DD5366692012-05-07 07-44-381070107数据

7、解析-用户应用程序/网站idP0143692D264FD906F10B8ECAB0F139D1explorer.exe0143692D264FD906F10B8ECAB0F139D1360Safe.exe0143692D264FD906F10B8ECAB0F139D1360chrome.exe0143692D264FD906F10B8ECAB0F139D1liveupdate.exe0143692D264FD906F10B8ECAB0F139D1QQ.exe0143692D264FD906F10B8ECAB0F139D1iexplore.exe0143692D264FD906F10B8ECAB

8、0F139D1360leakfixer.exe0143692D264FD906F10B8ECAB0F139D1popup_QQ.exe0143692D264FD906F10B8ECAB0F139D1360sd.exe01EDEB72AE20FE8BD93D126EC0FBAF91360chrome.exe01EDEB72AE20FE8BD93D126EC0FBAF91logon.scr01EDEB72AE20FE8BD93D126EC0FBAF91QQ.exe01EDEB72AE20FE8BD93D126EC0FBAF91explorer.exe01EDEB72AE20FE8BD93D126E

9、C0FBAF91popup_QQ.exe0238BF8BE49F536ABC8F6670B766CAA7explorer.exe0238BF8BE49F536ABC8F6670B766CAA7idU0143692D264FD906F10B8ECAB0F139D10143692D264FD906F10B8ECAB0F139D1NULL0143692D264FD906F10B8ECAB0F139D1hao1230143692D264FD906F10B8ECAB0F139D1gougou0143692D264FD906F10B8ECAB0F139D1baidu0143692D264FD906F10B

10、8ECAB0F139D1aipai0143692D264FD906F10B8ECAB0F139D1qzone0143692D264FD906F10B8ECAB0F139D1qq01EDEB72AE20FE8BD93D126EC0FBAF91qq01EDEB72AE20FE8BD93D126EC0FBAF9101EDEB72AE20FE8BD93D126EC0FBAF9152bus01EDEB72AE20FE8BD93D126EC0FBAF91baidu01EDEB72AE20FE8BD93D126EC0FBAF91i01EDEB72AE20FE8BD93D126EC0FBAF91doyouhi

11、ke01EDEB72AE20FE8BD93D126EC0FBAF91haishanghua用户-应用程序用户-网站应用程序的关联关系items support130242 explorer.exe 0.938257993384785130243 QQ.exe 0.845644983461962130241 explorer.exe,QQ.exe 0.797133406835722130240 popup_QQ.exe,QQ.exe 0.558985667034179130244 popup_QQ.exe 0.558985667034179130239 explorer.exe,popup_QQ

12、.exe 0.531422271223815130245 iexplore.exe 0.527012127894157130236 explorer.exe,iexplore.exe 0.482910694597574130237 iexplore.exe,QQ.exe 0.458654906284454130248 360leakfixer.exe 0.453142227122382130246 360MobileMgr.exe 0.449834619625138130249 360se.exe 0.449834619625138130221 360leakfixer.exe,explore

13、r.exe 0.439911797133407130232 360MobileMgr.exe,explorer.exe 0.437706725468578130214 360se.exe,explorer.exe 0.427783902976847130250 360tray.exe 0.426681367144432130206 360tray.exe,explorer.exe 0.416758544652701130247 rundll32.exe 0.412348401323043130197 360sd.exe,explorer.exe 0.407938257993385130251

14、360sd.exe 0.407938257993385130227 explorer.exe,rundll32.exe 0.398015435501654130215 360se.exe,QQ.exe 0.395810363836825130233 360MobileMgr.exe,QQ.exe 0.395810363836825130222 360leakfixer.exe,QQ.exe 0.390297684674752130207 360tray.exe,QQ.exe 0.379272326350606130228 QQ.exe,rundll32.exe 0.36714443219404

15、6应用程序分布图应用相关性-余弦公示计算结果0143692D264FD906F10B8ECAB0F139D1:c(714, 1685, 77, 70, 1621)01EDEB72AE20FE8BD93D126EC0FBAF91:c(714, 1685, 1621, 77, 70)0238BF8BE49F536ABC8F6670B766CAA7:c(714, 1685, 77, 70, 109)02BF8ABFA665EA584964322AEC740E20:c(714, 1685, 1621, 70, 109)0339FC64528A066AB3EE8E1C90CA0B56:c(1685, 7

16、14, 1621, 109, 1876)03AF956B4521A1AE096D3F10FB662D23:c(1685, 714, 1876, 1621, 2185)03C3D1E536560A76B66CAC523F258385:c(714, 1685, 77, 93, 70)03D2A06576A04EEF4BE27B87FE84C7D2:c(714, 1685, 1876, 70, 1621)03DF84A064F0A0A3FC0EBB8B1A833DD5:c(714, 1685, 77, 70, 93)03E7A57A54E58BEC81CCD69D8239583B:c(714, 16

17、85, 77, 70, 2448)03ED6C135C5912CF3BD6060F43EDEDF1:c(714, 1685, 1032, 109, 1621)0496990FDBBC5B49DF47D934CFF28CFB:c(714, 2448, 77, 2446, 70)04BABE9EBBC0F48F79ABD7CB6191508E:c(714, 1685, 1621, 1032, 109)0564D287D61CDB0C9870A405E34A5E82:c(29, 2688, 587, 712, 586)06329D6F99A22DD5B296988E9F716422:c(2602,

18、2165, 1818, 2110, 2021)066906620E978664F94CD28D5E2AED02:c(1685, 714, 1621, 1876, 1032)067FC9C8D8D15B80CA775A1D9433CB48:c(714, 1685, 806, 822, 1678)06E9A101B77B2DAEBE8E57420B7E683E:c(1685, 714, 1621, 1032, 1318)074AC99B8EB791C444E70D63CD55EB10:c(714, 1685, 77, 1876, 70)0785946275B91BE0F31FAD4A225ECFC

19、2:c(714, 1685, 2495, 70, 93)078D31F088D04F0A05AD1F68791F0653:c(2448, 2446, 714, 2456, 70)07AAAC73119FFA7CF51E682D60AAD759:c(714, 1685, 371, 1245, 1419)07BBF3D62377B1FF602C823E8DA9FC4D:c(1685, 714, 1621, 1876, 2495)07C72DF2929683FCDC1EBCF48F40DDCE:c(714, 1685, 2495, 77, 93)0825DF17932DDAC75AD58D0AD5339AA0:c(1685, 714, 1621, 77, 1032)085DCFD65206B0BB8CF78072664FA04F:c(2, 3, 1685, 839, 1032)应用程序关联度应用程序关联性应用程序的关联关系 用途1:应用商店可以用来做程序推荐 用户2:应用厂家选择在引流用户时尽量选择关联度小的应用合作,避免引入同一类用户年龄与上网页时间的关系年龄集中在22-35