python线性回归实验

实验算法python线性回归实验【实验名称】Python线性回归实验【实验要求】掌握Python线性回归模型应用过程，根据模型要求进行数据预处理，建模，评价与应用；【背景描述】线性回归是利用数理统计中回归分析，来确定两种或两种以上变量间相互依赖的定量关系的一种统计分析方法，运用十分广泛。其表达形式为y二w'x+e，e为误差服从均值为0的正态分布。【知识准备】了解线性回归模型的使用场景，数据标准。了解Python/Spark数据处理一般方法。了解spark模型调用，训练以及应用方法【实验设备】Windows或Linux操作系统的计算机。部署Python，本实验提供centos6.8环境。【实验说明】采用成绩数据集作为算法数据，对模型进行训练和回归。【实验环境】Pyrhon3.X，实验在命令行python中进行，或者把代码写在py脚本，由于本次为实验，以学习模型为主，所以在命令行中逐步执行代码，以便更加清晰地了解整个建模流程。【实验步骤】第一步：启动python：

命令行键入python,启动python终端第二步：导入用到的包，并读取数据：(1).导入包:importnumpyasnpimportpandasaspdimportstatsmodels.formula.apiassmfimportscipy⑵•读取数据并随机打SL文件路径为:/opt/algorithm/scoreset/scoregather.txtdf=pd.read_csv("/opt/algorithm/scoreset/scoregather.txt",index_col=0,header=0).sample(frac=1)⑶•展示数据df.head()»>>»AR»>inpDirtnumpyasnpinportpandsssspdimportstatsmodels.formula„^pi恥smfijipo『»>>»AR»>inpDirtnumpyasnpinportpandsssspdimportstatsmodels.formula„^pi恥smfijipo『tscipy#址取薮拾并踱机灯乱d1-pd.read_csv('7opt/algdnthm/scoresat/scoragathe「・t:<thrindex_col-&rheader-O)津展示救指df.head()己修弭1己施別2己惟利3己桧科A己毬斜5己磁亂5己性利7己桧科战己储科9・£白1|卩1口1彳口£-1]己储耳址...己储別17 \56375494策鬼汩77835;S486a5s6ae58别弱0699913S8S]關盟75関65罚77956753旳88-音1610学讴S162481C967669gs0976 3488 9490 9587 S76D 88已检料罚94728D8585B95®2OJ7-joQ-9w-67cg0B&7&982购sgffi强已ttr^23已怯科“已怯利巧冃标学拜07 9088 7586 7092 9506 97第三步：数据预处理说明,数据集包含25门已修学科,以及一门目标学科⑴•划分训练A测试集PP=0.8df_train=df.iloc[:int(np.ceil(len(df)*PP))]df_test=df.iloc[int(np.ceil(len(df)*PP)):](2).数据标准化,获取每列均值,标准差avg_col=df_train.mean()td_col=df_train.std()(3).标准化结果df_train_norm=(df_train-avg_col)/td_coldf_train_norm.head()>»#说明」数据倉包管25门三楼学科「以及一门目标学科...#划分训综隼「测试隼...PP=0.8»>dftrain=df.iloc[:int(np.ceil(len(df)*PP)】]»>df_test=df,.iloc[int(np.ceil(len(df)*PPJ)：]AAAfl昼培标准化「扶取毎列力值「标凑差...avg_col=df_train.mean()»>td_col=df_train.std()>»#祐准化结東...df_t^ain_norm=(df_train-a^g_col)/td_colaaat「minnanr.hedij〔、 —三桂科1 三烤科2三修科4 三催科5M癮科6三楼科7 \学号1Q60.5517751.148897 0.8613750.026243-0.402354-0.2643591.09617251G.698563■G.122947G.1S-1S99-G.51S745G.527762G.5S1132G.2S2222629.273194■0.972510-0.018117-0.140890-1.7975270.1484120.67&252480.2781940.5S31830.7147930.7269431.2253490.561182-1.4053491G-30.5517761.007470-0.0181171.4276430.0627041.3867231.304632三修牡8 三■愜科9三慨科19 \学寻1G61.,0078130.5D64490.9707931.2996131.091047-0.617823510.0596590.458437-0.632933-1.052337-0.5327520.97984062■1.,267752■1.5&1164-1.234326-2.3L2311-1.1310811.246117486.4339190.368608 1.371724・ ■m6.9030201.2534270.84G7Q11G20.8161930.919972 1.171258・B亠0.7116250.522717-0.085269三恼科21三修科22E.W23 三螫科碧H^Sl-25目标芋科学号106-0.0594721.376713-0.784379-0.5L&38S1.3872310.3317786.506330511.222414■0.955129 1.9439171.32262B0.9566990.5Q0510-1.592913521.4727G301G71S40,338995-1.502725-1,2749320.163045-2,292662400.3583940.422781-0.1424500.1599911.1020961.17544D1.2060851G-30.4976830.422781 1.1414070.9975891.2921850.1630451.485935第四步：特征提取及线性回归模型训练基于F检验的特征选择(1).由于建模需要标准化字符串,故需重命名表头list_columns_to_train=df_train_norm.columnschange_columns=['A'+str(zr+1)forzrinnp.arange(len(list_columns_to_train))]df_train_norm.columns=change_columnsco=dict(zip(change_columns,list_columns_to_train))defR_transform(data=[],k=0):r_L=data[k][k]shape=np.mat(data).shapeR_t=np.zeros(shape)foriinnp.arange(shape[0]):forjinnp.arange(shape[1]):if(i==k)&(j!=k):R_t[i][j]=data[k][j]/r_Lelif(i!=k)&(j==k):R_t[i][j]=-1*data[i][k]/r_Lelif(i==k)&(j==k):R_t[i][j]=1.0/r_Lelif(i!=k)&(j!=k):R_t[i][j]=data[i][j]-data[i][k]*data[k][j]/r_LreturnR_tdefforward_step(data,response="",F_in=0.01,F_out=0.5):remaining=list(data.columns)last_drop=""Vld="mark"n=len(data)#因变量选择列selected=[]l=len(selected)#计算相关系数矩阵corr_R=data.corr()mark=1while(mark):#首次变量选择ifl!=(len(data.columns)-1):r_taget=pd.Series(np.diagonal(np.mat(corr_R)))r_taget.index=remainingVi=corr_R[response]**2/r_tagetVc=Vi.drop(selected+[response],axis=0).sort_values()[-1]Vl=Vi.drop(selected+[response],axis=0).sort_values().index[-1]ifVld==last_drop:mark=0elifVld!=last_drop:Fi=Vc*(n-l-2)/(r_taget[response]-Vc)F_test=scipy.stats.f.isf(F_in,1,(n-l-2))ifFi>F_test:selected.append(Vl)l=len(selected)#协方差矩阵转换corr_R=pd.DataFrame(R_transform(np.array(corr_R),remaining.index(Vl)))corr_R.index=data.columnscorr_R.columns=data.columnsifl>=2:#考虑剔除变量foriinnp.arange(len(selected)):rd_taget=pd.Series(np.diagonal(np.mat(corr_R)))rd_taget.index=remainingVid=corr_R[response]**2/rd_tagetVcd=Vid[selected].sort_values(ascending=False)[-1]Vld=Vid[selected].sort_values(ascending=False).index[-1]Fd=Vcd*(n-(l-1)-2)/rd_taget[response]F_test_out=scipy.stats.f.isf(F_out,1,(n-(l-1)-2))if(Fd<F_test_out):selected.remove(Vld)last_drop=Vldcorr_R=pd.DataFrame(R_transform(np.array(corr_R),remaining.index(Vld)))corr_R.index=data.columnscorr_R.columns=data.columnsl=len(selected)else:mark=0else:mark=0returnselectedselect_list=forward_step(df_train_norm,response=change_columns[-1])select_list=change_columns[:2]#线性回归模型训练formula="{}~{}+1".format(change_columns[-1],'+'.join(select_list))model=smf.ols(formula,df_train_norm).fit()print(model.summary())

OLSRegressionResults,Variatile：A26R-squared:0.1.96Model:OLSAdj.R-squared：0.1.77Method；LeastSquaresF-statistic；1&.46Ddtv Thud13^?r2019Prab{F-statistic)；8.57e-05Time：15:34:12Log-Likelihood:-116.09No.Observations：89AIC：238.2DfRasiduals:86旳匚：245.6DfModel:2CavarianceType:nonrobustcaefstderr1 P>|t|[95.9%Conf.Dkt.]litereept-9.489e-150.096-9.37e-15 1.000-0.1910.1.91Al Q.G11&B.&9S&,118 0.9G6-9.1830.206A2 Q.44D&G.&984.5G8 0.BG0G.246Q.&25Omnibus：7.052Durbin-Watson:2.192Prob(Cmnibjs):0.029Jarque-Eera(JE):7.155Skew:■fl.094Prob(JB):0.0279Kjrtosis:2-906Cond.No.1.16 第五步：模型预测展示与结果对比#模型预测展示#利用训练集的统计信息标准化测试数据df_test_norm=(df_test-avg_col)/td_coldf_test_norm.columns=change_columnsdf_test_norm.head

»>#模里预测昵示#利用讥僚案的统计涪息折准化测富髯雄l(lBBdf_te5t_nann={df_test-avg_GDl>/td_coldf_te5t_nann■匚oluinns=chang巳—匸Dlumnsdftestnarm・11皂吕日<boundnethodNDFrame.head口f AlA2A3MA5A6A7\孚号15-0.952928-0.540229-0.010117-D.516745-0.4023540.1404120.05907299-0.2699(72-1.396792-2.216&46-D.518745-2.0900560.561132-0.960420e9.263972-3・235344-2.656592D・9605100.0527042.212264-1.196068&60.551776Qb15S9D70.275G47D.259610-0.159625-Q.6771291.096172&Q0.27B1940.4417611.5942641.7290661.4570780.9729520.867712@6o.saassa1・GC747D0.861275D.101955-0.169025G.146412-0.779966300.6995680.156907-0.010117-0.6301600.295233-1.5G267D-0.369046650B141402G.300334-0.010117-O.44069Q-0.4023540.140412-0.36904613-0.91S137Qb2CO33*-1.337354-D.051612-0.159025-3.153751Q.05307229-0.132180・0・eaiQB20・714792D・026243-2.03DO56G.1484120.57150B69-1.5099931.4317510.457063-0.5107450.295222-1.069999-0.363046609.263972Qb15S9D70.1204651.2719220.295222G.1464120.867712220.4L4995-0・9725100.560211D・6047990.2952330.9739S30.4707&221-1.953902Q.G174BD0.120465D.5712321.4576780.5611320.05907294-2.320841-0.97251D-0.010117-D.6744570.5277620.5611320.8677121010.8252591.5721780.714752-0.12946S1.4578780.9720621.09617270.5517761B14^37Q・56S2111.1162211.457878-G.6771250.67925222-4.272711-Qn4G^02-0.751G26-1.998001-2.020056-G.264955fl.57150852-B.1221801・GG747D1.5942S41・1940771.457878G.5611821.096172550.551776G.2GG224-G・S576GSD・1S19550.295233G.5611820.679252740.U1402-1.538SL9S.01B117-1.608722G.527762-G.2642591.096172269.551776G"2G62241.4477G2-D・4408&QG.295223G.9720620.47Q7G2#模型预测展示#利用训练集的统计信息标准化测试数据df_test_norm=(df_test-avg_col)/td_coldf_test_norm.columns=change_columns#预测结果还原百分制predict=model.predict(df_test_norm)predict_df=predict*td_col[-1]+avg_col[-1]#被选择特征反编码select_list_old=[co[z]forzinselect_list]print(predict_df)[84.57745256M.61047822M.765367991.40266519[84.57745256M.61047822M.765367991.4026651981.5512361481.9526126285.9291455886.360655890.0444017987.37306915]75.1747252287.2291735283.3547716434.7402111786.9278470487.2&Q0827586.3244453739.5426869S87.7955941782.75479959S3.12880&84S7.37306915print(select_list_old[0]);print(select_list_old[1])>»print(selectlistnldt^]):print(select_LLst_Ql(l[1]]三储科1三储瞬2#结果对比F=pd.DataFrame(df_test["目标学科"]).join(pd.DataFrame(predict_df))#回归残差,以及结果相关系数corr=F.corr()["目标学科"][0]stds=abs(F["目标学科"]-F[0]).sum()/len(F)#输出程序运行摘要print("程序选择特征%s,预测结果相关系数is%s,回归残差is%s分"%(",".join(select_list_old),corr,stds))#律麋对比…・F=pdtDataFrane<df_tk11["s,B}}\oin(pd.DstaFrane(predict_dt))A* 归炭差”以圧里無柜黄誉融...csrr-F.corrC)[-冃标学科■][$]arstds=afcsCF严目塀孚制■"】-F[0D.surf)/len(Fl...print("程宇拙择斡征弼”预U1埜果相关至迪is%5.回归聂担15粘另'*富t"□"rjoii(3elect_list_olikuurr.stds)}程底选择峙证三齒科1,2焙科宀茨测注果相关系數LS&.4547512S3141.叵一曰荒歪is1.0918677240&359§7