机器学习驱动的基本面研究程序

1、等：学习驱动的基本面量化投资研究2019 年第 8 期附 录正文未报告部分程序列表1.数据及代码说明错误!未定义书签。1)数据说明：12)代码说明：12.代码整理4a)MainFile.py4b)DataTransfrom.py9c)NWttest.py12d)ReturnSeriesTest.py14e)StrategyConstruct.py16f)FactorTest.py27g)DFN.py45h)RNNM.py48i)Ensembleall.py51j)transecfee.py52k)selectFactor.py541等：学习驱动的基本面量化投资研究2019 年第 8 期数据及代

2、码说明1.1)数据说明：a)factor 文件夹中包含初始 96 项因子数据（由于杂志社对附件大小的要求，这里仅在factorselect文件夹内展示了 16 项因子数据，完整的 96 项因子数据请通过数据库自行或作者）b)factorselect 文件夹中包含筛选完成的 16 项因子数据c)returnseries 文件夹中包含 3/12/24/36滑动窗口下各算法构建投资组合月度收益序列d)ff3/ff5 分别为 Fama-French3 因子数据，RF 为月度无风险利率数据，final_return 为股票月度数据e)ff30 为去掉市值最小的 30%股票后分别根据市值（size)和价格

3、比（EP）分组后构建 MKT/SMB/VMG 3 因子数据f)factorEW/factorVW 分别为单因子检验 10-1组合（等权重/市值）月度序列2)代码说明：a)MainFile.py主程序，运行该函数即可得到各主要结果b)DataTransfrom.py导 入 基 础 数 据 并 进 行 预 处 理 ， 最 后 将 原 始 数 据 转 换 成 每 个 截 面 一 个 stocknumfactornum 的Dataframe,列名为各因子名称+stock'+'ret'，'stock'为股票代码，ret为对应截面股票月度c)NWttest.py定义

4、对某一序列是否异于 0 进行 Newey and West (1987) t 检验函数d)ReturnSeriesTest.py在获取各个算法构建组合月度序列后， 对各个序列与 OLS 回归（benchmark）和 DFN（表现最好的深度算法）序列是否显著差异进行 NW-T 检验1等：学习驱动的基本面量化投资研究2019 年第 8 期e)StrategyConstruct.pyi.投资组合构建通用函数（output),ii.FC 和 ensemble 无内置算法包，故单独构建 FC 和 ensemblennf)FactorTest.py因子检验补充结果，内容包含：i.在去掉市值最小的 30%股

5、票后分别根据市值（size)和价格比（EP）分组后构建 MKT/SMB/VMG 3 因子ii.单因子 10 分组 10-1/1-10组合因子调整iii.单因子 10 分组检验各组因子调整iv.各因子与size 因子双变量分组检验结果v.各因子与 BM 因子双变量分组检验结果vi.6.96 项因子 fama macbeth 回归检验结果g)DFN.py深度前馈库文件，包含深度前馈函数DFN()，需在 GPU 环境下运行h)RNNM.py循环神经库文件，包含如下内容：i.基础循环神经单元：BaseRnn()ii.可用于训练的循环神经整体架构：lstmmodule()，该架构基于 BaseRnn()

6、构建循环神经。该文件需在 GPU 环境下运行i)Ensembleall.py集成深度学习模型库文件，包含集成模型：Ensemblelr()。因其中集成深度学习模型，需在 GPU 环境下使用，如不集成深度学习模型，则无 GPU限制。j)transecfee.py计算不同成本下的投资组合绩效变化， 包含函数 transecfee() 以及showtransecfee()直接调用showtransecfee()即可2等：学习驱动的基本面量化投资研究2019 年第 8 期k) selectFactor.py用于筛选重要因子，包含：i.非循环神经模型所用的 dropimportant()函数ii.循环神

7、经模型所用的dropimportant2()函数iii.FC 方法的筛选函数 FCselect()函数注：获取结果需要运行的python 文件仅为：MainFile.py 和 FactorTest.py3等：学习驱动的基本面量化投资研究2019 年第 8 期2. 代码整理a) MainFile.py1234567891011121314151617181920212223242526272829303132333435363738#!/usr/bin/env python# -*- coding: utf-8 -*-"""description:构建学习驱动多因子

8、投资策略主函数1.首先输入各算法参数（参数根据第一个滑动窗口网格调参确定 此处直接输入）2.全样本 3/12/24/36滑动窗口函数运行,最终直接输出output'文件夹内组合月度FF3/5-alpha,sharperatio并将序列保43.全样本 3/12/24/36滑动窗口各个算法组合月度序列是否显著差异NW-T 检验4.不同费率下的组合绩效结果5.全样本 12华东窗口下各个算法的特征筛选6.特征筛选后 16 项因子 12滑动窗口函数运行,最终输出组合月度FF3/5-alpha,sharpe ratio 并将序列保output'文件夹内注：深度学习算法要在使有GPU 的环境

9、下进行训练"""from StrategyConstruct import FC, output, output2, comboutput, ensemblennfrom selectFactor import dropimportant, dropimportant2, FCselectfrom DataTransfrom import datatransfrom, datatransfrom2from xgboost.sklearn import XGBRegressorfrom sklearn.ensemble import GradientBoosting

10、Regressorfrom sklearn.linear_mimport LinearRegression,Lasso,ElasticNet,Ridgefrom sklearn.cross_decomposition import PLSRegressionfrom sklearn.neural_network import MLPRegressorfrom sklearn.svm import SVRimport DFNimport RNNMas rmimport Ensembleall as eaimport warningsfrom mxnet import gpuimport osfr

11、om transecfee import showtrasecfeefrom ReturnSeriesTest import returnseriestestwarnings.filterwarnings('ignore')#各个算法参数（根据第一个窗口网格调参确定）window=3,12,24,36PLS_params=2,2,1,1等：学习驱动的基本面量化投资研究2019 年第 8 期3940414243444546474849505152535455565758596061626364656667686970717273747576777879805lasso_param

12、s=1e-3,5e-4,0.01,0.01ridge_params=0.1,0.005,0.01,0.005elasticnet_params='alpha':0.01,1e-3,0.01,0.1,'l1_ratio':0.3,0.3,0.7,0.3SVR_params='kernel':'linear','linear','rbf','rbf','gamma':1e-3,1e-3,1e-3,1e-4,'C':0.01,0.001,0.01,1e-4G

13、BDT_params='learning_rate':0.1,0.1,0.1,0.1,'maxdepth':2,3,2,2,'n_estimators':100,100,100,100#XGBOOST 与GBDT 相同 此处共用ENANN_params = 'max_iter': 100, 100, 200, 300, 'p': 0.3, 0.5, 0.7, 0.5DFN_params = 'learning_rate':0.1, 0.1, 0.1, 0.001, 'batch':

14、300, 400, 300, 400LSTM_params = 'learning_rate':1e-4, 1e-5, 1e-4, 1e-6, 'depth': 2, 2, 1, 2,'hidden_number': 256*4RNN_params = 'learning_rate':0.1, 0.1, 0.1, 0.001, 'depth': 1, 1, 2, 1,'hidden_number': 256*4#*2.全样本 3/12/24/36滑动窗口函数运行*#path = r'.Dat

15、aBasefactor'#96 项因子所在路径factorname = x1:-4 for x in os.listdir(path)riskfree, timeseries, factor, timeseries2, index = datatransfrom(path)0,datatransfrom(path)1, datatransfrom(path)2, datatransfrom2(path)0,datatransfrom2(path)1for i in range(4):i= 0output(windowi,LinearRegression(),'OLS'+

16、str(windowi),riskfreei, timeseries)FC(windowi, riskfreei, timeseries, 96,'FC')output(windowi, PLSRegression(PLS_paramsi), 'PLS' + str(windowi), riskfreei,timeseries)output(windowi,Lasso(alpha=lasso_paramsi),'Lasso'+ str(windowi), riskfreei,timeseries)output(windowi,Ridge(alph

17、a=ridge_paramsi),'Ridge'+str(windowi),riskfreei,timeseries)output(windowi,ElasticNpha= elasticnet_params'alpha' i,l1_ratio=elasticnet_params'l1_ratio'i),'ElasticNet'+str(windowi),riskfreei, timeseries)output(windowi,SVR(kernel=SVR_params'kernel'i,gamma= SVR_pa

18、rams 'gamma'i,C=SVR_params 'C'i ),'SVR'+str(windowi),riskfreei, timeseries)output(windowi,GradientBoostingRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'maxdepth'i,learning_rate=GBDT_params'learning_rate'i), 'GBDT' +str

19、(windowi),riskfreei, timeseries)output(windowi,XGBRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'maxdepth'i等：学习驱动的基本面量化投资研究2019 年第 8 期818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211226, learning_rate=G

20、BDT_params'learning_rate'i), 'XGBOOST' + str(windowi), riskfreei,timeseries)output(windowi, ensemblenn(5,muse = MLPRegressor(solver = 'lbfgs',max_iter=ENANN_params'max_iter'i), pickpercent=ENANN_params'p'i), 'ENANN' +str(windowi), riskfreei, timeseries

21、)output(windowi, DFN.DFN(outputdim=1, neuralset=96, 50, 25, 10, 5, 2, ctx=gpu(0),epoch=10, batch_size=DFN_params'batch'i, lr=DFN_params'learning_rate'i), 'DFN' +str(windowi), riskfreei, timeseries)output2(windowi, rm.lstmmodule(96, LSTM_params'hidden_number'i,LSTM_par

22、ams'depth'i, 100, 3571, lr=LSTM_params'learning_rate'i), 'LSTM'+str(windowi) ,riskfreei, timeseries2)output2(windowi, rm.lstmmodule(96, RNN_params'hidden_number'i,RNN_params'depth'i, 100, 3571, lr=RNN_params'learning_rate'i, ntype='RNN'), '

23、RNN'+str(windowi), riskfreei, timeseries2)mlist = DFN.DFN(outputdim=1, neuralset=96, 50, 25, 10, 5, 2, ctx=gpu(0),epoch=10, batch_size=DFN_params'batch'i, lr=DFN_params'learning_rate'i),ensemblenn(5,muse = MLPRegressor(solver = 'lbfgs',max_iter=ENANN_params'max_iter&#

24、39;i), pickpercent=ENANN_params'p'i),XGBRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'maxdepth'i, learning_rate=GBDT_params'learning_rate'i),GradientBoostingRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'm

25、axdepth'i,learning_rate=GBDT_params'learning_rate'i),PLSRegression(PLS_paramsi),Ridge(alpha=ridge_paramsi),SVR(kernel=SVR_params'kernel'i,gamma= SVR_params 'gamma'i,C=SVR_params 'C'i)# PLS 一定要放在倒数第三个（PLS 输出形式为list 故进行了进一步处理）nmolist = rm.lstmmodule(96, LSTM_params&

26、#39;hidden_number'i, LSTM_params'depth'i,100, 3571, lr=LSTM_params'learning_rate'i),rm.lstmmodule(96, RNN_params'hidden_number'i, RNN_params'depth'i,100, 3571, lr=RNN_params'learning_rate'i, ntype='RNN')# 循环神经模型mname = 'DFN', 'En-ann

27、9;, 'xgboost', 'GBDT', 'lasso', 'Elasticnet', 'pls', 'Ridge','svm', 'LSTM', 'RNN'ensemblem= ea.Ensemblelr(mlist, nmolist, mname)comboutput(windowi,ensemblem, 'Ensemble'+str(windowi),riskfreei, timeseries2,index)#*3.各算法序列

28、差异NW-t 检验*#for i in window:returnseriestest(i)等：学习驱动的基本面量化投资研究2019 年第 8 期1231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631647#*4.不同费率情形*#showtrasecfee(0.005)showtrasecfee(0.0075)showtrasecfee(0.01)#*5.全样本 12特征筛选过程*#i = 1#选取 1

29、2滑动窗口筛选因子dropimportant(windowi ,LinearRegression(), 'OLS'+str(windowi), factorname,timeseries,0.0201)FCselect(factorname, timeseries)dropimportant(windowi, PLSRegression(PLS_paramsi), 'PLS', factorname, timeseries,0.0230)dropimportant(windowi, Lasso(alpha=lasso_paramsi), 'Lasso&#

30、39;, factorname, timeseries,0.0208)dropimportant(windowi, Ridge(alpha=ridge_paramsi), 'Ridge', factorname, timeseries,0.0208)dropimportant(windowi, ElasticNpha= elasticnet_params'alpha' i,l1_ratio=elasticnet_params'l1_ratio'i), 'ElasticNet', factorname, timeseries, 0.

31、0212)dropimportant(windowi, SVR(kernel=SVR_params'kernel'i,gamma= SVR_params'gamma'i,C= SVR_params 'C'i ), 'SVR', factorname, timeseries, 0.0225)dropimportant(windowi,GradientBoostingRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'm

32、axdepth'i,learning_rate=GBDT_params'learning_rate'i), 'GBDT', factorname,timeseries, 0.0268)dropimportant(windowi,XGBRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'maxdepth'i, learning_rate=GBDT_params'learning_rate'i), 'XGBOOS

33、T',factorname, timeseries, 0.0273)dropimportant(windowi, ensemblenn(5,muse = MLPRegressor(solver = 'lbfgs',max_iter=ENANN_params'max_iter'i), pickpercent=ENANN_params'p'i), 'ENANN',factorname, timeseries, 0.0234)dropimportant(windowi, DFN.DFN(outputdim=1, neuralse

34、t=96, 50, 25, 10, 5, 2, ctx=gpu(0),epoch=10, batch_size=DFN_params'batch'i, lr=DFN_params'learning_rate'i), 'DFN',factorname, timeseries, 0.0278)dropimportant2(windowi, rm.lstmmodule(95, LSTM_params'hidden_number'i,LSTM_params'depth'i, 100, 3571, lr=LSTM_param

35、s'learning_rate'i), 'LSTM', factorname,timeseries2, 0.0257)dropimportant2(windowi, rm.lstmmodule(95, RNN_params'hidden_number'i,等：学习驱动的基本面量化投资研究2019 年第 8 期1651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042

36、052068RNN_params'depth'i, 100, 3571, lr=RNN_params'learning_rate'i, ntype='RNN'), 'RNN',factorname, timeseries2, 0.0210)#*6.特征筛选后 16 项因子 12滑动窗口函数运行*#path = r'.DataBasefactorselect'#经过筛选后因子集合所在路径riskfree,timeseries,factor,timeseries2=datatransfrom(path)0,datatr

37、ansfrom(path)1,datatransfrom(path)2,datatransfrom2(path,after=True)0i=1 #选取 12滑动窗口测试筛选后因子集合绩效表现output(windowi,LinearRegression(),'OLS'+str(windowi),riskfreei, timeseries)FC(windowi, riskfreei, timeseries, 11,'FC')output(windowi, PLSRegression(PLS_paramsi), 'PLS' + str(windowi

38、), riskfreei,timeseries)output(windowi,Lasso(alpha=lasso_paramsi),'Lasso'+ str(windowi), riskfreei,timeseries)output(windowi,Ridge(alpha=ridge_paramsi),'Ridge'+str(windowi),riskfreei,timeseries)output(windowi,ElasticNpha= elasticnet_params'alpha' i,l1_ratio=elasticnet_params&

39、#39;l1_ratio'i),'ElasticNet'+str(windowi),riskfreei, timeseries)output(windowi,SVR(kernel=SVR_params'kernel'i,gamma= SVR_params 'gamma'i,C=SVR_params 'C'i ),'SVR'+str(windowi),riskfreei, timeseries)output(windowi,GradientBoostingRegressor(n_estimators=GBDT

40、_params'n_estimators'i,max_depth=GBDT_params'maxdepth'i,learning_rate=GBDT_params'learning_rate'i), 'GBDT' +str(windowi),riskfreei, timeseries)output(windowi,XGBRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'maxdepth'i, learnin

41、g_rate=GBDT_params'learning_rate'i), 'XGBOOST' + str(windowi), riskfreei,timeseries)output(windowi, ensemblenn(5,muse = MLPRegressor(solver = 'lbfgs',max_iter=ENANN_params'max_iter'i), pickpercent=ENANN_params'p'i), 'ENANN' +str(windowi), riskfreei, ti

42、meseries)output(windowi, DFN.DFN(outputdim=1, neuralset=16, 50, 25, 10, 5, 2, ctx=gpu(0),epoch=10, batch_size=DFN_params'batch'i, lr=DFN_params'learning_rate'i), 'DFN' +str(windowi), riskfreei, timeseries)output2(windowi, rm.lstmmodule(11, LSTM_params'hidden_number'i,

43、LSTM_params'depth'i, 100, 3571, lr=LSTM_params'learning_rate'i), 'LSTM'+str(windowi) ,riskfreei, timeseries2)output2(windowi, rm.lstmmodule(11, RNN_params'hidden_number'i,RNN_params'depth'i, 100, 3571, lr=RNN_params'learning_rate'i, ntype='RNN'

44、), 'RNN'+str(windowi), riskfreei, timeseries2)等：学习驱动的基本面量化投资研究2019 年第 8 期b)DataTransfrom.py123456789101112131415161718192021222324252627282930313233343536373839409#!/usr/bin/env python# -*- coding: utf-8 -*-"""description:导入基础数据并进行一些预处理 最后变成每个截面一个Dataframe,列名为各因子名称+stock'+

45、'ret' index 代表单只股票"""import glob,osimport pandas as pdimport warnings#*1.导入因子数据 无风险利率 股票月度数据*#warnings.filterwarnings('ignore')def datatransfrom(datapath):path=datapathfile = glob.glob(os.path.join(path, "*.csv")k=for i in range(len(file):k.append(pd.read_csv

46、(filei)#股票月度ret=pd.read_csv('.DataBasefinal_return.csv')#无风险利率rf=pd.read_csv('.DataBaseRF.csv')rf3=rf.iloc3:-1,:rf12=rf.iloc12:-1,:rf24=rf.iloc24:-1,:rf36=rf.iloc36:-1,:riskfree = rf3, rf12, rf24, rf36#因子名称factor=for i in range(len(file):factor.append(filei20:-4)factor.append('st

47、ock')#*对原始数据进行预处理 每个截面一个Dataframe,列名为96 因子名称+stock'+'ret'index 代表单只股票*#timeseries=for i in range(len(ret.columns)-1):等：学习驱动的基本面量化投资研究2019 年第 8 期414243444546474849505152535455565758596061626364656667686970717273747576777879808182# 加入月度令月度不为null方便函数处理for i in range(len(timeseries2):10k

48、l=pd.concat(kj.iloc:,i+1 for j in range(len(file),axis=1)kl'stock' = ret.iloc:,0kl.columns = factorkl=kl.iloc:-2,:timeseries.append(kl)#删除月度不的数据条for i in range(len(timeseries):timeseriesi'ret'=ret.iloc:,i+1timeseriesi'ret'=timeseriesi'ret'.fillna('null')timese

49、riesi=timeseriesitimeseriesi'ret'.isin('null')return riskfree,timeseries,factor# 为LSTMRNN 设计的数据函数def datatransfrom2(datapath, after=False):path=datapathfile = glob.glob(os.path.join(path, "*.csv")k=for i in range(len(file):k.append(pd.read_csv(filei)#股票月度ret=pd.read_csv(

50、9;.DataBasefinal_return.csv')#因子名称factor=for i in range(len(file):factor.append(filei20:-4)factor.append('stock')#*对原始数据进行预处理 每个截面一个Dataframe,列名为96 因子名称+stock'+'ret'index 代表单只股票*#timeseries2=index = for i in range(len(ret.columns)-1):kl=pd.concat(kj.iloc:,i+1 for j in range(l

51、en(file),axis=1)kl'stock' = ret.iloc:,0kl.columns = factorif after:# 保证筛选后因子个数为 3571 个kl = kl.iloc:, :else:kl = kl.iloc:-2,:timeseries2.append(kl)等：学习驱动的基本面量化投资研究2019 年第 8 期8384858611timeseries2i'ret' = ret.iloc:, i + 1timeseries2i'ret' = timeseries2i'ret'.fillna('

52、;null')index.append(timeseries2i'ret'.isin('null')return timeseries2, index等：学习驱动的基本面量化投资研究2019 年第 8 期c)NWttest.py12345678910111213141516171819202122232425262728293031323334353637383940414212# -*- coding:utf-8 -*-'''description:NW-t 检验所用包'''import numpy as

53、 npfrom collections import namedtuplefrom scipy.stats import distributionsdef _ttest_finish(df, t):''':param df:自由度:param t: t 值:return: 输出 t 和对应p 值'''prob = distributions.t.sf(np.abs(t), df) * 2 # use np.abs to get upper tailif t.ndim = 0:t = t()return t, probNWt_1sampleResu

54、lt = namedtuple('NWT_1sampResult', ('statistic', 'pvalue')def nwttest_1samp(a, popmean, axis=0,L=1):'''主函数:param a: 数据列表:param popmean: 原假设值u0:param axis: 行还是列 默认行:param L: lag滞后多少 默认 1:return: 输出 nw-t 和对应p 值'''a = np.array(a)N = len(a)df = N-1e = a - np.mean(a)residuals = np.sum(e*2)Q = 0for i in range(L):w_l = 1 - (i+1)/(1+L)for j in range(1,N):Q += w_l*ej*ej-(i+1)S = residuals + 2*Q等：学习驱动的基本面量化投资研究2019 年第 8 期434445464748495013nw_var = S/Nd = np.m