深学习—训练集验证集和测试集概念

1、 Training, Validation and Test DataExample:(A)We have data on 16 data items , their attributes and class labels.RANDOMLY divide them into 8 for training, 4 for validation and 4 for testing.Training Item No. d Attributes Class1.02.03.KNOWN FOR ALL14.15.DATA ITEMS16.17.08.0Validation 9.010.011.112.0Te

2、st 13.014.015.116.1(B). Next, suppose we develop, three classification models A, B, C from the training data. Let the training errors on these models be as shown below (recall that the models do not necessarily provide perfect results on training dataneither they are required to). Classification res

3、ults fromItem No.d- AttributesTrue Class Model A Model BModel C 1.00112.ALL KNOWN00003.10104.11015.10006.11117.00008.0000Classification Error2/83/83/8 (C). Next, use the three models A, B, C to classify each item in the validation set based on its attribute vales. Recall that we do know their true l

4、abels as well. Suppose we get the following results: Classification results fromItem No.d- AttributesTrue Class Model A Model BModel C 9.010010.001011.101012.0010Classification Error2/42/41/4If we use minimum validation error as model selection criterion, we would select model C.(D). Now use model C

5、 to determine class values for each data point in the test set. We do so by substituting the (known) attribute value into the classification model C. Again, recall that we know the true label of each of these data items so that we can compare the values obtained from the classification model with th

6、e true labels to determine classification error on the test set. Suppose we get the following results.Classification results from Item No.d- AttributesTrue ClassModel C13.0014. ALL KNOWN0015.1016.11Classification Error1/4(E). Based on the above, an estimate of generalization error is 25%. What this

7、means is that if we use Model C to classify future items for which only the attributes will be known, not the class labels, we are likely to make incorrect classifications about 25% of the time.(F). A summary of the above is as follows:ModelTrainingValidation Test A2550 -B37.550-C37.52525 Cross Vali

8、dationIf available data are limited, we employ Cross Validation (CV). In this approach, data are randomly divided into almost k equal sets. Training is done based on (k-1) sets and the k-th set is used for test. This process is repeated k times (k-fold CV). The average error on the k repetitions is

9、used as a measure of the test error.For the special case when k=1, the above is called Leave- One Out-Cross-Validation (LOO-CV).EXAMPLE: Consider the above data consisting of 16 items.(A). Let k= 4, i.e., 4- fold Cross Validation. Divide the data into four sets of 4 items each.Suppose the following

10、set up occurs and the errors obtained are as shown.Set 1 Set 2 Set 3 Set 4Training Items 1 - 12Items 1 - 813-16Items 1 - 49-16Items 5-16Test Items 13-16Items 9-12Items 5 - 8Items 1 4 Error on test set (assume)25%35%28%32%Estimated Classification Error (CE) = 25+35+28+32 = 30% 4(B). LOO CV For this, data are divided into 16 sets, each consisting of 15 training data and one test data. Set 1 Set 2 Set 15Set 16Training Items 1 - 15Items 1 14,16Item 1,3-8Items 2-16Test Item 16Item 15Item 2Ite