Kaggle—Digit Recognizer竞赛-LMLPHP

手写体数字识别 MNIST数据集

本赛 train 42000样例 test 28000样例，原始MNIST是 train 60000 test 10000

我分别用 Logistic Regression/ 784-200-200-10的Sparse AutoEncoder/Convolution AutoEncoder刷了下

Kaggle—Digit Recognizer竞赛-LMLPHP

===============方法一、 One-Vs-All 的Logistic Regression===================

%%

ccc

load digitData

%%

input_layer_size  = 28*28;

num_ys = 10;

X = train_x;

[~,y] = max(train_y, [], 2);

lambda = 0.1;

lambda = 100;

[all_theta] = oneVsAll(X, y, num_ys, lambda);

%% ================ Part: Predict for One-Vs-All ================

%  After ...

pred = predictOneVsAll(all_theta, X);

fprintf('\nTraining Set Accuracy: %f\n', mean(double(pred == y)) * 100);

%% ============== 计算test准确度(test_y 是基于KNN的 只作为参考)

[~,test_y] = max(test_y, [], 2);

pred = predictOneVsAll(all_theta, test_x);

fprintf('\nTest Set Accuracy: %f\n', mean(double(pred == test_y)) * 100);

%% write csv file

pred(pred==10) = 0;

M = [(1:length(pred))' pred(:)];

csvwrite('LiFeiteng0824.csv',M)

Kaggle—Digit Recognizer竞赛-LMLPHP

===============方法二、 784-200-200-10的Sparse AutoEncoder ===================

%% STEP 0: Here we provide the relevant parameters values that will

tic

inputDim = 28;

inputSize = 28 * 28;

numClasses = 10;

hiddenSizeL1 = 200;    % Layer 1 Hidden Size

hiddenSizeL2 = 200;    % Layer 2 Hidden Size

sparsityParam = 0.1;   % desired average activation of the hidden units.

                       % (This was denoted by the Greek alphabet rho, which looks like a lower-case "p",

		               %  in the lecture notes).

lambda = 3e-3;         % weight decay parameter

beta = 3;              % weight of sparsity penalty term

maxIter = 100;

%% STEP 1: Load data

load digitData

trainData = train_x';

[~, trainLabels] = max(train_y, [], 2);

%%% 增加数据

%%% ZCA白化 像素值范围变化 []

% trainData = ZCAWhite(trainData);

%% STEP 2: Train the first sparse autoencoder

sae1Theta = initializeParameters(hiddenSizeL1, inputSize);

options.Method = 'lbfgs';

options.maxIter = 200;	  % Maximum number of iterations of L-BFGS to run

options.display = 'on';

[sae1OptTheta, cost] = minFunc( @(p) sparseAutoencoderCost(p, ...

                                   inputSize, hiddenSizeL1, ...

                                   lambda, sparsityParam, ...

                                   beta, trainData), ...

                              sae1Theta, options);

% -------------------------------------------------------------------------

W1 = reshape(sae1OptTheta(1:hiddenSizeL1*inputSize), hiddenSizeL1, inputSize);

display_network(W1', 12);

%% STEP 2: Train the second sparse autoencoder

[sae1Features] = feedForwardAutoencoder(sae1OptTheta, hiddenSizeL1, ...

                                        inputSize, trainData);

%  Randomly initialize the parameters

sae2Theta = initializeParameters(hiddenSizeL2, hiddenSizeL1);

options.Method = 'lbfgs';

options.maxIter = 100;	  % Maximum number of iterations of L-BFGS to run

options.display = 'on';

[sae2OptTheta, cost] = minFunc( @(p) sparseAutoencoderCost(p, ...

                                   size(sae1Features,1), hiddenSizeL2, ...

                                   lambda, sparsityParam, ...

                                   beta, sae1Features), ...

                              sae2Theta, options);

%% STEP 3: Train the softmax classifier

[sae2Features] = feedForwardAutoencoder(sae2OptTheta, hiddenSizeL2, ...

                                        hiddenSizeL1, sae1Features);

%  Randomly initialize the parameters

saeSoftmaxTheta = 0.005 * randn(hiddenSizeL2 * numClasses, 1);

lambda = 1e-4;

options.maxIter = 200;

softmaxModel = softmaxTrain(hiddenSizeL2, numClasses, lambda, ...

                            sae2Features, trainLabels, options);

% -------------------------------------------------------------------------

 saeSoftmaxOptTheta = softmaxModel.optTheta(:);

%% STEP 5: Finetune softmax model

% Implement the stackedAECost to give the combined cost of the whole model

% then run this cell.

% Initialize the stack using the parameters learned

stack = cell(2,1);

stack{1}.w = reshape(sae1OptTheta(1:hiddenSizeL1*inputSize), ...

                     hiddenSizeL1, inputSize);

stack{1}.b = sae1OptTheta(2*hiddenSizeL1*inputSize+1:2*hiddenSizeL1*inputSize+hiddenSizeL1);

stack{2}.w = reshape(sae2OptTheta(1:hiddenSizeL2*hiddenSizeL1), ...

                     hiddenSizeL2, hiddenSizeL1);

stack{2}.b = sae2OptTheta(2*hiddenSizeL2*hiddenSizeL1+1:2*hiddenSizeL2*hiddenSizeL1+hiddenSizeL2);

% Initialize the parameters for the deep model

[stackparams, netconfig] = stack2params(stack);

stackedAETheta = [ saeSoftmaxOptTheta ; stackparams ];

options.Method = 'lbfgs';

options.maxIter = 400;	  % Maximum number of iterations of L-BFGS to run

options.display = 'on';

[stackedAEOptTheta, cost] = minFunc( @(p) stackedAECost(p, ...

                                   hiddenSizeL2 , hiddenSizeL2, ...

                                   numClasses, netconfig, ...

                                   lambda, trainData, trainLabels), ...

                              stackedAETheta, options);

% -------------------------------------------------------------------------

%% STEP 6: Test

%  Instructions: You will need to complete the code in stackedAEPredict.m

%                before running this part of the code

%

testData = test_x';

[~, testLabels] = max(test_y, [], 2);

[pred] = stackedAEPredict(stackedAETheta, inputSize, hiddenSizeL2, ...

                          numClasses, netconfig, testData);

acc = mean(testLabels(:) == pred(:));

fprintf('Before Finetuning Test Accuracy: %0.3f%%\n', acc * 100);

[pred] = stackedAEPredict(stackedAEOptTheta, inputSize, hiddenSizeL2, ...

                          numClasses, netconfig, testData);

acc = mean(testLabels(:) == pred(:));

fprintf('After Finetuning Test Accuracy: %0.3f%%\n', acc * 100);

toc

pred(pred==10) = 0;

tmp = [(1:length(pred))' pred(:)];

csvwrite('LiFeiteng0824.csv',tmp)

test准确率基于Knn的pred-label

Kaggle—Digit Recognizer竞赛-LMLPHP

===============方法三、 784-200-200-10的Sparse AutoEncoder ===================

使用DeepLearnToolbox

%%

clear

close all

clc

%% load data label

load digitData

%%% pre-processing

%% ex2 train a X-X hidden unit SDAE and use it to initialize a FFNN

%  Setup and train a stacked denoising autoencoder (SDAE)

rng(0);

nDim = [784 200 200];

sae = saesetup(nDim);

sae.ae{1}.activation_function       = 'sigm';

sae.ae{1}.learningRate              = 1;

sae.ae{1}.inputZeroMaskedFraction   = 0.5;

sae.ae{2}.activation_function       = 'sigm';

sae.ae{2}.learningRate              = 1;

sae.ae{2}.inputZeroMaskedFraction   = 0.5;

% sae.ae{3}.activation_function       = 'sigm';

% sae.ae{3}.learningRate              = 0.8;

% sae.ae{3}.inputZeroMaskedFraction   = 0.5;

opts.numepochs =   30;

opts.batchsize = 100;

% opts.sparsityTarget = 0.05;%$LiFeiteng

% opts.nonSparsityPenalty = 1;

opts.dropoutFraction = 0.5;

sae = saetrain(sae, train_x, opts);

visualize(sae.ae{1}.W{1}(:,2:end)')

%% Use the SDAE to initialize a FFNN

nn = nnsetup([nDim 10]);

nn.activation_function              = 'sigm';%'sigm';

nn.learningRate                     = 1;

%add pretrained weights

nn.W{1} = sae.ae{1}.W{1};

nn.W{2} = sae.ae{2}.W{1};

%nn.W{3} = sae.ae{3}.W{1};

% Train the FFNN

fprintf('\n')

opts.numepochs =   40;

opts.batchsize = 100;

nn = nntrain(nn, train_x, train_y, opts);

%% test

[er, bad, pred] = nntest(nn, test_x, test_y);

pred(pred==10) = 0;

tmp = [(1:length(pred))' pred(:)];

csvwrite('LiFeiteng0824.csv',tmp)

start of the art！

==================================================================

排名200多好伤感！！！

Kaggle—Digit Recognizer竞赛-LMLPHP

Leaderboard上好多100%的，其实我也可以做到——作弊——把错误的部分逐一用肉眼扫下，更改test_label就可，不过这就没意思了。

Kaggle—Digit Recognizer竞赛-LMLPHP

Y. LeCun 维护的

THE MNIST DATABASE

最好成绩：

Kaggle—Digit Recognizer竞赛-LMLPHP

==============================

可以提高准确率的方法：

1.增加train的个数，对增加原始图像平移旋转等构造新图像；

2.对图像做预处理等；直接用PCA or ZCA白化会改变像素值范围；

3.卷积-池化等加入Deep Networks中去；

4.New Model。。。