(更新:Posted最终结果作为单独的答案)

我开始尝试了解如何使用scikit模型进行培训。我已经试验了虹膜,MNIST等众所周知的数据集-它们都是结构良好的数据,随时可以使用。这是我第一次尝试根据自己的原始数据构建模型,结果却不尽人意。

我选择使用的数据是过去3年的NHSTA's crash data

这是数据的快照,可让您了解字段,而不必下载数据。

python - scikit的MLPClassifier(和其他分类器)中的培训得分较低-LMLPHP

我的第一个实验很简单-尝试建立一个给出“许可证状态代码”和“年龄”的模型,并尝试预测性别(男或女)。

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn import metrics
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import learning_curve
from sklearn.model_selection import ShuffleSplit
import tensorflow.contrib.learn as skflow
from tensorflow.contrib.learn.python.learn.estimators import run_config
from sklearn.svm import SVC
import pickle, seaborn


def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None,
                        n_jobs=1, train_sizes=np.linspace(.1, 1.0, 5)):
    #http://scikit-learn.org/stable/auto_examples/model_selection/plot_learning_curve.html
    plt.figure()
    plt.title(title)
    if ylim is not None:
        plt.ylim(*ylim)
    plt.xlabel("Training examples")
    plt.ylabel("Score")
    train_sizes, train_scores, test_scores = learning_curve(
        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)
    plt.grid()

    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,
                     train_scores_mean + train_scores_std, alpha=0.1,
                     color="r")
    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,
                     test_scores_mean + test_scores_std, alpha=0.1, color="g")
    plt.plot(train_sizes, train_scores_mean, 'o-', color="r",
             label="Training score")
    plt.plot(train_sizes, test_scores_mean, 'o-', color="g",
             label="Cross-validation score")

    plt.legend(loc="best")
    plt.show()

#MAIN

crashes = pd.read_csv("crashes.csv", nrows=100000)

# drop useless cols

crashes.drop(["Year","Case Individual ID", "Case Vehicle ID", "Transported
By", "Injury Location", "Role Type"],axis=1, inplace=True)

crashes = crashes [pd.notnull(crashes['Age'])]
crashes = crashes[crashes.Age >= 10 ] # There are ages < 10 - likely junk data. I don't think they drive


# lets drop rows that are empty
crashes = crashes [pd.notnull(crashes['License State Code'])]
crashes = crashes [pd.notnull(crashes['Injury Severity'])]
crashes = crashes [pd.notnull(crashes['Safety Equipment'])]
crashes = crashes [pd.notnull(crashes['Sex'])]

# converts text fields to numerical values
le = LabelEncoder()
crashes = crashes[crashes.columns[:]].apply(le.fit_transform)
crashes = crashes._get_numeric_data()

# lets plot a heat map to show correlation
corr = crashes.corr()
ax = seaborn.heatmap (corr, xticklabels=corr.columns.values,
yticklabels=corr.columns.values, annot=True)
plt.setp( ax.xaxis.get_majorticklabels(), rotation=45 )
plt.setp( ax.yaxis.get_majorticklabels(), rotation=-45 )
plt.show()

crashes_train, crashes_test = train_test_split(crashes, test_size = 0.2)
Y_train = crashes_train['Sex']
X_train =  crashes_train[[ 'Age',  'License State Code']]
Y_test = crashes_test['Sex']
X_test =  crashes_test[[ 'Age', 'License State Code']]


names_train  = crashes_train.columns.values

print "train size ",len (X_train)
print "test size",len (X_test)
#
# cls = RandomForestClassifier(verbose = True)
#
cls = MLPClassifier(hidden_layer_sizes=(10,10,10), max_iter=500, alpha=1e-4,
                  solver='sgd', verbose=10, tol=1e-4, random_state=1,
                  learning_rate_init=0.01)

#cls = tf.contrib.learn.DNNClassifier(feature_columns=feats,
#                                               hidden_units=[50, 50, 50],
#                                              n_classes=3)

#
#

#cls = SVC(verbose = True)

print "Fitting..."
cls.fit(X_train, Y_train)

plot_learning_curve(cls,"Crash Learning", X_train, Y_train)


print("Training set score: %f" % cls.score(X_train, Y_train))
print("Test set score: %f" % cls.score(X_test, Y_test))


我尝试了多种模型(从RandomForest,到SVC到MLP等)-它们都给出了大约0.56的训练得分和0.6倍的损失

最后,这是在当前配置中为MLP生成的图:
python - scikit的MLPClassifier(和其他分类器)中的培训得分较低-LMLPHP

这是当我切换到RandomForest时的图。 python - scikit的MLPClassifier(和其他分类器)中的培训得分较低-LMLPHP

看起来RandomForest中的分数下降了,但总体而言,其结局类似于MLP。我在做错什么,如何改进这种方法?谢谢

编辑:基于以下两个答案,我对所有列之间的相关性做了一个热图(在删除了明显无用的列之后)-那很糟糕,但这是正确的方法吗?我也可以进行PCA,但是如果基本场间相关性太差,是否表明该数据集在很大程度上无法用于预测?

python - scikit的MLPClassifier(和其他分类器)中的培训得分较低-LMLPHP

最佳答案

我的第一个实验很简单-尝试建立一个给定的模型
  尝试使用“许可州代码”和“年龄”来预测性别(男或女)。


好吧,这不是那么简单。您不能简单地获取任何数据并尝试进行预测。数据至少需要关联。

要做好的几件事:


绘制数据。绘制这三个变量(年龄与性别,许可证状态代码与性别),看它们是否具有一定的相关性。
计算变量之间的相关性,例如Person's Correlation Coefficient
使用您拥有的所有功能以及RandomForest / DecisionTree分类器,它们具有名为feature_importances_的属性。此属性告诉您数据集中哪些功能最重要(当然,取决于模型)
功能的重要性(越高,功能越重要)。
阅读有关MLP和分类器一般工作原理的更多信息。


分类算法仅将输入数据映射到类别。但是,如果您的输入和输出之间完全没有关系,则此任务不可行。特征选择是机器学习中非常重要的领域。从wikipedia


  在机器学习和统计中,特征选择(也称为变量选择,属性选择或变量子集选择)是选择用于模型构建的相关特征子集(变量,预测变量)的过程。

08-24 22:26