本文介绍了ValueError:检查目标时出错:预期密集_2的形状为(1,),但数组的形状为(14,)的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我正在尝试以分布式方式训练分类模型.我正在使用Yahoo开发的TensorflowOnSpark库.我使用的示例 github链接

I am trying to train a classification model in a distributed way. I am using TensorflowOnSpark library developed by Yahoo. The example I am using github link

我正在使用mnist以外的数据集,该数据集在github链接中提到的示例中使用.我正在使用的该数据集经过预处理后的尺寸如下(260000,28047),并且类(标签)的范围为0:13.

I am using dataset other than mnist which is used in example mentioned in the github link. This dataset I am using is of dimensions as follows after preprocessing (260000,28047) and also the classes(labels) range from 0:13.

>>> import os
>>> import tensorflow as tf
>>> from tensorflow.python import keras
>>> from tensorflow.python.keras import backend as K
>>> from tensorflow.python.keras.models import Sequential, load_model, save_model
>>> from tensorflow.python.keras.layers import Dense, Dropout
>>> from tensorflow.python.keras.callbacks import LambdaCallback, TensorBoard
>>> from tensorflow.python.saved_model import builder as saved_model_builder
>>> from tensorflow.python.saved_model import tag_constants
>>> from tensorflow.python.saved_model.signature_def_utils_impl import predict_signature_def
>>> from tensorflowonspark import TFNode
>>> from pyspark.context import SparkContext
>>> from pyspark.conf import SparkConf
>>> from tensorflowonspark import TFCluster
>>> import numpy
>>>
>>>
...
>>>
>>> data = sc.textFile('ReducedFeatures.tsv')
>>>
>>> data = data.map(lambda l: l.encode("UTF8", "ignore").split('\t'))
>>>
>>> labels = data.map(lambda x: x[1])
>>> data = data.map(lambda x: x[19:28066])
>>>
>>> header = data.first()
>>> data = data.filter(lambda line: line != header)
>>> label_header = labels.first()
>>> labels = labels.filter(lambda line: line != label_header)
>>>
>>> #convert values to float
... convertToFloat = lambda data: [float(str(x)) for x in data]
>>> dataset = data.map(convertToFloat)
>>> labels = labels.map(lambda x:float(x))
>>>
>>>
>>> labels = labels.map(lambda x: keras.utils.to_categorical(x, num_classes=14))
>>>
>>>
>>> # zip the data as tuple
... dataRDD = dataset.zip(labels)
>>>
>>> sampledata = dataRDD.take(20)
>>>
>>> model = Sequential()
>>> model.add(Dense(512, activation='relu', input_shape=(28047,)))
>>> model.add(Dropout(0.2))
>>> model.add(Dense(512, activation='relu'))
>>> model.add(Dropout(0.2))
>>> model.add(Dense(14, activation='softmax'))
>>>
>>> model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense (Dense)                (None, 512)               14360576
_________________________________________________________________
dropout (Dropout)            (None, 512)               0
_________________________________________________________________
dense_1 (Dense)              (None, 512)               262656
_________________________________________________________________
dropout_1 (Dropout)          (None, 512)               0
_________________________________________________________________
dense_2 (Dense)              (None, 14)                7182
=================================================================
Total params: 14,630,414
Trainable params: 14,630,414
Non-trainable params: 0
_________________________________________________________________
>>>
>>> model.compile(loss='sparse_categorical_crossentropy',
...                       optimizer=tf.train.RMSPropOptimizer(learning_rate=0.001),
...                       metrics=['accuracy'])
>>>
>>> print("model.inputs: {}".format(model.inputs))
model.inputs: [<tf.Tensor 'dense_input:0' shape=(?, 28047) dtype=float32>]
>>> print("model.outputs: {}".format(model.outputs))
model.outputs: [<tf.Tensor 'dense_2/Softmax:0' shape=(?, 14) dtype=float32>]
>>>
>>> def generate_rdd_data(dataRDD):
...     while True:
...             feature_vector = []
...             lbls = []
...             for item in dataRDD:
...                     #record = item[0]
...                     feature_vector.append(item[0])
...                     lbls.append(item[1])
...             features = numpy.array(feature_vector).astype('float32')
...             labels = numpy.stack(lbls).astype('float32')
...             return (features, labels)
...
>>>
>>> feat, lbls = generate_rdd_data(sampledata)
>>> lbls
array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],
      dtype=float32)
>>>
>>>
>>> x_train = tf.placeholder(tf.float32, [None, 28047], name="x_train")
>>> y_train = tf.placeholder(tf.float32, [None, 14], name="y_train")
>>>
...
>>> model.fit_generator(generator=generate_rdd_data(sampledata),steps_per_epoch=200,epochs=5,verbose=1,validation_data=(x_train, y_train))

还请看一下下面的追溯,我对标签做了OneHotEncoding,同时添加了outuput层,如代码和模型摘要所示,它是14.

Also Please take a look at the traceback below, I did OneHotEncoding for the labels and also while adding the outuput layers it is 14 as you can see it in the code and model summary.

   >     Traceback (most recent call last):
    >       File "<stdin>", line 1, in <module>
    >       File "/usr/lib/python2.7/site-packages/tensorflow/python/keras/engine/training.py",
    > line 2177, in fit_generator
    >         initial_epoch=initial_epoch)
    >       File "/usr/lib/python2.7/site-packages/tensorflow/python/keras/engine/training_generator.py",
    > line 104, in fit_generator
    >         val_x, val_y, val_sample_weights)
    >       File "/usr/lib/python2.7/site-packages/tensorflow/python/keras/engine/training.py",
    > line 992, in _standardize_user_data
    >         class_weight, batch_size)
    >       File "/usr/lib/python2.7/site-packages/tensorflow/python/keras/engine/training.py",
    > line 1154, in _standardize_weights
    >         exception_prefix='target')
    >       File "/usr/lib/python2.7/site-packages/tensorflow/python/keras/engine/training_utils.py",
    > line 332, in standardize_input_data
    >         ' but got array with shape ' + str(data_shape))
    >     ValueError: Error when checking target: expected dense_2 to have shape (1,) but got array with shape (14,)

但是当我开始使用生成器方法的输出进行训练时.对于类,我得到以下尺寸错误.请帮助

But when I start training using the output from generator method. I get the following dimension errors for the classes. Please help

推荐答案

您使用了错误的损失函数.

You are using wrong loss function.

"sparse_categorical_crossentropy".但是您的输出是onehot编码的[0,0,... 1,0].

"sparse_categorical_crossentropy" is used when your input labels are integers like [0, 1, 2, 3, ...13]. But your output is onehot encoded [0,0,...1,0].

因此使用损失函数"categorical_crossentropy".

model.compile(loss='categorical_crossentropy',
              optimizer=tf.train.RMSPropOptimizer(learning_rate=0.001),
              metrics=['accuracy'])

这篇关于ValueError:检查目标时出错:预期密集_2的形状为(1,),但数组的形状为(14,)的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

09-03 07:46