tensorflow - TFRecords用于嵌入式文本数据

对于Uni的一个项目，我正在使用TensorFlow中的Neural Nets来实现一个问答系统（目前是bAbI数据集Task 5，请参见https://research.fb.com/downloads/babi/），并且我想在输入管道中使用TFRecords。

我的想法是，用TFRecords术语表示的一个Example应该包括问题的上下文，问题本身，答案以及支持的句子编号（int指向上下文中能够回答问题的最重要的句子）。这是我定义函数的方式：

def make_example(context, question, answer, support):
 ex = tf.train.SequenceExample()

 fl_context = ex.feature_lists.feature_list["context"]
 fl_question = ex.feature_lists.feature_list["question"]
 fl_answer = ex.feature_lists.feature_list["answer"]
 ex.context.feature["support"].int64_list.value.append(support)

 for token in context:
    fl_context.feature.add().int64_list.value.append(token)
 for qWord in question:
    fl_question.feature.add().int64_list.value.append(qWord)
 for ansWord in answer:
    fl_answer.feature.add().int64_list.value.append(ansWord)
 fl_support.feature.add().int64_list.value.append(support)

return ex

但是，在传递上下文，问题和答案之前，我想嵌入单词并通过其GloVe向量表示它们，即通过（m，d）矩阵表示，其中m是句子中标记的数量，d是每个单词向量具有的维数。当我得到时，我的make_example函数似乎无法很好地处理此问题：

theTypeError: (array([[ -9.58490000e-01,   1.73210000e-01,
2.51650000e-01,
 -5.61450000e-01,  -1.21440000e-01,   1.54350000e+00,
 -1.28930000e+00,  -9.77790000e-01,  -1.35480000e-01,
 -6.06930000e-01,  -1.37810000e+00,   6.33470000e-01,
  1.33160000e-01,   2.46320000e-01,   6.60260000e-01,
 -4.46130000e-02,   4.09510000e-01,  -7.61670000e-01,
  4.67530000e-01,  -6.67810000e-01,   2.99850000e-01,
 -2.74810000e-01,  -5.47990000e-01,  -8.56820000e-01,
  5.30880000e-02,  -2.01700000e+00,   7.48530000e-01,
 -1.27830000e-01,   1.32050000e-01,  -2.19450000e-01,
  2.29830000e+00,  -3.17680000e-01,  -8.64940000e-01,
 -1.08630000e-01,  -8.13770000e-02,  -7.03420000e-01,
  4.60000000e-01,  -3.34730000e-01,   4.37030000e-02,
 -7.55080000e-01,  -6.89710000e-01,   7.14380000e-01,
 -8.35950000e-02,   1.58620000e-02,  -5.23850000e-01,
  1.72520000e-01,  -4.98740000e-01,   2.30810000e-01,
 -3.64690000e-01,   1.5 has type <class 'tuple'>, but expected one of:
(<class 'int'>,)

指向上面的fl_context.feature.add().int64_list.value.append(token) ...有人可以指出我误解了TFRecords的概念的地方，并给我一些如何解决该问题的建议吗？
我已经搜索了大量学习资料，但是通常TFRecords上的示例都包含图像数据。到目前为止，我的引用是https://medium.com/@TalPerry/getting-text-into-tensorflow-with-the-dataset-api-ffb832c8bec6和http://web.stanford.edu/class/cs20si/lectures/notes_09.pdf。

在此先多谢！

最佳答案

我的问题的解决方案可以在这里找到：https://github.com/simonada/q-and-a-tensorflow/blob/master/src/Q%26A%20with%20TF-%20TFRecords%20and%20Eager%20Execution.ipynb

我的方法如下：

将文本存储到一个csv文件中：每行（上下文，问题，答案）
在我的情况下，定义一个将序列转换为tf_example的函数

def sequence_to_tf_example(context, question, answer):
    context_ids= vectorize(context, False, word_to_index)
    question_ids= vectorize(question, False, word_to_index)
    answer_ids= vectorize(answer, True, word_to_index)
    ex = tf.train.SequenceExample()

    context_tokens = ex.feature_lists.feature_list["context"]
    question_tokens = ex.feature_lists.feature_list["question"]
    answer_tokens = ex.feature_lists.feature_list["answer"]

    for token in context_ids:
        context_tokens.feature.add().int64_list.value.append(token)
    for token in question_ids:
        question_tokens.feature.add().int64_list.value.append(token)
    for token in answer_ids:
        #print(token)
        answer_tokens.feature.add().int64_list.value.append(token)

    return ex

定义写功能

def write_example_to_tfrecord(context, question, answer, tfrecord_file, writer):
      example= sequence_to_tf_example(context, question, answer)
      writer.write(example.SerializeToString())

def write_data_to_tf_record(filename):
    file_csv= filename+'.csv'
    file_tfrecords= filename+'.tfrecords'
    with open(file_csv) as csvfile:
       readCSV = csv.reader(csvfile, delimiter=',')
       next(readCSV) #skip header
       writer= tf.python_io.TFRecordWriter(file_tfrecords)
       for row in readCSV:
       write_example_to_tfrecord(row[0], row[1], row[2], file_tfrecords, writer)
       writer.close()

定义读取功能

def read_from_tfrecord(ex):

   sequence_features = {
     "context": tf.FixedLenSequenceFeature([], dtype=tf.int64),
     "question": tf.FixedLenSequenceFeature([], dtype=tf.int64),
     "answer": tf.FixedLenSequenceFeature([], dtype=tf.int64)
 }

# Parse the example (returns a dictionary of tensors)
_, sequence_parsed = tf.parse_single_sequence_example(
    serialized=ex,
    sequence_features=sequence_features
)

return {"context": sequence_parsed['context'], "question": sequence_parsed['question'],
        "answer": sequence_parsed['answer']}

创建数据集

def make_dataset(path, batch_size=128):
  '''
  Makes  a Tensorflow dataset that is shuffled, batched and parsed.
  '''
   # Read a tf record file. This makes a dataset of raw TFRecords
   dataset = tf.data.TFRecordDataset([path])
   # Apply/map the parse function to every record. Now the dataset is a bunch of dictionaries of Tensors
   dataset =  dataset.map(read_from_tfrecord)
   #Shuffle the dataset
   dataset = dataset.shuffle(buffer_size=10000)

# specify padding for each tensor seperatly
 dataset = dataset.padded_batch(batch_size, padded_shapes={
    "context": tf.TensorShape([None]),
    "question": tf.TensorShape([None]),
    "answer": tf.TensorShape([None])
})

return dataset