Numpy 到 TFrecords:有没有更简单的方法来处理来自 tfrecords 的批处理输入?

问题描述

我的问题是关于如何从多个(或分片的)tfrecord 中获取批量输入.我已经阅读了示例 https://github.com/tensorflow/models/blob/master/inception/inception/image_processing.py#L410.基本的pipeline是，以训练集为例，(1)首先生成一系列tfrecords(例如，train-000-of-005, train-001-of-005, ...), (2) 从这些文件名中，生成一个列表并将它们送入 tf.train.string_input_producer 以获得一个队列，(3) 同时生成一个 tf.RandomShuffleQueue 做其他事情，(4)使用 tf.train.batch_join 生成批量输入.

My question is about how to get batch inputs from multiple (or sharded) tfrecords. I've read the example https://github.com/tensorflow/models/blob/master/inception/inception/image_processing.py#L410. The basic pipeline is, take the training set as as example, (1) first generate a series of tfrecords (e.g., train-000-of-005, train-001-of-005, ...), (2) from these filenames, generate a list and fed them into the tf.train.string_input_producer to get a queue, (3) simultaneously generate a tf.RandomShuffleQueue to do other stuff, (4) using tf.train.batch_join to generate batch inputs.

我觉得这很复杂，我不确定这个程序的逻辑.就我而言，我有一个 .npy 文件列表，我想生成分片 tfrecords(多个单独的 tfrecords，而不仅仅是一个大文件).这些 .npy 文件中的每一个都包含不同数量的正样本和负样本(2 类).一种基本方法是生成一个单独的大 tfrecord 文件.但是文件太大(~20Gb).所以我求助于分片 tfrecords.有没有更简单的方法来做到这一点?谢谢.

I think this is complex, and I'm not sure the logic of this procedure. In my case, I have a list of .npy files, and I want to generate sharded tfrecords(multiple seperated tfrecords, not just one single large file). Each of these .npy files contains different number of positive and negative samples (2 classes). A basic method is to generate one single large tfrecord file. But the file is too large (~20Gb). So I resort to sharded tfrecords. Are there any simpler way to do this? Thanks.

推荐答案

使用 Dataset API 简化了整个过程.以下是两个部分:(1):将 numpy 数组转换为 tfrecords 和 (2,3,4):读取 tfrecords 以生成批次.

The whole process is simplied using the Dataset API. Here are both the parts: (1): Convert numpy array to tfrecords and (2,3,4): read the tfrecords to generate batches.

    def npy_to_tfrecords(...):
       # write records to a tfrecords file
       writer = tf.python_io.TFRecordWriter(output_file)

       # Loop through all the features you want to write
       for ... :
          let say X is of np.array([[...][...]])
          let say y is of np.array[[0/1]]

         # Feature contains a map of string to feature proto objects
         feature = {}
         feature['X'] = tf.train.Feature(float_list=tf.train.FloatList(value=X.flatten()))
         feature['y'] = tf.train.Feature(int64_list=tf.train.Int64List(value=y))

         # Construct the Example proto object
         example = tf.train.Example(features=tf.train.Features(feature=feature))

         # Serialize the example to a string
         serialized = example.SerializeToString()

         # write the serialized objec to the disk
         writer.write(serialized)
      writer.close()

2.使用数据集 API (tensorflow >=1.2) 读取 tfrecords:

    # Creates a dataset that reads all of the examples from filenames.
    filenames = ["file1.tfrecord", "file2.tfrecord", ..."fileN.tfrecord"]
    dataset = tf.contrib.data.TFRecordDataset(filenames)
    # for version 1.5 and above use tf.data.TFRecordDataset

    # example proto decode
    def _parse_function(example_proto):
      keys_to_features = {'X':tf.FixedLenFeature((shape_of_npy_array), tf.float32),
                          'y': tf.FixedLenFeature((), tf.int64, default_value=0)}
      parsed_features = tf.parse_single_example(example_proto, keys_to_features)
     return parsed_features['X'], parsed_features['y']

    # Parse the record into tensors.
    dataset = dataset.map(_parse_function)  

    # Shuffle the dataset
    dataset = dataset.shuffle(buffer_size=10000)

    # Repeat the input indefinitly
    dataset = dataset.repeat()  

    # Generate batches
    dataset = dataset.batch(batch_size)

    # Create a one-shot iterator
    iterator = dataset.make_one_shot_iterator()

    # Get batch X and y
    X, y = iterator.get_next()

这篇关于Numpy 到 TFrecords:有没有更简单的方法来处理来自 tfrecords 的批处理输入?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！