是否可以在未存储在像LMDB这样的数据源中的数据集上运行caffe模型?

问题描述

我有2组图像补丁数据，即训练和测试集.这两个都已写入LMDB文件.我正在使用Caffe在此数据上运行卷积神经网络.

I have 2 sets of image patches data i.e. training and testing sets. Both of these have been written to LMDB files. I am running convolutional neurall network on this data using Caffe.

问题在于存储在硬盘上的数据占用了大量空间，并且妨碍了我通过故意添加噪声来引入更多训练数据以使我的模型更健壮的工作.

The problem is that the data stored on hard disk is occupying considerable amount of space and is hampering my efforts to introduce more training data with deliberate noise addition to make my model more robust.

有没有一种方法可以将程序中的图像补丁直接发送到CNN(在Caffe中)而无需将其存储在LMDB中?我目前正在使用python从图像中为训练数据集生成补丁.

Is there a way where I can send image patches from my program directly to the CNN (in Caffe) without storing them in LMDB? I am currently using python to generate patches from the images for the training data set.

推荐答案

您可以编写自己的python数据层.请参见此处的讨论以及视频流输入数据层的实现此处.

You can write your own python data layer. See discussions here and implementation for of input data layer for video stream here.

基本上，您将需要在网络描述层中添加以下内容:

Basically you will need add to you network description layer like:

layer {
  type: 'Python'
  name: 'data'
  top: 'data'
  top: 'label'
  python_param {
    # the module name -- usually the filename -- that needs to be in $PYTHONPATH
    module: 'filename'
    # the layer name -- the class name in the module
    layer: 'CustomInputDataLayer'
  }
}

并在Python中实现图层接口:

and implement the layer interface in Python:

class CustomInputDataLayer(caffe.Layer):
    def setup(self):
         ...

    def reshape(self, bottom, top)
        top[0].reshape(BATCH_SIZE, your_data.shape)
        top[1].reshape(BATCH_SIZE, your_label.shape)

    def forward(self, bottom, top):
        # assign output
        top[0].data[...] = your_data
        top[1].data[...] = your_label

    def backward(self, top, propagate_down, bottom):
        pass

这篇关于是否可以在未存储在像LMDB这样的数据源中的数据集上运行caffe模型?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！