Keras(TensorFlow、CPU):在循环中训练序列模型会占用内存

本文介绍了Keras(TensorFlow、CPU):在循环中训练序列模型会占用内存的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我正在尝试循环训练 1000 倍的序列模型.在每个循环中，我的程序都会泄漏内存，直到用完并出现 OOM 异常.

I am trying to train 1000x of Sequential models in a loop. In every loop my program leaks memory until I run out and get an OOM exception.

我之前已经问过类似的问题(连续训练多个 Sequential 模型会变慢)

I already asked a similar question before(Training multiple Sequential models in a row slows down)

并且已经看到其他人遇到类似的问题(Keras: 进行超参数网格搜索时内存不足)

and have seen others in similar problems (Keras: Out of memory when doing hyper parameter grid search)

解决方案总是在您使用完模型后将 K.clear_session() 添加到您的代码中.所以我在上一个问题中这样做了，但我仍然在泄漏内存

and the solution is always to add K.clear_session() to your code after you have finished using the model. So I did that in my previous question and I am still leaking memory

这是重现问题的代码.

import random
import time
from keras.models import Sequential
from keras.layers import Dense
from keras import backend as K
import tracemalloc


def run():
    tracemalloc.start()
    num_input_nodes = 12
    num_hidden_nodes = 8
    num_output_nodes = 1

    random_numbers = random.sample(range(1000), 50)
    train_x, train_y = create_training_dataset(random_numbers, num_input_nodes)

    for i in range(100):
        snapshot = tracemalloc.take_snapshot()
        for j in range(10):
            start_time = time.time()
            nn = Sequential()
            nn.add(Dense(num_hidden_nodes, input_dim=num_input_nodes, activation='relu'))
            nn.add(Dense(num_output_nodes))
            nn.compile(loss='mean_squared_error', optimizer='adam')
            nn.fit(train_x, train_y, nb_epoch=300, batch_size=2, verbose=0)
            K.clear_session()
            print("Iteration {iter}. Current time {t}. Took {elapsed} seconds".
                  format(iter=i*10 + j + 1, t=time.strftime('%H:%M:%S'), elapsed=int(time.time() - start_time)))

        top_stats = tracemalloc.take_snapshot().compare_to(snapshot, 'lineno')

        print("[ Top 5 differences ]")
        for stat in top_stats[:5]:
            print(stat)


def create_training_dataset(dataset, input_nodes):
    """
    Outputs a training dataset (train_x, train_y) as numpy arrays.
    Each item in train_x has 'input_nodes' number of items while train_y items are of size 1
    :param dataset: list of ints
    :param input_nodes:
    :return: (numpy array, numpy array), train_x, train_y
    """
    data_x, data_y = [], []
    for i in range(len(dataset) - input_nodes - 1):
        a = dataset[i:(i + input_nodes)]
        data_x.append(a)
        data_y.append(dataset[i + input_nodes])
    return numpy.array(data_x), numpy.array(data_y)

run()

这是我从第一个内存调试打印得到的输出

Here is the output I get from the first memory debug print

/tensorflow/python/framework/ops.py:121: size=3485 KiB (+3485 KiB), count=42343 (+42343)/tensorflow/python/framework/ops.py:1400: size=998 KiB (+998 KiB), count=8413 (+8413)/tensorflow/python/framework/ops.py:116: size=888 KiB (+888 KiB), count=32468 (+32468)/tensorflow/python/framework/ops.py:1185: size=795 KiB (+795 KiB), count=3179 (+3179)/tensorflow/python/framework/ops.py:2354: size=599 KiB (+599 KiB), count=5886 (+5886)

系统信息:

python 3.5
keras (1.2.2)
张量流(1.0.0)

序列模型

Keras(TensorFlow、CPU):在循环中训练序列模型会占用内存

问题描述

推荐答案