我有一个fit()函数,该函数使用ModelCheckpoint()回调保存模型(如果比以前的模型更好),使用save_weights_only = False,因此可以保存整个模型。这应该允许我以后通过使用load_model()恢复训练。
不幸的是,在save()/load_model()往返中的某处,度量标准值未保留-例如,val_loss设置为inf。这意味着当训练恢复时,在第一个时期之后,ModelCheckpoint()将始终保存模型,这几乎总是比上一届 session 的前任冠军更糟糕。
我确定可以在继续训练之前设置ModelCheckpoint()的当前最佳值,如下所示:
myCheckpoint = ModelCheckpoint(...)
myCheckpoint.best = bestValueSoFar
显然,我可以监视所需的值,将它们写到文件中,然后在恢复时再次读取它们,但是鉴于我是Keras新手,我想知道自己是否错过了明显的事情。
最佳答案
最后,我很快编写了自己的回调,该回调可跟踪最佳训练值,以便重新加载它们。看起来像这样:
# State monitor callback. Tracks how well we are doing and writes
# some state to a json file. This lets us resume training seamlessly.
#
# ModelState.state is:
#
# { "epoch_count": nnnn,
# "best_values": { dictionary with keys for each log value },
# "best_epoch": { dictionary with keys for each log value }
# }
class ModelState(callbacks.Callback):
def __init__(self, state_path):
self.state_path = state_path
if os.path.isfile(state_path):
print('Loading existing .json state')
with open(state_path, 'r') as f:
self.state = json.load(f)
else:
self.state = { 'epoch_count': 0,
'best_values': {},
'best_epoch': {}
}
def on_train_begin(self, logs={}):
print('Training commences...')
def on_epoch_end(self, batch, logs={}):
# Currently, for everything we track, lower is better
for k in logs:
if k not in self.state['best_values'] or logs[k] < self.state['best_values'][k]:
self.state['best_values'][k] = float(logs[k])
self.state['best_epoch'][k] = self.state['epoch_count']
with open(self.state_path, 'w') as f:
json.dump(self.state, f, indent=4)
print('Completed epoch', self.state['epoch_count'])
self.state['epoch_count'] += 1
然后,在fit()函数中,如下所示:
# Set up the model state, reading in prior results info if available
model_state = ModelState(path_to_state_file)
# Checkpoint the model if we get a better result
model_checkpoint = callbacks.ModelCheckpoint(path_to_model_file,
monitor='val_loss',
save_best_only=True,
verbose=1,
mode='min',
save_weights_only=False)
# If we have trained previously, set up the model checkpoint so it won't save
# until it finds something better. Otherwise, it would always save the results
# of the first epoch.
if 'best_values' in model_state.state:
model_checkpoint.best = model_state.state['best_values']['val_loss']
callback_list = [model_checkpoint,
model_state]
# Offset epoch counts if we are resuming training. If you don't do
# this, only epochs-initial_epochs epochs will be done.
initial_epoch = model_state.state['epoch_count']
epochs += initial_epoch
# .fit() or .fit_generator, etc. goes here.