在Google ML Engine的微调工作中，某些训练配置导致NaN损失，并因此导致错误。我希望能够忽略这些试验，并继续使用不同的参数进行微调。
我正在使用带有fail_on_nan_loss = False的NanTensorHook，当不执行任何并行试验时（maxParallelTrials：1），它在ML Engine中可以成功运行，但是在多个并行试验中（maxParallelTrials：3）则失败。
有人遇到过此错误吗？关于如何解决的任何想法？
这是我的配置文件：
trainingInput:
scaleTier: CUSTOM
masterType: standard
workerType: standard
parameterServerType: standard
workerCount: 4
parameterServerCount: 1
hyperparameters:
goal: MAXIMIZE
maxTrials: 5
maxParallelTrials: 3
enableTrialEarlyStopping: False
hyperparameterMetricTag: auc
params:
- parameterName: learning_rate
type: DOUBLE
minValue: 0.0001
maxValue: 0.01
scaleType: UNIT_LOG_SCALE
- parameterName: optimizer
type: CATEGORICAL
categoricalValues:
- Adam
- Adagrad
- Momentum
- SGD
- parameterName: batch_size
type: DISCRETE
discreteValues:
- 128
- 256
- 512
这是我设置NanTensorHook的方法：
hook = tf.train.NanTensorHook(loss,fail_on_nan_loss=False)
train_op = tf.contrib.layers.optimize_loss(
loss=loss, global_step=tf.train.get_global_step(),
learning_rate=lr, optimizer=optimizer)
model_fn = tf.estimator.EstimatorSpec(mode=mode, loss=loss,
eval_metric_ops=eval_metric_ops, train_op=train_op,
training_hooks=[hook])
我收到的错误消息是：
Hyperparameter Tuning Trial #4 Failed before any other successful
trials were completed. The failed trial had parameters: optimizer=SGD,
batch_size=128, learning_rate=0.00075073617775056709, . The trial's ror
message was: The replica worker 1 exited with a non-zero status of 1.
Termination reason: Error. Traceback (most recent call last): [...]
File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 421, in
train_and_evaluate executor.run() File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 522, in run
getattr(self, task_to_run)() File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 532, in
run_worker return self._start_distributed_training() File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 715, in
_start_distributed_training saving_listeners=saving_listeners) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/estimator.py", line 352, in train
loss = self._train_model(input_fn, hooks, saving_listeners) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/estimator.py", line 891, in
_train_model _, loss = mon_sess.run([estimator_spec.train_op,
estimator_spec.loss]) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 546, in
run run_metadata=run_metadata) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1022,
in run run_metadata=run_metadata) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1113,
in run raise six.reraise(*original_exc_info) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1098,
in run return self._sess.run(*args, **kwargs) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1178,
in run run_metadata=run_metadata)) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/basic_session_run_hooks.py", line
617, in after_run raise NanLossDuringTrainingError
NanLossDuringTrainingError: NaN loss during training. The replica
worker 3 exited with a non-zero status of 1. Termination reason: Error.
Traceback (most recent call last): [...] File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 421, in
train_and_evaluate executor.run() File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 522, in run
getattr(self, task_to_run)() File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 532, in
run_worker return self._start_distributed_training() File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 715, in
_start_distributed_training saving_listeners=saving_listeners) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/estimator.py", line 352, in train
loss = self._train_model(input_fn, hooks, saving_listeners) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/estimator.py", line 891, in
_train_model _, loss = mon_sess.run([estimator_spec.train_op,
estimator_spec.loss]) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 546, in
run run_metadata=run_metadata) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1022,
in run run_metadata=run_metadata) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1113,
in run raise six.reraise(*original_exc_info) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1098,
in run return self._sess.run(*args, **kwargs) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1178,
in run run_metadata=run_metadata)) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/basic_session_run_hooks.py", line
617, in after_run raise NanLossDuringTrainingError
NanLossDuringTrainingError: NaN loss during training.
谢谢大家！
最佳答案
超参数调整作业中的不同试验在运行时是隔离的。因此，为一个试验添加的钩子将不受其他试验中其他钩子的影响。
我怀疑该问题是由用于试验的超参数的特定组合引起的。为了确认这一点，我建议您针对失败的试验使用超参数值运行常规培训，然后查看错误是否会再次发生。
您能否将项目编号和职位ID发送给[email protected]，我们可以进行更多调查。