tf.keras.callbacks.BackupAndRestore( backup_dir, save_freq="epoch", delete_checkpoint=True, save_before_preemption=False )
Callback to back up and restore the training state.
BackupAndRestore callback is intended to recover training from an
interruption that has happened in the middle of a
Model.fit execution, by
backing up the training states in a temporary checkpoint file (with the help
tf.train.CheckpointManager), at the end of each epoch. Each backup
overwrites the previously written checkpoint file, so at any given time
there is at most one such checkpoint file for backup/restoring purpose.
If training restarts before completion, the training state (which includes
Model weights and epoch number) is restored to the most recently saved
state at the beginning of a new
Model.fit run. At the completion of a
Model.fit run, the temporary checkpoint file is deleted.
Note that the user is responsible to bring jobs back after the interruption. This callback is important for the backup and restore mechanism for fault tolerance purpose, and the model to be restored from a previous checkpoint is expected to be the same as the one used to back up. If user changes arguments passed to compile or fit, the checkpoint saved for fault tolerance can become invalid.
Model.fitredoes any partial work during the unfinished epoch in which the training got restarted (so the work done before the interruption doesn't affect the final model state).
Model.fitis used with
tf.distribute, it supports
>>> class InterruptingCallback(tf.keras.callbacks.Callback): ... def on_epoch_begin(self, epoch, logs=None): ... if epoch == 4: ... raise RuntimeError('Interrupting!') >>> callback = tf.keras.callbacks.BackupAndRestore(backup_dir="/tmp/backup") >>> model = tf.keras.models.Sequential([tf.keras.layers.Dense(10)]) >>> model.compile(tf.keras.optimizers.SGD(), loss='mse') >>> try: ... model.fit(np.arange(100).reshape(5, 20), np.zeros(5), epochs=10, ... batch_size=1, callbacks=[callback, InterruptingCallback()], ... verbose=0) ... except: ... pass >>> history = model.fit(np.arange(100).reshape(5, 20), np.zeros(5), ... epochs=10, batch_size=1, callbacks=[callback], ... verbose=0) >>> # Only 6 more epochs are run, since first training got interrupted at >>> # zero-indexed epoch 4, second training will continue from 4 to 9. >>> len(history.history['loss']) 6
Besides the option to save at the end of every epoch or every N steps, if
you are doing distributed training with
tf.distribute.MultiWorkerMirroredStrategy on Google Cloud Platform or
Google Borg, you can also use the
to enable saving a checkpoint right before a worker gets preempted
by other jobs and training gets interrupted. See
tf.distribute.experimental.PreemptionCheckpointHandler for more details.
backup_dir = os.path.join(working_dir, 'backup'). This is the directory in which the system stores temporary files to recover the model from jobs terminated unexpectedly. The directory cannot be reused elsewhere to store other files, e.g. by the
BackupAndRestorecallback of another training run, or by another callback (e.g.
ModelCheckpoint) of the same training.
'epoch', integer, or
False. When set to
'epoch'the callback saves the checkpoint at the end of each epoch. When set to an integer, the callback saves the checkpoint every
Falseif only using preemption checkpointing (with
BackupAndRestorecallback works by saving a checkpoint to back up the training state. If
delete_checkpoint=True, the checkpoint will be deleted after training is finished. Use
Falseif you'd like to keep the checkpoint for future usage.
tf.distribute.MultiWorkerMirroredStrategyon Google Cloud Platform or Google Borg for now.