AdamW
classkeras.optimizers.AdamW(
learning_rate=0.001,
weight_decay=0.004,
beta_1=0.9,
beta_2=0.999,
epsilon=1e-07,
amsgrad=False,
clipnorm=None,
clipvalue=None,
global_clipnorm=None,
use_ema=False,
ema_momentum=0.99,
ema_overwrite_frequency=None,
name="adamw",
**kwargs
)
Optimizer that implements the AdamW algorithm.
AdamW optimization is a stochastic gradient descent method that is based on adaptive estimation of first-order and second-order moments with an added method to decay weights per the techniques discussed in the paper, 'Decoupled Weight Decay Regularization' by Loshchilov, Hutter et al., 2019.
According to Kingma et al., 2014, the underying Adam method is "computationally efficient, has little memory requirement, invariant to diagonal rescaling of gradients, and is well suited for problems that are large in terms of data/parameters".
Arguments
keras.optimizers.schedules.LearningRateSchedule
instance, or
a callable that takes no arguments and returns the actual value to
use. The learning rate. Defaults to 0.001
.0.9
.0.999
.False
.use_ema=True
.
This is the momentum to use when computing
the EMA of the model's weights:
new_average = ema_momentum * old_average + (1 - ema_momentum) *
current_variable_value
.use_ema=True
. Every ema_overwrite_frequency
steps of iterations,
we overwrite the model variable by its moving average.
If None, the optimizer
does not overwrite model variables in the middle of training, and you
need to explicitly overwrite the variables at the end of training
by calling optimizer.finalize_variable_values()
(which updates the model
variables in-place). When using the built-in fit()
training loop,
this happens automatically after the last epoch,
and you don't need to do anything.None
. If a float, the scale factor will
be multiplied the loss before computing gradients, and the inverse of
the scale factor will be multiplied by the gradients before updating
variables. Useful for preventing underflow during mixed precision
training. Alternately, keras.optimizers.LossScaleOptimizer
will
automatically set a loss scale factor.References
adam
amsgrad
.