General description of the Optimizers section

The section contains various optimizers. After the optimizer is created, it needs to be installed on a trainable module. For each trainable variable, the optimizer creates its own internal state, state, which allows it to calculate corrections to the true gradient.

All internal states can be combined into one global using the key useGlobalState, which aligns all states into one long array in memory, as well as the variables themselves.

Optimizers that are implemented in PuzzleLib:

• AdaDelta
• AdaGrad
• Adam
• MomentumSGD
• NesterovSGD
• RMSProp
• RMSPropGraves
• SGD
• SMORMS3