RMSPropGraves

Description

Info

Parent class: Optimizer

Derived classes: -

This module implements a modification of the RMSProp, algorithm proposed in an article by Alex Graves.

In this algorithm, as in Adam, we introduce a term of sum that adds inertia to the optimization process.

\begin{equation} m_t = \alpha{m_{t-1}} + (1 - \alpha)g_t \end{equation}

\begin{equation} \upsilon_t = \alpha{\upsilon_{t-1}} + (1 - \alpha)g_t^2 \end{equation}

where \alpha - momentum exponential decay rate.

We also introduce a moving average for parameter update values:

\begin{equation} \Delta\theta_t = \gamma\Delta\theta_{t-1} - \eta\frac{g_t}{\sqrt{m_t - \upsilon_t^2 + \epsilon}} \end{equation}

\begin{equation} \theta_{t + 1} = \theta_t + \Delta\theta_t \end{equation}

where \gamma - parameter update exponential decay rate.

Initializing

def __init__(self, learnRate=1e-4, alpha=0.95, momRate=0.9, epsilon=1e-4, nodeinfo=None):

Parameters

Parameter Allowed types Description Default
learnRate float Learning rate 1e-4
alpha float Momentum exponential decay rate 0.95
momRate float Parameter update exponential decay rate 0.9
epsilon float Smoothing parameter 1e-4
nodeinfo NodeInfo Object containing information about the computational node None

Explanations

-

Examples


Necessary imports:

import numpy as np
from PuzzleLib.Optimizers import RMSPropGraves
from PuzzleLib.Backend import gpuarray

Info

gpuarray is required to properly place the tensor in the GPU.

Let us set up a synthetic training dataset:

data = gpuarray.to_gpu(np.random.randn(16, 128).astype(np.float32))
target = gpuarray.to_gpu(np.random.randn(16, 1).astype(np.float32))

Let us set up a synthetic training dataset:

optimizer = RMSPropGraves(learnRate=0.001)

Suppose there is already some net network defined, for example, through Graph, then in order to install the optimizer on the network, the following is required:

optimizer.setupOn(net, useGlobalState=True)

Info

You can read more about optimizer methods and their parameters in the description of the Optimizer parent class

Moreover, let there be some loss error function, inherited from Cost, calculating its gradient as well. Then we get the implementation of the optimization process:

for i in range(100):
... predictions = net(data)
... error, grad = loss(predictions, target)

... optimizer.zeroGradParams()
... net.backward(grad)
... optimizer.update()

... if (i + 1) % 5 == 0:
...   print("Iteration #%d error: %s" % (i + 1, error))