Optimizer

Warning

Documentation for the module is under development.

Info

This module is intended primarily for developers who want to better understand the structure of the library, as well as those who are going to implement their own modules.

Description

It is a base class for the optimizer family. Optimizers are a set of mathematical methods for solving the optimization problem for the error function.

Initializing

def __init__(self, nodeinfo=None):

Parameters

Parameter Allowed types Description Default
nodeinfo NodeInfo Object containing information about the computational node None

Explanations

nodeinfo - special object containing information about the machine on which the calculations are performed; necessary for parallelization of calculations.

Methods

setAttr

def setAttr(self, name, attr):

Functionality

Sets the value of an attribute, for example, a learning rate. It is used as a helper method in derived classes.

Parameters

Parameter Allowed types Description Default
name str Parameter name -
attr float Parameter name -

Explanations

-

getAttrDict

def getAttrDict(self):

Functionality

Returns the optimizer attributes as a dictionary.

Parameters

-

Explanations

-

addHook

def addHook(self, hook):

Functionality

Adds the hook function to the optimizer.

Parameters

Parameter Allowed types Description Default
hook Hook Hook -

Explanations

-

setupOn

def setupOn(self, mod, useGlobalState=False):

Functionality

Installs the optimizer to a trainable neural network.

Parameters

Parameter Allowed types Description Default
mod Module Trainable neural network -
useGlobalState bool Condition for optimizing the placement of tensors in the GPU False

Explanations

useGlobalState - this flag is responsible for two main points: if it is set, then, firstly, tensors of the neural network variables will be redistributed in the GPU so that weights can be optimized with a minimum number of queries to the GPU, and secondly, the preparation of the optimizer to the parallel mode of calculations will be implemented, if it is initiated.

setupGlobalState

def setupGlobalState(self, vartable):

Functionality

Performs the redistribution of tensors of neural network variables in the GPU in such a way so that weights can be optimized with a minimum number of queries to the GPU, and prepares the optimizer to the parallel mode of calculations, if it is initiated.

Parameters

Parameter Allowed types Description Default
vartable dict Dictionary of neural network variables -

Explanations

vartable - information presented in the form of a dictionary on the variables of the neural network on which the optimizer is installed. Variables mean, for example, the weights and offsets of layers that will be updated during training.

setupLocalStates

def setupGlobalState(self, vartable):

Functionality

Prepares the attributes of internal use for calculations.

Parameters

Parameter Allowed types Description Default
vartable dict Dictionary of neural network variables -

Explanations

vartable - information on the variables of the neural network on which the optimizer is installed, presented as a dictionary. Variables mean, for example, the weights and offsets of layers that will be updated during training.

zeroGradParams

def zeroGradParams(self):

Functionality

A wrapper function around the operation query to reset the accumulated gradients.

Parameters

-

Explanations

-

zeroGradGlobalParams

def zeroGradParams(self):

Functionality

Performs the operation of zeroing the accumulated gradients when working in global mode (the useGlobalState flag is set when the setupOn method is called).

Parameters

-

Explanations

-

zeroGradLocalParams

def zeroGradParams(self):

Functionality

Performs the operation of zeroing the accumulated gradients when working in local mode (the useGlobalState flag is not set when the setupOn method is called).

Parameters

-

Explanations

-

setupState

def setupState(self, var):

Functionality

An abstract helper method implemented in derived classes. It generates information about tensors placed in the GPU.

Parameters

Parameter Allowed types Description Default
var Variable Object of the Variable library class, which is a wrapper around the GPU tensors -

Explanations

-

update

def update(self, useStreams=False, sync=True):

Functionality

A wrapper around the method for updating the variables of a trainable neural network.

Parameters

Parameter Allowed types Description Default
useStreams bool Flag responsible for the use of multistreaming when working in local mode False
sync bool Flag responsible for stream synchronization True

Explanations

-

updateGlobalState

def update(self):

Functionality

Performs the operation of updating variables of the trainable neural network when working in local mode (the useGlobalState flag is not set when the setupOn method is called).

Parameters

-

Explanations

-

updateLocalStates

def update(self, useStreams, sync):

Functionality

Performs the operation of updating variables of the trainable neural network when working in local mode (the useGlobalState flag is not set when the setupOn method is called).

Parameters

Parameter Allowed types Description Default
useStreams bool Flag responsible for use of multithreading -
sync bool Flag responsible for stream synchronization -

Explanations

-

updateVar

def updateVar(self, var, state, stream=None):

Functionality

An abstract method that must be implemented in derived classes. It is this method that has to define the logic according to which the transferred variable tensor will be updated.

Parameters

Parameter Allowed types Description Default
var Variable Object of the Variable library class, which is a wrapper around the GPU tensors -
state str - -
stream - - None

Explanations

-

save

def save(self, hdf, name=None):

Functionality

Saves the state of the gradients to an hdf-file.

Parameters

Parameter Allowed types Description Default
hdf Union[str, h5py.File] File path or an hdf-file object -
name str Name of the current state None

Explanations

-

load

def load(self, hdf, name=None):

Functionality

Loads the state of the gradients from an hdf-file.

Parameters

Parameter Allowed types Description Default
hdf Union[str, h5py.File] File path or an hdf-file object -
name str Name of the current state None

Explanations

-

ensureHdf

def ensureHdf(file, mode):

Functionality

A wrapper method that opens and returns either an hdf-file in case the file path is specified in the file parameter, or trivially returns the file parameter if it is already a file object.

Parameters

Parameter Allowed types Description Default
file Union[str, h5py.File] File path or an hdf file object -
mode str File open mode ('w','r','a', and etc.) None

Explanations

-