Optimizer¶
Warning
Documentation for the module is under development.
Info
This module is intended primarily for developers who want to better understand the structure of the library, as well as those who are going to implement their own modules.
Description¶
It is a base class for the optimizer family. Optimizers are a set of mathematical methods for solving the optimization problem for the error function.
Initializing¶
def __init__(self, nodeinfo=None):
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| nodeinfo | NodeInfo | Object containing information about the computational node | None |
Explanations
nodeinfo - special object containing information about the machine on which the calculations are performed; necessary for parallelization of calculations.
Methods¶
setAttr¶
def setAttr(self, name, attr):
Functionality
Sets the value of an attribute, for example, a learning rate. It is used as a helper method in derived classes.
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| name | str | Parameter name | - |
| attr | float | Parameter name | - |
Explanations
-
getAttrDict¶
def getAttrDict(self):
Functionality
Returns the optimizer attributes as a dictionary.
Parameters
-
Explanations
-
addHook¶
def addHook(self, hook):
Functionality
Adds the hook function to the optimizer.
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| hook | Hook | Hook | - |
Explanations
-
setupOn¶
def setupOn(self, mod, useGlobalState=False):
Functionality
Installs the optimizer to a trainable neural network.
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| mod | Module | Trainable neural network | - |
| useGlobalState | bool | Condition for optimizing the placement of tensors in the GPU | False |
Explanations
useGlobalState - this flag is responsible for two main points: if it is set, then, firstly, tensors of the neural network variables will be redistributed in the GPU so that weights can be optimized with a minimum number of queries to the GPU, and secondly, the preparation of the optimizer to the parallel mode of calculations will be implemented, if it is initiated.
setupGlobalState¶
def setupGlobalState(self, vartable):
Functionality
Performs the redistribution of tensors of neural network variables in the GPU in such a way so that weights can be optimized with a minimum number of queries to the GPU, and prepares the optimizer to the parallel mode of calculations, if it is initiated.
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| vartable | dict | Dictionary of neural network variables | - |
Explanations
vartable - information presented in the form of a dictionary on the variables of the neural network on which the optimizer is installed. Variables mean, for example, the weights and offsets of layers that will be updated during training.
setupLocalStates¶
def setupGlobalState(self, vartable):
Functionality
Prepares the attributes of internal use for calculations.
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| vartable | dict | Dictionary of neural network variables | - |
Explanations
vartable - information on the variables of the neural network on which the optimizer is installed, presented as a dictionary. Variables mean, for example, the weights and offsets of layers that will be updated during training.
zeroGradParams¶
def zeroGradParams(self):
Functionality
A wrapper function around the operation query to reset the accumulated gradients.
Parameters
-
Explanations
-
zeroGradGlobalParams¶
def zeroGradParams(self):
Functionality
Performs the operation of zeroing the accumulated gradients when working in global mode (the useGlobalState flag is set when the setupOn method is called).
Parameters
-
Explanations
-
zeroGradLocalParams¶
def zeroGradParams(self):
Functionality
Performs the operation of zeroing the accumulated gradients when working in local mode (the useGlobalState flag is not set when the setupOn method is called).
Parameters
-
Explanations
-
setupState¶
def setupState(self, var):
Functionality
An abstract helper method implemented in derived classes. It generates information about tensors placed in the GPU.
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| var | Variable | Object of the Variable library class, which is a wrapper around the GPU tensors | - |
Explanations
-
update¶
def update(self, useStreams=False, sync=True):
Functionality
A wrapper around the method for updating the variables of a trainable neural network.
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| useStreams | bool | Flag responsible for the use of multistreaming when working in local mode | False |
| sync | bool | Flag responsible for stream synchronization | True |
Explanations
-
updateGlobalState¶
def update(self):
Functionality
Performs the operation of updating variables of the trainable neural network when working in local mode (the useGlobalState flag is not set when the setupOn method is called).
Parameters
-
Explanations
-
updateLocalStates¶
def update(self, useStreams, sync):
Functionality
Performs the operation of updating variables of the trainable neural network when working in local mode (the useGlobalState flag is not set when the setupOn method is called).
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| useStreams | bool | Flag responsible for use of multithreading | - |
| sync | bool | Flag responsible for stream synchronization | - |
Explanations
-
updateVar¶
def updateVar(self, var, state, stream=None):
Functionality
An abstract method that must be implemented in derived classes. It is this method that has to define the logic according to which the transferred variable tensor will be updated.
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| var | Variable | Object of the Variable library class, which is a wrapper around the GPU tensors | - |
| state | str | - | - |
| stream | - | - | None |
Explanations
-
save¶
def save(self, hdf, name=None):
Functionality
Saves the state of the gradients to an hdf-file.
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| hdf | Union[str, h5py.File] | File path or an hdf-file object | - |
| name | str | Name of the current state | None |
Explanations
-
load¶
def load(self, hdf, name=None):
Functionality
Loads the state of the gradients from an hdf-file.
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| hdf | Union[str, h5py.File] | File path or an hdf-file object | - |
| name | str | Name of the current state | None |
Explanations
-
ensureHdf¶
def ensureHdf(file, mode):
Functionality
A wrapper method that opens and returns either an hdf-file in case the file path is specified in the file parameter, or trivially returns the file parameter if it is already a file object.
Parameters
| Parameter | Allowed types | Description | Default |
|---|---|---|---|
| file | Union[str, h5py.File] | File path or an hdf file object | - |
| mode | str | File open mode ('w','r','a', and etc.) | None |
Explanations
-