Graph

Description

Container that represents the topology of the network in the form of a graph.

Initializing

def __init__(self, inputs, outputs, unsafe=False, nodesOnly=False, name=None):

Parameters

Parameter Allowed types Description Default
inputs Union[Node, list] Input nodes of the graph -
outputs Union[Node, list] Output nodes of the graph -
unsafe bool Enables/disables secure placement of tensors in the GPU False
nodesOnly bool If True, then information about the modules used will not be stored in the attributes of the graph False
name str Container name None

Explanations

unsafe - for some modules, it is possible to overwrite their tensors in the GPU to save memory. If the unsafe flag is not set, the graph will be validated for the absence of sections leading to corruption of the data tensors in the GPU caused by inplace flags;


nodesOnly - the data on the nodes included in the graph is stored in its nodes attribute, and when the nodesOnly flag is not set, information about the modules corresponding to these nodes is stored in a separate modules attribute.

Methods

gatherTopology

def gatherTopology(self, node, nodesOnly):
Functionality
Collects information about the graph topology, stores it in the class attributes, and, under certain conditions, validates the graph for the presence of sections that can lead to corruption of the data tensors in the GPU.

Parameters

Parameter Allowed types Description Default
node Node Node descendant class object that is the input node of the graph. -
nodesOnly bool If True, then information about the modules used will not be stored in the attributes of the graph -

Explanations

-

getBlueprint

def getBlueprint(self):
Functionality
Generates a dictionary containing all the necessary information about the structure of the graph for its recovery in the absence of the code.

Parameters

-

Explanations

-

getNodeByName

def getNodeByName(self, name):
Functionality
Returns the node of the graph (an object of the Node descendant class) by its name.

Parameters

Parameter Allowed types Description Default
name str Node of interest name -

Explanations

-

optimizeForShape

def optimizeForShape(self, shape, memlimit=None):
Functionality
Performs graph optimization for a given input size

Parameters

Parameter Allowed types Description Default
shape tuple The input shape to optimize for -
memlimit int The input shape to optimize for None

Explanations

-

updateData

def updateData(self, data):
Functionality
Starting from the input nodes, it makes a sequential passage through the graph, applying data operations assigned to nodes.

Parameters

Parameter Allowed types Description Default
data tensor Input data tensor -

Explanations

-

dataShapeFrom

def dataShapeFrom(self, shape):
Functionality
Calculates the shape of the data after it passes through the graph.

Parameters

Parameter Allowed types Description Default
shape Union[list, tuple] Input data shape -

Explanations
shape - tuple, if there is one input node and List[tuple], if there are several input nodes

graphDataShape

def graphDataShape(self, shape, onmodule):
Functionality
The basic method to calculate the shape of the data after it passes through the graph. It takes up an auxiliary function as an argument, providing the ability to perform an additional operation when calculating shapes.

Parameters

Parameter Allowed types Description Default
shape Union[list, tuple] Input data shape -
onmodule Callable Additional function -

Explanations
shape - see dataShapeFrom;


onmmodule - an additional function that will be used in the calculation of shapes; as a rule, this is a function for optimizing a node for its output shape

backward

def backward(self, grad, updParamGrads=True, updGrad=True, scale=1.0, momentum=0.0):
Functionality
Starting from the output nodes, it goes through the graph in the opposite direction, implementing the algorithm for the back propagation of the error — calculating the gradients on the parameters of the nodes and their input data.

Parameters

Parameter Allowed types Description Default
grad tensor Gradient tensor of the target function -
updParamGrads bool Flag that regulates the calculation of the gradient on the parameters of the node True
updGrad bool Flag that regulates the calculation of the gradient on the input data of the node True
scale float Scale: determines the scale the gradient 1.0
momentum float Momentum: determines how much gradient should be kept in this iteration 0.0

Explanations

-

gradShapeFrom

def gradShapeFrom(self, shape):
Functionality
Calculates the gradient shape on the input nodes of the graph.

Parameters

Parameter Allowed types Description Default
shape Union[list, tuple] Gradient shape of the output nodes of the graph -

Explanations

-

updateGrad

def updateGrad(self, grad):
Functionality
A stub

Parameters

Parameter Allowed types Description Default
grad tensor Gradient tensor -

Explanations

-

reset

def reset(self):
Functionality
Resets the internal state of the graph and its nodes (for details, see Node.reset)

Parameters

-

Explanations

-

clearTraverse

def clearTraverse(self):
Functionality
Unsets the traverse flagsf wdVisited and bwdVisited for all the graph nodes.

Parameters

-

Explanations

-

Examples


Let us compose a simple graph that performs the following operations:

  1. Two fully connected input layers 'linear0' and 'linear1'
  2. Splitting the data tensor after the first input layer into three blocks (‘split’)
  3. Concatenation of two blocks from the first tensor with the data tensor of the second fully connected layer (‘concat0’)
  4. Activation 'act' on the concatenation 'concat0'
  5. Final concatenation of the remaining block from 'split' and the data after the ‘act’ activation (‘concat1’)

Necessary imports:

>>> import numpy as np
>>> from PuzzleLib.Backend import gpuarray
>>> from PuzzleLib.Containers import Graph
>>> from PuzzleLib.Modules import Linear, Split, Concat, Activation, relu

Info

gpuarray is necessary for the correct placement of the tensor in the GPU

Creating nodes of the future graph:

>>> v1 = Linear(10, 5, name="linear0").node()
>>> h1 = Split(axis=1, sections=(2, 2, 1), name="split").node(v1)
>>> v2 = Linear(10, 5, name="linear1").node()
>>> h2 = Concat(axis=1, name="concat0").node((h1, [1, 2]), v2)
>>> h3 = Activation(relu, name="act").node(h2)
>>> h4 = Concat(axis=1, name="concat1").node((h1, 0), h3)

Initialization of the graph itself:

>>> graph = Graph(inputs=[v1, v2], outputs=h4)

Synthetic data preparation and uploading it into the GPU:

>>> v1data = gpuarray.to_gpu(np.random.randint(0, 255, (5, 10)).astype(np.float32))
>>> v2data = gpuarray.to_gpu(np.random.randint(0, 255, (5, 10)).astype(np.float32))

Graph data processing:

>>> graph([v1data, v2data])

Data preparation (loading from the GPU into RAM using the get method) from the 'split' layer, and its output to the screen. The round method hereinafter is used only to simplify the visual perception of the tensors:

>>> splitData = [d.get() for d in graph["split"].data]
>>> for i, data in enumerate(splitData):
...     print("split block {} results: \n{}".format(i, data.round(1)))
split block 0 results:
[[ -0.2 -59.7  11.9]
 [-37.7 -75.4  21.1]
 [-21.5 -23.8  30.9]
 [-53.7 -60.4  46.5]
 [-69.7 -60.   25.8]]
split block 1 results:
[[ -9.7  -8.7]
 [ 47.2 -49.4]
 [ 44.7 -69.7]
 [ 21.2 -93.9]
 [  0.8 -31. ]]
split block 2 results:
[[130. ]
 [141.3]
 [ 99.6]
 [135. ]
 [112. ]]

Results from the other layers:

>>> # 'linear1' layer results
>>> print(graph["linear1"].data.get().round(1))
[[-111.7  -16.6]
[-117.8   23.6]
[-101.4   29.1]
[ -86.7   30. ]
[-129.4  -22.5]]
>>> # 'concat0' layer results
>>> print(graph["concat0"].data.get().round(1))
[[  -9.7   -8.7  130.  -111.7  -16.6]
[  47.2  -49.4  141.3 -117.8   23.6]
[  44.7  -69.7   99.6 -101.4   29.1]
[  21.2  -93.9  135.   -86.7   30. ]
[   0.8  -31.   112.  -129.4  -22.5]]
>>> # 'act' layer results
>>> print(graph["act"].data.get().round(1))
[[  0.    0.  130.    0.    0. ]
[ 47.2   0.  141.3   0.   23.6]
[ 44.7   0.   99.6   0.   29.1]
[ 21.2   0.  135.    0.   30. ]
[  0.8   0.  112.    0.    0. ]]
>>> # 'concat1' layer results
>>> print(graph["concat1"].data.get().round(1))
[[ -0.2 -59.7  11.9   0.    0.  130.    0.    0. ]
[-37.7 -75.4  21.1  47.2   0.  141.3   0.   23.6]
[-21.5 -23.8  30.9  44.7   0.   99.6   0.   29.1]
[-53.7 -60.4  46.5  21.2   0.  135.    0.   30. ]
[-69.7 -60.   25.8   0.8   0.  112.    0.    0. ]]