PReLU

Description

Info

Parent class: Module

Derived classes: -

This module implements a parametric rectified linear unit (PReLU). It is defined by the following formula:

\begin{equation}\label{eq:prelu} \text{PReLU}(x) = \begin{cases} x, & \text{ if } x \geq 0 \\ ax, & \text{ otherwise } \end{cases} \end{equation}

where a is a trainable parameter.

Initializing

def __init__(self, maps, inplace=False, sharedMaps=False, name=None):

Parameters

Parameter Allowed types Description Default
maps int Number of maps in the tensor -
inplace bool If True, the output tensor will be written in memory in the place of the input tensor. False
sharedMaps bool Uniformity of the coefficient for all maps False
name str Layer name None

Explanations

inplace - flag showing whether additional memory resources should be allocated for the result. If True, then the output tensor will be written in the place of the input tensor in memory, which can negatively affect the network, if the input tensor takes part in calculations on other branches of the graph.


sharedMaps - if this parameter is set to True, the a coefficient in prelu(x)=max(ax, 0) function will be the same for all maps.

Examples

Necessary imports.

>>> import numpy as np
>>> from PuzzleLib.Backend import gpuarray
>>> from PuzzleLib.Modules import PRelu

Info

gpuarray is required to properly place the tensor in the GPU

>>> batchsize, maps, h, w = 1, 3, 5, 5
>>> np.random.seed(1234)
>>> data = gpuarray.to_gpu(np.random.randint(-9, 9, (batchsize, maps, h, w)).astype(np.float32))
>>> print(data)
[[[[ 2. -7.  6. -6.  5.]
   [-7. -5. -8.  1. -7.]
   [ 4. -6.  3. -4.  8.]
   [ 7. -1.  4.  2.  6.]
   [ 7. -9.  8. -9.  8.]]

  [[ 0. -6. -5. -8.  2.]
   [ 8. -3. -7. -2.  3.]
   [ 3. -9. -1.  6.  4.]
   [ 4.  0.  6. -5. -2.]
   [-2.  8. -2. -4. -7.]]

  [[-3. -1.  8.  0.  6.]
   [ 2.  3. -6.  1.  2.]
   [-7. -1. -8.  5. -5.]
   [ 1. -6.  8.  6. -9.]
   [ 4.  7.  0. -6.  7.]]]]

Let us initialize the module with default parameters. The slopes attribute is the exact trainable coefficient, whose initial value is 0.25 for all maps:

>>> preluMod = PRelu(maps)
>>> print(preluMod.slopes)
[0.25 0.25 0.25]

To demonstrate the operation of the module, let us assign our coefficients:

>>> slopes = np.array([0.5, 0, 0.25], dtype=np.float32)
>>> preluMod.setVar("slopes", Variable(gpuarray.to_gpu(slopes)))
>>> print(preluMod.slopes)
[0.5  0.   0.25]

Then:

>>> preluMod(data)
[[[[ 2.   -3.5   6.   -3.    5.  ]
   [-3.5  -2.5  -4.    1.   -3.5 ]
   [ 4.   -3.    3.   -2.    8.  ]
   [ 7.   -0.5   4.    2.    6.  ]
   [ 7.   -4.5   8.   -4.5   8.  ]]

  [[ 0.   -0.   -0.   -0.    2.  ]
   [ 8.   -0.   -0.   -0.    3.  ]
   [ 3.   -0.   -0.    6.    4.  ]
   [ 4.    0.    6.   -0.   -0.  ]
   [-0.    8.   -0.   -0.   -0.  ]]

  [[-0.75 -0.25  8.    0.    6.  ]
   [ 2.    3.   -1.5   1.    2.  ]
   [-1.75 -0.25 -2.    5.   -1.25]
   [ 1.   -1.5   8.    6.   -2.25]
   [ 4.    7.    0.   -1.5   7.  ]]]]

If sharedMaps=True the coefficient will be the same for all maps:

>>> preluMod = PRelu(maps, sharedMaps=True)
>>> print(preluMod.slopes)
[0.25]