PReLU

Description

Info

Parent class: Module

Derived classes: -

This module implements a parametric rectified linear unit (PReLU). It is defined by the following formula:

$$\begin{equation}\label{eq:prelu} \text{PReLU}(x) = \begin{cases} x, & \text{ if } x \geq 0 \\ ax, & \text{ if } x < 0 \end{cases} \end{equation}$$

where a is a trainable parameter.

Initializing

def __init__(self, maps, inplace=False, sharedMaps=False, name=None):

Parameters

Parameter	Allowed types	Description	Default
maps	int	Number of maps in the tensor	-
inplace	bool	If True, the output tensor will be written in memory in the place of the input tensor	False
sharedMaps	bool	Uniformity of the coefficient for all maps	False
name	str	Layer name	None

Explanations

inplace - flag showing whether additional memory resources should be allocated for the result. If True, then the output tensor will be written in the place of the input tensor in memory, which can negatively affect the network, if the input tensor takes part in calculations on other branches of the graph.

---

sharedMaps - if this parameter is set to True, the $a$ coefficient in $prelu(x)=max(ax, 0)$ function will be the same for all maps.

Examples

Necessary imports.

import numpy as np
from PuzzleLib.Backend import gpuarray
from PuzzleLib.Modules import PRelu

Info

gpuarray is required to properly place the tensor in the GPU

batchsize, maps, h, w = 1, 3, 5, 5
np.random.seed(1234)
data = gpuarray.to_gpu(np.random.randint(-9, 9, (batchsize, maps, h, w)).astype(np.float32))
print(data)

[[[[ 2. -7.  6. -6.  5.]
   [-7. -5. -8.  1. -7.]
   [ 4. -6.  3. -4.  8.]
   [ 7. -1.  4.  2.  6.]
   [ 7. -9.  8. -9.  8.]]

  [[ 0. -6. -5. -8.  2.]
   [ 8. -3. -7. -2.  3.]
   [ 3. -9. -1.  6.  4.]
   [ 4.  0.  6. -5. -2.]
   [-2.  8. -2. -4. -7.]]

  [[-3. -1.  8.  0.  6.]
   [ 2.  3. -6.  1.  2.]
   [-7. -1. -8.  5. -5.]
   [ 1. -6.  8.  6. -9.]
   [ 4.  7.  0. -6.  7.]]]]

Let us initialize the module with default parameters. The slopes attribute is the exact trainable coefficient, whose initial value is 0.25 for all maps:

preluMod = PRelu(maps)
print(preluMod.slopes)

[0.25 0.25 0.25]

To demonstrate the operation of the module, let us assign our coefficients:

slopes = np.array([0.5, 0, 0.25], dtype=np.float32)
preluMod.setVar("slopes", Variable(gpuarray.to_gpu(slopes)))
print(preluMod.slopes)

[0.5  0.   0.25]

Then:

print(preluMod(data))

[[[[ 2.   -3.5   6.   -3.    5.  ]
   [-3.5  -2.5  -4.    1.   -3.5 ]
   [ 4.   -3.    3.   -2.    8.  ]
   [ 7.   -0.5   4.    2.    6.  ]
   [ 7.   -4.5   8.   -4.5   8.  ]]

  [[ 0.   -0.   -0.   -0.    2.  ]
   [ 8.   -0.   -0.   -0.    3.  ]
   [ 3.   -0.   -0.    6.    4.  ]
   [ 4.    0.    6.   -0.   -0.  ]
   [-0.    8.   -0.   -0.   -0.  ]]

  [[-0.75 -0.25  8.    0.    6.  ]
   [ 2.    3.   -1.5   1.    2.  ]
   [-1.75 -0.25 -2.    5.   -1.25]
   [ 1.   -1.5   8.    6.   -2.25]
   [ 4.    7.    0.   -1.5   7.  ]]]]

If sharedMaps=True the coefficient will be the same for all maps:

preluMod = PRelu(maps, sharedMaps=True)
print(preluMod.slopes)

[0.25]