SoftMax

Description

Info

Parent class: Module

Derived classes: -

This module applies the softmax function to the input tensor. Softmax is a generalization of the logistic function for the multidimensional case, determined by the formula:

\begin{equation}\label{eq:softmax} \sigma(z)_i = \frac{e^{z_i}}{\sum \limits_{k=1}^{K} e^{z_k}} \end{equation}

The softmax function is used along the tensor depth (along the maps) and scales their values so that the elements lie in the range [0, 1] and add up to 1, i.e. if there is a tensor of shape (N, C, H, W), where N - batch size, C - number of maps (channels), H - map height, W - map width, provided that each element of the maps is x_{nchw}\in{R}, then when passing through the softmax function x_{nchw}\in[0, 1], whereby \displaystyle\sum_{c=1}^C x_{nchw} = 1.

Softmax is often used for the last layer of deep neural networks for classification tasks. To train the neural network, cross entropy is used as the loss function.

Initializing

def __init__(self, name=None):

Parameters

Parameter Allowed types Description Default
name str Layer name None

Explanations

-

Examples

Necessary imports.

>>> import numpy as np
>>> from PuzzleLib.Backend import gpuarray
>>> from PuzzleLib.Modules import SoftMax

Info

gpuarray is required to properly place the tensor in the GPU

For simplicity, let us take maps of unit height:

>>> batchsize, maps, h, w = 1, 3, 1, 3
>>> data = gpuarray.to_gpu(np.random.randn(batchsize, maps, 1, 3).astype(np.float32))
[[[[ 0.47143516 -1.1909757   1.432707  ]]

  [[-0.3126519  -0.72058874  0.8871629 ]]

  [[ 0.8595884  -0.6365235   0.01569637]]]]
>>> softmax = SoftMax()
>>> outdata = softmax(data)
>>> print(outdata)
[[[[0.3412047  0.2303298  0.5488582 ]]

  [[0.15577234 0.36866897 0.31807783]]

  [[0.50302297 0.4010012  0.13306393]]]]
>>> print(np.sum(outdata.get(), axis=1))
[[[1. 1. 1.]]]