Handler designed to simplify the process of model inference for the user by eliminating the need to manually prescribe a sequence of actions. It is a kind of a wrapper function around the operations on data.
In handlers, depending on the location of the data, splitting can be performed in the following ways:
- data is placed on the disk: first, data is split into macrobatches - blocks that are entirely placed in the GPU, whereafter the macrobatch is split into smaller batches, which are then fed directly to the model input;
- data has already been placed in the GPU: it is split into batches, which are then fed directly to the input of the model.
def __init__(self, mod, onBatchFinish=None, batchsize=128):
|mod||Module||Trainable neural network||-|
|onBatchFinish||callable||Function that will be called upon completion of processing of a data batch||None|
|batchsize||int||Size of a data batch||128|
All the basic methods of handlers can be found in the documentation for the parent class Handler.
def calcFromHost(self, data, macroBatchSize=10000, onMacroBatchFinish=None):
Wrapper function around the handleFromHost() method of the Handler parent class, which runs the data directly through the model (inference). Returns an array of network responses.
|macroBatchSize||int||Size of a macrobatch. The data will be split into macrobatches sized macrobatchSize||10000|
|onMacroBatchFinish||callable||Function that will be called after processing the macrobatch||None|
def calc(self, data, target):
Wrapper function around the handle() method of the Handler parent class, which runs the data directly through the model (inference). Returns an array of network responses.
|data||GPUArray||Data tensor hosted in the GPU||-|
def handleBatch(self, batch, idx, resid, state):
Root method of the inference handler. It processes the transmitted batch with the model and writes the results to
|batch||GPUArray||Data tensor hosted in the GPU||-|
|idx||int||Index number of the data batch||-|
|state||dict||Dictionary containing model predictions||-|