Import ONNX models

Preliminary steps

ONNX generators may generate complicated models, in order to take into account for example dynamic size or shape calculation, from previous operator outputs dimensions. This can be the case even when the graph is static and the dimensions are known in the ONNX model. While such model may be imported in DL frameworks using standard operators/layers, it would be vastly sub-optimal, as some part of the graph would require unnecessary dynamic allocation, and would be very hard to optimize for inference on embedded platforms.

For this reason, we do not always try to allow proper import of such graph in N2D2 as is. While some simplifications may be handled directly in N2D2, we recommend using the ONNX Simplifier tool on your ONNX model before importing it into N2D2.

With an INI file

It is possible to include an ONNX model inside a N2D2 INI file, as part of a graph. This is particularly useful to add pre-processing and post-processing to an existing ONNX model. Below is an example with the MobileNet ONNX model provided by Google:

$BATCH_SIZE=256

DefaultModel=Frame_CUDA

; Database
[database]
Type=ILSVRC2012_Database
RandomPartitioning=0
Learn=1.0
BackgroundClass=1  ; Necessary for Google MobileNet pre-trained models

; Environment
[sp]
SizeX=224
SizeY=224
NbChannels=3
BatchSize=${BATCH_SIZE}

[sp.Transformation-1]
Type=RescaleTransformation
Width=256
Height=256

[sp.Transformation-2]
Type=PadCropTransformation
Width=224
Height=224

[sp.Transformation-3]
Type=ColorSpaceTransformation
ColorSpace=RGB

[sp.Transformation-4]
Type=RangeAffineTransformation
FirstOperator=Minus
FirstValue=127.5
SecondOperator=Divides
SecondValue=127.5

; Here, we insert an ONNX graph in the N2D2 flow the same way as a regular Cell
[onnx]
Input=sp
Type=ONNX
File=mobilenet_v1_1.0_224.onnx

; We can add targets to ONNX cells
[MobilenetV1/Predictions/Softmax:0.Target-Top5]
TopN=5

A N2D2 target must be associated to the output layer of the ONNX model in order to compute the score in N2D2.

Note

The imported ONNX layer names in N2D2 is the name of their first output ( the operator “name” field is indeed optional in the ONNX standard). You can easily find the ONNX cell names after running N2D2 or by opening the ONNX graph in a graph viewer like NETRON (https://lutzroeder.github.io/netron/).

Once the INI file including the ONNX model is ready, the following command must be used to run N2D2 in test (inference) mode:

n2d2 MobileNet_ONNX.ini -seed 1 -w /dev/null -test

There required command line arguments for running INI files including ONNX model are described above:

Command line argument	Description
-seed 1	Initial seed, necessary for test without learning before
-w /dev/null	No external weight loading: trained weight values are contained in the ONNX model

ONNX INI section type

The table below summarizes the parameters of an ONNX INI section. To declare an ONNX section, the Type parameter must be equal to ONNX. The name of the section can be arbitrary.

Option [default value]	Description
Type=ONNX	ONNX section type
File	Path to the ONNX file
Ignore []	Space-separated list of ONNX operators to ignore during import
IgnoreInputSize [0]	If true (1), the input size specified in the ONNX model is ignored and the N2D2 StimuliProvider size is used
Transpose [0]	If true (1), the first 2 dimensions are transposed in the whole ONNX graph (1D graph are first interpreted as 2D with the second dimension equal to 1)

Transpose option usage

The Transpose option allows to transpose the first two dimensions of a whole graph. This can be used in practice to used transposed inputs (like a transposed image, or a transposed vector for 1D signal inputs), like shown below:

[sp]
Size=8000 1 1
BatchSize=${BATCH_SIZE}

; Transpose the input:
[trans]
Input=sp
Type=Transpose
NbOutputs=1
Perm=1 0 2 3
; Output dimensions are now "1 8000 1 ${BATCH_SIZE}"

[onnx]
Input=trans
Type=ONNX
Transpose=1
; The graph originally expects an input dimension of "8000"
; After "Transpose=1", the expected input dimension becomes "1 8000"
File=sound_processing_graph.onnx

Supported operators

The table below summarizes the currently implemented ONNX operators:

Operator	Support	Remarks
Add	✓
AveragePool	✓	Exc. ceil_mode and count_include_pad attributes
BatchNormalization	✓
Cast	✗	Ignored
Clip	✓	Only for min = 0 and max > 0
Concat	✓	Only for layers that support it
Constant	✓	In some contexts only
Conv	✓
Dropout	✓	Exc. mask output
Div	✓	With constant second operand only
Flatten	✓	Ignored (not necessary)
Gemm	✓	Only for fully-connected layers
GlobalAveragePool	✓
GlobalMaxPool	✓
LRN	✓
LeakyRelu	✓
MatMul	✓	Only for fully-connected layers
Max	✓
MaxPool	✓	Exc. Indices output
Mul	✓
Pad	✓
Relu	✓
Reshape	✓	Only for fixed dimensions
Resize	✗	Planned (partially)
Shape	✗	Ignored
Sigmoid	✓
Softmax	✓
Softplus	✓
Squeeze	✗	Ignored
Sub	✓
Sum	✓
Tanh	✓
Transpose	✓
Upsample	✗	Planned