ONNX Support#

ONNX supported operators#

Currently, HElayers partially supports ONNX files. It is recommended to review the supported operations below before loading an ONNX file into HElayers. When an unsupported operator is loaded to HElayers, it will try to automatically identify it and alert the user about it.

The supported ONNX operators are:

  • AveragePool

  • BatchNormalization

  • Cast

  • Concat

  • Constant

  • Conv

  • Expand (can be applied on weights only)

  • Flatten

  • Gather

  • Gemm

  • Identity

  • MatMul

  • Pad

  • PolyApprox

  • Reciprocal (can be applied on weights only)

  • ReduceMean

  • ReduceSum

  • Reshape

  • Shape

  • Slice

  • Squeeze

  • Transpose

  • Unsqueeze

In addition, the following operators are interpreted by HElayers as polynomial activation layers:

  • Constant

  • Div

  • Neg

  • Pow

  • The “Add”, “Sub” and “Mul” operators are also supported and may be used either as a separate layer that performs element-wise addition, subtraction or multiplication on the outputs of two previous layers together, or on the output of a previous layer and a weight tensor, or as part of an activation layer.

HElayers support non-sequential graphs of operators: operator’s output can be the input of multiple operators, and the outputs of two operators can be added, subtracted, multiplied or concatenated to one another as long as the tensor shapes are compatible. The operators must form a DAG, where all paths start and end in the same nodes. For example, an NN must have a single input, which can then be fed into multiple layers.

Note: dimension 0 is assumed to be the batch dimension of the NN input and output. This dimension must not be changed, manipulated or involved in operations in any way. To overcome this limitation, set a model-specified fixed batch size (see PlainModelHyperParams::modelSpecifiedFixedBatchSize).

Below is a description of each of the supported operators, including a list of its supported attributes. Any other attribute that is not specified will raise an error.

Add#

  • Must have two inputs: the input data and weights, or two inputs.

  • Must have one output.

  • No attributes are supported.

AveragePool#

  • Must have one input: the input data.

  • Must have one output.

  • The supported attributes are:

    • ”kernel_shape” - required, must contain two integers

    • ”strides” - optional (default [1,1]), if given must contain two integers

    • ”pads” - optional (default [0,0,0,0]), if given must contain 4 integers. If a “Pad” operator exists just before this layer, and if that “Pad” operator defines pads that are not all-zeros, the pads for this layer will be taken from that previous “Pad” operator, and in such case, this layer’s pads must either not be specified or define pads that are all-zeros.

    • ”count_include_pad” - optional. NOTICE: when an average pooling layer includes padding, its evaluation by Helayers will include the pads when averaging, regardless of the value of this attribute.

    • ”auto_pad” - optional, if given must have the value “NOTSET”

    • ”ceil_mode” - optional, if given must have the value 0

    • ”dilations” - optional, if given must have the value [1,1]

BatchNormalization#

  • Must have five inputs: the input data, the scales, the biases, the means and the variances.

  • Must have one output.

  • The supported attributes are:

    • ”epsilon” - optional (default 1e-5), the value to use for the normalization, to avoid deviding by zero.

    • ”momentum” - optional, ignored

    • ”training_mode” - optional, if given must have the value 0

  • The batch normalization is assumed to be over dimension #1.

Cast#

  • Must have one input.

  • Must have one output.

  • The supported attributes are:

    • ”to” - required. If applied on a non-weight, must either equal onnx::TensorProto::FLOAT or onnx::TensorProto::DOUBLE. Notice that in this case, no actual casting will be applied under FHE, and values will always be represented by the numeric type supported by the FHE library. If applied on a weight, must equal one of the following: TensorProto::UINT8, TensorProto::INT8, TensorProto::UINT16, TensorProto::INT16, TensorProto::UINT32, TensorProto::INT32, TensorProto::UINT64, TensorProto::INT64.

Concat#

  • Must have two or more inputs.

  • Must have one output.

  • The supported attributes are:

    • ”axis” - required, the axis to concatenate along.

  • Must be applied on operands that are either all non-weights or all weights.

  • If wpplied on non-weights, must have exactly two inputs.

Constant#

  • Must have no inputs.

  • Must have one output.

  • The supported attributes are:

    • ”value” - required, must contain a valid tensor with values.

Conv#

  • Must have two or three inputs: the input data, filters and optional biases.

  • Must have one output.

  • The supported attributes are:

    • ”kernel_shape” - required, must contain two integers

    • ”strides” - optional (default [1,1]), if given must contain two integers

    • ”pads” - optional (default [0,0,0,0]), if given must contain 4 integers

    • ”group” - optional (default 1), if given, currently, must be either 1, or the number of input channels in case they also equal the number of output channels.

    • ”auto_pad” - optional, if given must have the value “NOTSET”

    • ”dilations” - optional, if given must have the value [1,1]

  • The number of filters will be deduced from the associated filters specified as input for the operator.

  • Filters are assumed to be of shape [filters][channels][rows][cols]

Div#

  • Must have two inputs.

  • Must have one output.

  • No attributes are supported.

Expand#

  • See ONNX documentation for “Expand” operator for the logic, supported inputs, outputs and attributes.

  • Can be applied on weights only.

Flatten#

  • Must have one input: the input data.

  • Must have one output.

  • The supported attributes are:

    • ”axis” - optional, if given must have the value 1

Gather#

  • See ONNX documentation for “Gather” operator for the logic, supported inputs, outputs and attributes.

  • Can be applied on weights only.

Gemm#

  • Must have two or three inputs: the input data, weights and optional bias.

  • Must have one output.

  • The supported attributes are:

    • ”alpha” - optional (default 1)

    • ”beta” - optional (default 1)

    • ”transA” - optional, if given must have the value 0

    • ”transB” - optional (default 0)

  • The output size will be deduced from the associated weights specified as input for the operator.

  • Weights are assumed to be of shape [output_size][input_size]

Identity#

  • Must have one input.

  • Must have one output.

  • No attributes are supported.

MatMul#

  • Must have two inputs: the input data and weights, or two inputs.

  • Must have one output.

  • No attributes are supported.

Mul#

  • Must have two inputs: the input data and weights, or two inputs.

  • Must have one output.

  • No attributes are supported.

Neg#

  • Must have one inputs: the input data.

  • Must have one output.

  • No attributes are supported.

Pad#

  • Must have one or two inputs: the input data and optional pad values.

  • Must have one output.

  • The supported attributes are:

    • ”pads” or “paddings” - required if pads are not provided as a second input. If provided, must contain 8 integers: 0,0,T,L,0,0,B,R where T,L,B,R are the paddings from top, left, bottom and right, and can be zero or non-zero values. In case the operator has two inputs, the pad values will be taken from the second input and not from the “pads” or “paddings” attribute, which in this case are ignored. In such case, the second input must contain a tensor of 8 integers that follow the above limitations.

    • ”mode” - optional, if given must have the value “constant”

    • ”value” - optional, if given must have the value 0

  • This operators must be followed by an AveragePool operator, and the paddings of this operator will be applied over that next AveragePool.

PolyApprox#

  • Must have one input: the input data.

  • Must have one output.

  • The supported attributes are:

    • ”function” - a string specifying the function to be approximated. Currently only “1/sqrt” is supported.

    • ”approximationMethod” - a string specifying the approximation method of the specified function. Currently only “goldschmidt” is supported, which computes 1/sqrt(x) using the goldschmidt approximation method.

    • ”numIterations” - The number of iterations of the approximation algorithm.

    • ”scale” - A scale to be applied on the input so it will fall within the supported approximation range.

Pow#

  • Must have two inputs: the input data and the exponent.

  • Must have one output.

  • No attributes are supported.

  • The input associated with the exponent must contain a valid tensor with a single value, or be a “Constant” operator representing a single scalar. The exponent value must be a non-negative integer.

Reciprocal#

  • Must have one input.

  • Must have one output.

  • No attributes are supported.

  • Can be applied on weights only.

ReduceMean#

  • Must have one or two inputs: the input data and optional axes value.

  • Must have one output.

  • The supported attributes are:

    • ”axes” - required, the axis to apply the reduce along. In case the operator has two inputs, the value will be taken from the second input and not from the “axes” attribute. Must contain a single integer.

    • ”keepdims” - optional (default 1), whether to keep the reduced dim as a degenerate dim or to remove it.

    • ”noop_with_empty_axes” - optional, ignored as an empty list of “axes” is not supported.

ReduceSum#

  • Must have one or two inputs: the input data and optional axes value.

  • Must have one output.

  • The supported attributes are:

    • ”axes” - required, the axis to apply the reduce along. In case the operator has two inputs, the value will be taken from the second input and not from the “axes” attribute. Must contain a single integer.

    • ”keepdims” - optional (default 1), whether to keep the reduced dim as a degenerate dim or to remove it.

    • ”noop_with_empty_axes” - optional, ignored as an empty list of “axes” is not supported.

Reshape#

  • Must have one or two inputs: the input data and optional shape value.

  • Must have one output.

  • The supported attributes are:

    • ”shape” - required, the new shape to reshape into. In case the operator has two inputs, the value will be taken from the second input and not from the “shape” attribute. Must contain a list of integers.

    • ”allowzero” - optional, if given must have the value 0.

Shape#

  • See ONNX documentation for “Shape” operator for the logic, supported inputs, outputs and attributes.

Slice#

  • See ONNX documentation for “Slice” operator for the logic, supported inputs, outputs and attributes.

  • Currently, “steps”, if provided, must all equal 1.

Squeeze#

  • Must have one or two inputs: the input data and optional axes values.

  • Must have one output.

  • The supported attributes are:

    • ”axes” - optional, the dimensions to squeeze. If empty, all the dimensions of size 1 will be squeezed. If provided, must contain a list of integers. In case the operator has two inputs, the axes values will be taken from the second input and not from the “axes” attribute, which in this case is ignored. In such case, the second input must contain a list of integers.

Sub#

  • Must have two inputs: the input data and weights, or two inputs.

  • Must have one output.

  • No attributes are supported.

Transpose#

  • Must have one input.

  • Must have one output.

  • The supported attributes are:

    • ”perm” - required, the permutation to apply.

Unsqueeze#

  • Must have one or two inputs: the input data and optional axes values.

  • Must have one output.

  • The supported attributes are:

    • ”axes” - required if axes are not provided as a second input. If provided, must contain a list of integers. In case the operator has two inputs, the axes values will be taken from the second input and not from the “axes” attribute, which in this case is ignored. In such case, the second input must contain a list of integers.