From Python to C#: Transferring Models Using the ONNX Format

Python has established itself as the go-to language for AI researchers, engineers, and practitioners. The most feature-complete implementations of popular ML frameworks like PyTorch and Tensorflow are written in Python. So how do you natively use your AI models in an application written in a different language? The ONNX format addresses exactly this problem.

Let's walk through how it works with this tutorial!

What Is ONNX?

ONNX stands for Open Neural Network Exchange. It's an open-source project designed to facilitate interoperability between machine learning systems, regardless of the language they were built in. You can find the project's GitHub page here. The format has gradually become an industry standard.

Companies supporting the ONNX project

Beyond enabling the migration of an AI model from one platform to another, ONNX also makes it easier to optimize those models through its converter, which transforms models into a standardized format while incorporating performance optimizations. These optimizations can include neuron layer fusion, redundant node elimination, and mathematical operation simplification.

These optimizations speed up execution times and reduce the memory footprint of models. This is especially beneficial when deploying models on resource-constrained devices like mobile phones or IoT devices. In this way, ONNX helps make machine learning models lighter and faster, without any loss of information or functionality compared to the original.

Prerequisites

The goal of this tutorial is to guide you through a practical example of transferring a model from Python to C#. We won't spend much time explaining how to train a model, how neural networks work, etc., since we want to focus specifically on the ONNX transfer process.

It assumes you have the following prerequisites:

• A working Python environment for machine learning
• A working .NET Core environment (I used version 7)

The code you'll write should run on any platform. Feel free to let me know in the comments if you run into any issues!

What You'll Build

You will:

1. Train an image classifier based on resnet18 using fastai, a wrapper library around PyTorch that makes it very easy to build neural network training pipelines.
2. We'll specialize this model by training it on MNIST, the benchmark dataset for deep learning.
3. This classifier's job will be to recognize digits 0 through 9 from an input image taken from the MNIST training set.
4. We'll then export it to ONNX so it can be consumed by a .NET Core 7 application, enabling classifications in that language!

Your reference point will be this git repository. Feel free to check it out for the full code. I'll focus here on the points I found most important for understanding how this type of export works.

Key Considerations Before Export

Exhaustively List All Preprocessing Steps for the Input to Classify

Most popular AI frameworks and tools apply several transformations to input data even if you haven't explicitly specified them in your code. This is the case with fastai. For example, I spent quite some time figuring out that fastai's visual_learner was actually applying several default transformations to my training images during training, even though I wanted to create the simplest possible data loader:

the data loader in question

You can verify this after training the model like this =>

In summary, 2 transformations are applied by default by fastai during training:

1. The integer tensor of the imported image is converted to a float tensor and each pixel is divided by 255 for normalization purposes.
2. The pixels are normalized again using the mean and standard deviation to ensure all pixel values are distributed on the correct scale.

Knowing exactly which preprocessing steps were applied is crucial, as you'll see in the C# section.

Know the Shape of the Input to Classify

This applies to traditional programs and neural networks alike (at least for now): a model expects an input of a specific shape to perform its task.

Once you've exported your model for use in your target programming language, you'll need to ensure that the input for prediction has exactly the same shape as the one used during training:

We can see above that our input is a 28x28 pixel image with three channels: R, G, and B. So we can call this a rank-3 tensor.

Knowing the shape of the input to classify will help us define a sample input when exporting to ONNX:

The four dimensions you see above are, from left to right:

• the batch size to feed the model at classification time (here 1, but during training we loaded images in batches of 64)
• the image channels (RGB)
• the pixels in width and height

Switch the Model to Evaluation Mode Before Export

This small line of code ensures your model is ready for production:

This will remove redundant neuron layers, certain normalization operations in hidden layers, and eliminate all dropout since we're no longer in the training phase.

Export the Model

The code above is extensively commented in the notebook.

However, some interesting points are worth highlighting:

• If you're not sure that the target machines hosting your ONNX model for running inferences have a GPU, play it safe and pass the export parameter specifying that the model will run on CPU; likewise, make sure the test input is a CPU tensor
• Don't forget to specify that you want to apply compiler optimizations during export. This is a best practice.
• For a production application, you need to include the model weights in the export.
• Specify dynamic axes for I/O: this will allow you to run batch inferences with your exported model (classifying images in batches in our case).

Validate Your Export in Python

Before switching to another platform to use your ONNX export and adding complexity to your work, save yourself a few hours of pain: validate that your export works on the same platform you used for the export by simply re-importing your ONNX model and verifying that it works as expected =>

• Technical verification
• Reproduce the preprocessing

As mentioned earlier, for classification to succeed, the model needs to receive the same kind of input it learned from during training (a principle that's absolutely fundamental yet so easy to forget in the world of deep learning).

• Test inference with the exported model

Here, the test is very simple, but to rigorously test that your model doesn't lose performance from one platform to another, you should run batch inferences on a representative dataset.

Let's Do This in C#!

You're ready to run your model in C#!

The C# code essentially mirrors what we did in Python:

• ... the initial preprocessing
• running inference

Conclusion

Through this example (and a few hours of figuring out how it works), I'm happy to share with you how to consume an AI model developed in Python in another programming language (here C#).

This learning experience was incredibly exciting for me, as I now see more clearly the realm of possibilities when it comes to embedding AI-powered applications into any codebase, even legacy ones!

I'd also add that if you need to modernize and add a touch of AI to your digital transformation without having to tear apart your existing system, you can call on the AI Squad!

It couldn't be simpler: contact https://reboot-conseil.com/!