Little over a year ago, I blogged about how to do image classification using WinML in UWP. Since then we got some updates of Windows and Visual Studio which result in some changes.

Windows May 2019 Update (1809)

Over time the WinML apis went out of preview. They no longer live in
‘Windows.AI.MachineLearning.Preview’, instead you can find them in ‘Windows.AI.MachineLearning’. Classes like ‘LearningModelPreview’ lost their ‘Preview’ suffix and are now just called ‘LearningModel’.
For the sample I’ve build in my previous blogpost, this shouldn’t be such a big deal as these classes were only used in classes that were generated by Visual Studio and the Microsoft Visual Studio Tools for AI .

Visual Studio 2019 and Windows Machine Learning Code Generator

I noticed that the Visual Studio Tools for AI that I already reffered to in this post as well as in my previous blogpost about this topic, isn’t compatible with Visual Studio 2019. It turns out that for Visual Studio 2019 you should download and install Windows Machine Learning Code Generator . What this tool does is whenever you add an ONNX file to your project, it will automatically generate a wrapper around this model so you can easily use it from code.

ONNX 1.2

Turns out that ONNX has been evolving a lot over the year as well. So much even that my ‘old’ ONNX model wasn’t compatible any more. When I added my existing ONNX model to my project, nothing really happend, exept this message in my output window:
mlgen failed: Node:BatchNormalization Required attribute ‘consumed_inputs’ is missing.

So I went back to Custom Vision Service to find out that I removed the model I generated a year ago. Too bad… 🙁 But I took this opportunity to generate a new simple model: ‘Chocolat or Raisins’. I uploaded a bunch of delecious chocolat chip cookies to Custom Vision Service and tagged them as ‘chocolat’. I also uploaded a ton of disgusting Raisin cookies and tagged them as ‘raisin’. Trained my model (Multiclass, food compact) and exported it as ONNX 1.2. When exporting, it immediately became apparent why my old ONNX model wasn’t working any more:

Tying it all together

Now that I got everything in place, I just had to add my exported ONNX model to my project. When doing so, the Windows Machine Learning Code Generator kicks in and generates some classes around this model.

The code that uses this model to get the image classification doesn’t have to change a lot compared to the sample I’ve written in my old blogpost.

public class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Running...");

        Run().ContinueWith((_) => Console.WriteLine("Done!"));

        Console.ReadKey();
    }

    private static async Task Run()
    {
        StorageFile modelFile = 
            await StorageFile.GetFileFromApplicationUriAsync(new Uri($"ms-appx:///Assets/ChocolatOrRaisin.onnx"));
        ChocolatOrRaisinModel model = await ChocolatOrRaisinModel.CreateFromStreamAsync(modelFile as IRandomAccessStreamReference);

        foreach (StorageFile image in await GetImagesFromExecutingFolder())
        {
            Console.Write($"Processing {image.Name}... ");
            var videoFrame = await GetVideoFrame(image);

            ChocolatOrRaisinInput ModelInput = new ChocolatOrRaisinInput();
            ModelInput.data = ImageFeatureValue.CreateFromVideoFrame(videoFrame);

            var ModelOutput = await model.EvaluateAsync(ModelInput);
            var topCategory = ModelOutput.loss.SelectMany(l => l).OrderByDescending(kvp => kvp.Value).FirstOrDefault();
            var label = ModelOutput.classLabel.GetAsVectorView().ToList().FirstOrDefault();

            Console.Write($"DONE ({topCategory.Key} {topCategory.Value:P2}) - ClassLabel => {label} \n");

            await UpdateImageTagMetadata(await image.Properties.GetImagePropertiesAsync(), topCategory.Key);
        }
    }

    private static async Task<IEnumerable<StorageFile>> GetImagesFromExecutingFolder()
    {
        var folder = await StorageFolder.GetFolderFromPathAsync(Environment.CurrentDirectory);

        var queryOptions = new QueryOptions(CommonFileQuery.DefaultQuery, new List<string>() { ".jpg", ".png" });

        return await folder.CreateFileQueryWithOptions(queryOptions)?.GetFilesAsync();
    }

    private static async Task UpdateImageTagMetadata(ImageProperties imageProperties, params string[] tags)
    {
        var propertiesToSave = new List<KeyValuePair<string, object>>()
        {
            new KeyValuePair<string, object>("System.Keywords", String.Join(';', tags))
        };
        await imageProperties.SavePropertiesAsync(propertiesToSave);
    }

    private static async Task<VideoFrame> GetVideoFrame(StorageFile file)
    {
        SoftwareBitmap softwareBitmap;
        using (IRandomAccessStream stream = await file.OpenAsync(FileAccessMode.Read))
        {
            // Create the decoder from the stream 
            BitmapDecoder decoder = await BitmapDecoder.CreateAsync(stream);

            // Get the SoftwareBitmap representation of the file in BGRA8 format
            softwareBitmap = await decoder.GetSoftwareBitmapAsync();
            softwareBitmap = SoftwareBitmap.Convert(softwareBitmap, BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied);

            return VideoFrame.CreateWithSoftwareBitmap(softwareBitmap);
        }
    }
}

That’s it

So as you can see, over the past year a lot has changed to some WinML boiler plate stuff and Visual Studio 2019 comes a new tool that generated the wrappers around your ONNX model, updates to ONNX, … Other than that, the essence remains: using Machine Learning locally on your device is super simple with WinML!

Try it out yourself

You can get the sources from this sample from GitHub. Just run it once, then you can go to any folder using the command line, type ‘TagPhotos’ and the UWP console app will start up to analyse and tag all pictures in that folder (for chocolat chip cookies and raisin cookies that is). 🙂