GIF rendering on WinRT and UWP

I've been spending some time learning new things, and the blog has been silent for the same reason - some of the things I've been digging into is Rx.Net, Reactive UI and Win2D. Remember when I showed how we could use Win2D to blur images? I'll try to share my experiences over the next few months, but let's start by combining everything to build a gif rendering control. 
 

Rendering gifs on WinRT and UWP

Uwp Gif Rendering.gif

GIF rendering is something that's rarely seen inside Windows apps for many reasons, but with Windows 10 the image preview app actually renders gifs, but there's no control out there that can actually do this. Why? Mainly for performance reasons and because it's a bit complicated.
You can google a lot, and find multiple solutions, but if you want to show more than one gif on screen at a time, you're pretty much out of luck. 

I've created an experimental sample that allows rendering multiple gifs, and when a gif leaves the viewport, it stops rendering until it's back on the screen where it will restart it self. 

Check out the source

File explanation

It takes quite some code to do this, so here's a short overview over the files.

  • GifFileHandler.cs - Responsible loading or downloading and caching the file locally
  • GifPropertiesHelper.cs - Extract image details, such as size, and loads the gif frame details such as delay per frame, size, top, left, height and width 
  • InactiveGifManager.cs - A container for inactive gifs outside the viewport. Is responsible for checking if they are back in the view, and restarting the rendering if so. 
  • GifRenderer.cs - control containing the rendering mechanisms using Win2D, ReactiveUI and Rx.net. 

Source Explanation

The source it's around a 1000 lines, which is way to much to describe in this blogpost. I'll outline the most important code, and the rest you'll have to investigate yourself.
The way the machinery works is fairly simple.

  1. Prepare file
  2. Render as long as it's visible on the screen
  3. If not visible, look once in a while, and go back to the step above if visible

Preparing for rendering

When the control is rendered, and a source is set, we'll notify the ReadyForRendering subject which will:

  1. Download the file
  2. Exact the meta data about the gif and it's frames
  3. Create a imagebuffer which we'll need for rendering frames in the gif.
  4. Create a Win2D CanvasControl and hook into CreateResource and the Draw events and add it to the UI
public GifRenderer()
{
... code ommitted..
    ReadyGifRenderer();
}

private void ReadyGifRenderer()
{
    _disp.Add(this.WhenAnyObservable(x => x.ReadyForRendering)
          .Where(x => x)
          .DistinctUntilChanged()
          .SelectMany(_ => PrepareGifRendering().ToObservable())
          .SelectMany(x => CreateCanvas().ToObservable())
          .Subscribe());

    _disp.Add(this.WhenAnyObservable(x => x._nextFrame)
        .SelectMany(_ => ChangeCurrentFrameAsync().ToObservable())
        .Subscribe());

    this.Unloaded += GifRenderer_Unloaded;
}

private async Task CreateCanvas()
{
    await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
    {
        _canvasControl = new CanvasControl { UseSharedDevice = false };
        _canvasControl.CreateResources += Canvas_CreateResources;
        _canvasControl.Draw += Canvas_Draw;
        this._grid.Children.Add(_canvasControl);
    });
}

A few important things here is the use of a CanvasControl rather than what might intuitively spring to mind, which is the CanvasAnimatedControl.
The CanvasAnimatedControl has a dedicated thread per control, while CanvasControl runs on the UI thread, also theCanvasAnimatedControl defaults to a separate device per control, while CanvasControl defaults to sharing its devices between all in the UI.
This is highly memory efficient and exactly what we want.

I did some seperate tests showing 10 times better memory usage with a shared device, but obviously it comes with a cost - shared performance. But since we have a delay between each frame delay, we should be more than fine. Again this allows us to render lots of gifs without running out of memory.

Rendering while on screen

While it's visible on screen, we'll render the gif one frame at a time only keeping that single frame in memory. This is important since we want to be able to show multiple gifs at a time. This will cost us a bit of performance, but improve our memory usage which is key - especially on low end devices. 

private async Task ChangeCurrentFrameAsync()
{
    try
    {
        var time = Stopwatch.StartNew();
        var frame = await _decoder.GetFrameAsync((uint)_currentFrameIndex);
        var pixelData = await frame.GetPixelDataAsync(
            BitmapPixelFormat.Bgra8,
            BitmapAlphaMode.Straight,
            new BitmapTransform(),
            ExifOrientationMode.IgnoreExifOrientation,
            ColorManagementMode.DoNotColorManage
            );

        CreateActualPixels(pixelData.DetachPixelData());

        var newDelay = _currentGifFrame.DelayMilliseconds - time.ElapsedMilliseconds;
        if (newDelay > 0 && time.ElapsedMilliseconds < 300)
            await Task.Delay((int)newDelay);

        time.Stop();

        _canvasControl?.Invalidate();

        SetNextFrame();
    }
    catch (Exception)
    {
        // We could potentially crash by leaving the viewport in the middle of a sequence
        StopByCatch();
    }
}

We'll extract the bytes per frame basis, and manipulate our existing byte array from the previous frame. The way gifs work is each frame only contains the changes since the last frame. We'll add those changes to the previous frame, and delay if necessary based on the frame-specific delay.
When all that is done, we'll call Invalidate(); on the CanvasControl which triggers the Draw event which will update the UI. 

Drawing the frame

private void Canvas_Draw(CanvasControl sender, CanvasDrawEventArgs args)
{
    if (_pixels == null) return;
    using (var session = args.DrawingSession)
    {
        var frameBitmap = CanvasBitmap.CreateFromBytes(session,
            _pixels,
            _imageProperties.PixelWidth,
            _imageProperties.PixelHeight,
            DirectXPixelFormat.B8G8R8A8UIntNormalized);

        using (frameBitmap)
        {
            _scaleEffect.Source = frameBitmap;
            _scaleEffect.Scale = new Vector2()
            {
                X = (float)_scaleX,
                Y = (float)_scaleY
            };

            session.DrawImage(_scaleEffect, 0f, 0f);
        }
    }
}

The draw event was triggered by the Invalidate call. We use the GPU to scale the the frame. The scaling it self was calculated when we prepared the frame initially. 

This process keeps going as long as the gif is on screen. 

Edits and improvements

Since the original post I've made a few improvements and the code above has been altered correspondingly. Here's the list of changes so far:

  • Removed unnecessary logic from the UI thread
  • Improved the gif frame by creation by using buffers rather than manual by swapping. (This uses a bit more memory but is close to 3 times faster)

Conclusion

It was a lot of fun to get this working. There's a lot of samples out there, but none who actually renders multiple gifs at the same time.
Hopefully this could help bringing gifs into the wild. There's a lot of people who've helped me get this far and get it working. A huge thanks to Shawn Hargreaves (Win2D), Paul Betts (ReactiveUI and Rx) and Rodrigo Díaz for his tireless efforts and discussions with me.

The code is experimental and very large gifs and cause low framerates when rendering on small low-end devices. Also, the source is placed inside my experimental toolkit, so it might be subject to changes over time.