GIF rendering on WinRT and UWP

I've been spending some time learning new things, and the blog has been silent for the same reason - some of the things I've been digging into is Rx.Net, Reactive UI and Win2D. Remember when I showed how we could use Win2D to blur images? I'll try to share my experiences over the next few months, but let's start by combining everything to build a gif rendering control. 
 

Rendering gifs on WinRT and UWP

Uwp Gif Rendering.gif

GIF rendering is something that's rarely seen inside Windows apps for many reasons, but with Windows 10 the image preview app actually renders gifs, but there's no control out there that can actually do this. Why? Mainly for performance reasons and because it's a bit complicated.
You can google a lot, and find multiple solutions, but if you want to show more than one gif on screen at a time, you're pretty much out of luck. 

I've created an experimental sample that allows rendering multiple gifs, and when a gif leaves the viewport, it stops rendering until it's back on the screen where it will restart it self. 

Check out the source

File explanation

It takes quite some code to do this, so here's a short overview over the files.

  • GifFileHandler.cs - Responsible loading or downloading and caching the file locally
  • GifPropertiesHelper.cs - Extract image details, such as size, and loads the gif frame details such as delay per frame, size, top, left, height and width 
  • InactiveGifManager.cs - A container for inactive gifs outside the viewport. Is responsible for checking if they are back in the view, and restarting the rendering if so. 
  • GifRenderer.cs - control containing the rendering mechanisms using Win2D, ReactiveUI and Rx.net. 

Source Explanation

The source it's around a 1000 lines, which is way to much to describe in this blogpost. I'll outline the most important code, and the rest you'll have to investigate yourself.
The way the machinery works is fairly simple.

  1. Prepare file
  2. Render as long as it's visible on the screen
  3. If not visible, look once in a while, and go back to the step above if visible

Preparing for rendering

When the control is rendered, and a source is set, we'll notify the ReadyForRendering subject which will:

  1. Download the file
  2. Exact the meta data about the gif and it's frames
  3. Create a imagebuffer which we'll need for rendering frames in the gif.
  4. Create a Win2D CanvasControl and hook into CreateResource and the Draw events and add it to the UI
public GifRenderer()
{
... code ommitted..
    ReadyGifRenderer();
}

private void ReadyGifRenderer()
{
    _disp.Add(this.WhenAnyObservable(x => x.ReadyForRendering)
          .Where(x => x)
          .DistinctUntilChanged()
          .SelectMany(_ => PrepareGifRendering().ToObservable())
          .SelectMany(x => CreateCanvas().ToObservable())
          .Subscribe());

    _disp.Add(this.WhenAnyObservable(x => x._nextFrame)
        .SelectMany(_ => ChangeCurrentFrameAsync().ToObservable())
        .Subscribe());

    this.Unloaded += GifRenderer_Unloaded;
}

private async Task CreateCanvas()
{
    await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
    {
        _canvasControl = new CanvasControl { UseSharedDevice = false };
        _canvasControl.CreateResources += Canvas_CreateResources;
        _canvasControl.Draw += Canvas_Draw;
        this._grid.Children.Add(_canvasControl);
    });
}

A few important things here is the use of a CanvasControl rather than what might intuitively spring to mind, which is the CanvasAnimatedControl.
The CanvasAnimatedControl has a dedicated thread per control, while CanvasControl runs on the UI thread, also theCanvasAnimatedControl defaults to a separate device per control, while CanvasControl defaults to sharing its devices between all in the UI.
This is highly memory efficient and exactly what we want.

I did some seperate tests showing 10 times better memory usage with a shared device, but obviously it comes with a cost - shared performance. But since we have a delay between each frame delay, we should be more than fine. Again this allows us to render lots of gifs without running out of memory.

Rendering while on screen

While it's visible on screen, we'll render the gif one frame at a time only keeping that single frame in memory. This is important since we want to be able to show multiple gifs at a time. This will cost us a bit of performance, but improve our memory usage which is key - especially on low end devices. 

private async Task ChangeCurrentFrameAsync()
{
    try
    {
        var time = Stopwatch.StartNew();
        var frame = await _decoder.GetFrameAsync((uint)_currentFrameIndex);
        var pixelData = await frame.GetPixelDataAsync(
            BitmapPixelFormat.Bgra8,
            BitmapAlphaMode.Straight,
            new BitmapTransform(),
            ExifOrientationMode.IgnoreExifOrientation,
            ColorManagementMode.DoNotColorManage
            );

        CreateActualPixels(pixelData.DetachPixelData());

        var newDelay = _currentGifFrame.DelayMilliseconds - time.ElapsedMilliseconds;
        if (newDelay > 0 && time.ElapsedMilliseconds < 300)
            await Task.Delay((int)newDelay);

        time.Stop();

        _canvasControl?.Invalidate();

        SetNextFrame();
    }
    catch (Exception)
    {
        // We could potentially crash by leaving the viewport in the middle of a sequence
        StopByCatch();
    }
}

We'll extract the bytes per frame basis, and manipulate our existing byte array from the previous frame. The way gifs work is each frame only contains the changes since the last frame. We'll add those changes to the previous frame, and delay if necessary based on the frame-specific delay.
When all that is done, we'll call Invalidate(); on the CanvasControl which triggers the Draw event which will update the UI. 

Drawing the frame

private void Canvas_Draw(CanvasControl sender, CanvasDrawEventArgs args)
{
    if (_pixels == null) return;
    using (var session = args.DrawingSession)
    {
        var frameBitmap = CanvasBitmap.CreateFromBytes(session,
            _pixels,
            _imageProperties.PixelWidth,
            _imageProperties.PixelHeight,
            DirectXPixelFormat.B8G8R8A8UIntNormalized);

        using (frameBitmap)
        {
            _scaleEffect.Source = frameBitmap;
            _scaleEffect.Scale = new Vector2()
            {
                X = (float)_scaleX,
                Y = (float)_scaleY
            };

            session.DrawImage(_scaleEffect, 0f, 0f);
        }
    }
}

The draw event was triggered by the Invalidate call. We use the GPU to scale the the frame. The scaling it self was calculated when we prepared the frame initially. 

This process keeps going as long as the gif is on screen. 

Edits and improvements

Since the original post I've made a few improvements and the code above has been altered correspondingly. Here's the list of changes so far:

  • Removed unnecessary logic from the UI thread
  • Improved the gif frame by creation by using buffers rather than manual by swapping. (This uses a bit more memory but is close to 3 times faster)

Conclusion

It was a lot of fun to get this working. There's a lot of samples out there, but none who actually renders multiple gifs at the same time.
Hopefully this could help bringing gifs into the wild. There's a lot of people who've helped me get this far and get it working. A huge thanks to Shawn Hargreaves (Win2D), Paul Betts (ReactiveUI and Rx) and Rodrigo Díaz for his tireless efforts and discussions with me.

The code is experimental and very large gifs and cause low framerates when rendering on small low-end devices. Also, the source is placed inside my experimental toolkit, so it might be subject to changes over time.

 

Implementing Image blur and gaussian blur in a Windows universal app

Image blur and gaussian blur can give a nice visual effect if used right . Blurred images for backgrounds can give a nice subtle effect and for the same reason it has been used by designers for years. However it's not supported out of the box on Windows Universal apps platform. Luckily there's atleast  two different nuget packages which can help us get started. 
I tried both the Lumia Imaging SDK 2.0 and Win2D. 

Between the two I see Win2D as the winner because the rendering quality, speed and amount of documentation. The downside is  that you don't have the same layout flexibility out of the box as with the Lumia Image SDK. 
Below you'll find source for both implementations.

Win2D Gaussian Blur

Tap to see the Gaussian Blur rendering. The rendering quality way better than the Lumia Image SDK.

Win2D is an easy-to-use Windows Runtime API for immediate mode 2D graphics rendering. It makes use of the Direct2D engine and utilizes the GPU to create the new blur effect. 
Win2D comes with a great documentation that gets you started in seconds and it works on both Windows Phone and Windows Store. 
The rendering is smooth, crisp and it's really really fast. Only downside is that you have to work the with the Win2D CanvasControl and CanvasBitmap rather than a regular Bitmap or WriteableBitmap.

 

Download demo and sample source

 

Lumia Image SDK 2.0

Tap to see the Blur rendering. The rendering quality not quite as good as the Win2D rendering.

The Lumia Image SDK is a Nokia and now Microsoft owned SDK created to provide easy-to-use filtering and image manipulation capabilities. The Image SDK is known for being fast - especially when working with live camera feeds and it has been a part of the platform for a while. I assume it makes use of the GPU but didn't find any material on the core of the Lumia Image SDK.

The rendering is not nearly as smooth as the Win2D though. If you compare the images it looks like the Lumia Image SDK makes use of a blur algorithm that blurs in squares. Not nearly is neat and good looking as Win2D but on the upside you get to work directly with Bitmaps and Writeable bitmaps with offers layout flexibility compared to Win2D where you have to some of the calculations yourself to get the image in the right size. 

Download source and sample

 

Wrapup

Win2D is the clear winner here though the Lumia SDK is a bit easier to implement but offers less flexibility and quality. A huge shout out to Rodrigo Diaz (@r2d2rigo) for introducing me to Win2D! He's a absolutely brilliant and clever guy. One to follow for sure! 

If you prefer not downloading the source you can find it below this section.

Feel free to contact contact me or write in the comments below if you have any questions or problems getting the sample to work.

P.S Make sure you follow me on twitter @deanihansen for more news, design tips, articles and how-to's.


Code snippets for Lumia Image SDK blur and Win2D Gaussian Blur

Lumia Image SDK implementation:

1) This code snippet is a part of a templated control. The template contains an Image named "imagePresenter":

private async Task RenderImage()
{
    if (!string.IsNullOrWhiteSpace(ImageUrl))
    {
        var imageControl = GetTemplateChild("imagePresenter") as Image;

        if (imageControl != null)
        {
            var client = new HttpClient();

            // or get stream locally
            var stream = await client.GetStreamAsync(ImageUrl);

            var source = new Lumia.Imaging.StreamImageSource(stream, ImageFormat.Undefined);

            var source = new Lumia.Imaging.StreamImageSource(stream, ImageFormat.Undefined);

            var info = await source.GetInfoAsync();

            double displayScaling = DisplayInformation.GetForCurrentView().LogicalDpi / 96.0;


            double pixelWidth = Window.Current.Bounds.Width * displayScaling;

            var scalefactor = pixelWidth / info.ImageSize.Width;

            using (var filterEffect = new FilterEffect(source))
            {
                WriteableBitmap target = new WriteableBitmap((int)pixelWidth, (int)(info.ImageSize.Height * scalefactor));

                var filter = new BlurFilter(45);

                filterEffect.Filters = new IFilter[] { filter };

                using (var renderer = new WriteableBitmapRenderer(filterEffect, target, OutputOption.PreserveAspectRatio))
                {
                    await renderer.RenderAsync();

                    imageControl.Source = target;
                }
            }
        }
    }
}

Win2D Gaussian blur implementation

1) Create the CanvasControl which will host our Image and hook into the two events. Finally set the CanvasControl as Content on a contentpresenter or control.

control = new CanvasControl();
control.Draw += OnDraw;
control.CreateResources += OnCreateResources;
contentPresenter.Content = control;

2) Create and load the image from a URL

private async void OnCreateResources(CanvasControl sender, object args)
{
    if (!string.IsNullOrWhiteSpace(ImageUrl))
    {
        scaleEffect = new ScaleEffect();
        blurEffect = new GaussianBlurEffect();

        image = await CanvasBitmap.LoadAsync(sender.Device,
          new Uri(ImageUrl));

        imageLoaded = true;

        sender.Invalidate();
    }
}

3) On Draw we take the draw session and apply a scale effect to resize to the appropriate siize for the screen and DPI. Afterwards the blur effect is applied and finally the draw is called to render the image onto the Canvas.

private void OnDraw(CanvasControl sender, CanvasDrawEventArgs args)
{
    if (imageLoaded)
    {
        using (var session = args.DrawingSession)
        {
            session.Units = CanvasUnits.Pixels;

            double displayScaling = DisplayInformation.GetForCurrentView().LogicalDpi / 96.0;

            double pixelWidth = sender.ActualWidth * displayScaling;

            var scalefactor = pixelWidth / image.Size.Width;

            scaleEffect.Source = this.image;
            scaleEffect.Scale = new Vector2()
            {
                X = (float)scalefactor,
                Y = (float)scalefactor
            };

            blurEffect.Source = scaleEffect;
            blurEffect.BlurAmount = Blur;

            session.DrawImage(blurEffect, 0.0f, 0.0f);
        }
    }
}