GIF rendering on WinRT and UWP

I've been spending some time learning new things, and the blog has been silent for the same reason - some of the things I've been digging into is Rx.Net, Reactive UI and Win2D. Remember when I showed how we could use Win2D to blur images? I'll try to share my experiences over the next few months, but let's start by combining everything to build a gif rendering control. 
 

Rendering gifs on WinRT and UWP

Uwp Gif Rendering.gif

GIF rendering is something that's rarely seen inside Windows apps for many reasons, but with Windows 10 the image preview app actually renders gifs, but there's no control out there that can actually do this. Why? Mainly for performance reasons and because it's a bit complicated.
You can google a lot, and find multiple solutions, but if you want to show more than one gif on screen at a time, you're pretty much out of luck. 

I've created an experimental sample that allows rendering multiple gifs, and when a gif leaves the viewport, it stops rendering until it's back on the screen where it will restart it self. 

Check out the source

File explanation

It takes quite some code to do this, so here's a short overview over the files.

  • GifFileHandler.cs - Responsible loading or downloading and caching the file locally
  • GifPropertiesHelper.cs - Extract image details, such as size, and loads the gif frame details such as delay per frame, size, top, left, height and width 
  • InactiveGifManager.cs - A container for inactive gifs outside the viewport. Is responsible for checking if they are back in the view, and restarting the rendering if so. 
  • GifRenderer.cs - control containing the rendering mechanisms using Win2D, ReactiveUI and Rx.net. 

Source Explanation

The source it's around a 1000 lines, which is way to much to describe in this blogpost. I'll outline the most important code, and the rest you'll have to investigate yourself.
The way the machinery works is fairly simple.

  1. Prepare file
  2. Render as long as it's visible on the screen
  3. If not visible, look once in a while, and go back to the step above if visible

Preparing for rendering

When the control is rendered, and a source is set, we'll notify the ReadyForRendering subject which will:

  1. Download the file
  2. Exact the meta data about the gif and it's frames
  3. Create a imagebuffer which we'll need for rendering frames in the gif.
  4. Create a Win2D CanvasControl and hook into CreateResource and the Draw events and add it to the UI
public GifRenderer()
{
... code ommitted..
    ReadyGifRenderer();
}

private void ReadyGifRenderer()
{
    _disp.Add(this.WhenAnyObservable(x => x.ReadyForRendering)
          .Where(x => x)
          .DistinctUntilChanged()
          .SelectMany(_ => PrepareGifRendering().ToObservable())
          .SelectMany(x => CreateCanvas().ToObservable())
          .Subscribe());

    _disp.Add(this.WhenAnyObservable(x => x._nextFrame)
        .SelectMany(_ => ChangeCurrentFrameAsync().ToObservable())
        .Subscribe());

    this.Unloaded += GifRenderer_Unloaded;
}

private async Task CreateCanvas()
{
    await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
    {
        _canvasControl = new CanvasControl { UseSharedDevice = false };
        _canvasControl.CreateResources += Canvas_CreateResources;
        _canvasControl.Draw += Canvas_Draw;
        this._grid.Children.Add(_canvasControl);
    });
}

A few important things here is the use of a CanvasControl rather than what might intuitively spring to mind, which is the CanvasAnimatedControl.
The CanvasAnimatedControl has a dedicated thread per control, while CanvasControl runs on the UI thread, also theCanvasAnimatedControl defaults to a separate device per control, while CanvasControl defaults to sharing its devices between all in the UI.
This is highly memory efficient and exactly what we want.

I did some seperate tests showing 10 times better memory usage with a shared device, but obviously it comes with a cost - shared performance. But since we have a delay between each frame delay, we should be more than fine. Again this allows us to render lots of gifs without running out of memory.

Rendering while on screen

While it's visible on screen, we'll render the gif one frame at a time only keeping that single frame in memory. This is important since we want to be able to show multiple gifs at a time. This will cost us a bit of performance, but improve our memory usage which is key - especially on low end devices. 

private async Task ChangeCurrentFrameAsync()
{
    try
    {
        var time = Stopwatch.StartNew();
        var frame = await _decoder.GetFrameAsync((uint)_currentFrameIndex);
        var pixelData = await frame.GetPixelDataAsync(
            BitmapPixelFormat.Bgra8,
            BitmapAlphaMode.Straight,
            new BitmapTransform(),
            ExifOrientationMode.IgnoreExifOrientation,
            ColorManagementMode.DoNotColorManage
            );

        CreateActualPixels(pixelData.DetachPixelData());

        var newDelay = _currentGifFrame.DelayMilliseconds - time.ElapsedMilliseconds;
        if (newDelay > 0 && time.ElapsedMilliseconds < 300)
            await Task.Delay((int)newDelay);

        time.Stop();

        _canvasControl?.Invalidate();

        SetNextFrame();
    }
    catch (Exception)
    {
        // We could potentially crash by leaving the viewport in the middle of a sequence
        StopByCatch();
    }
}

We'll extract the bytes per frame basis, and manipulate our existing byte array from the previous frame. The way gifs work is each frame only contains the changes since the last frame. We'll add those changes to the previous frame, and delay if necessary based on the frame-specific delay.
When all that is done, we'll call Invalidate(); on the CanvasControl which triggers the Draw event which will update the UI. 

Drawing the frame

private void Canvas_Draw(CanvasControl sender, CanvasDrawEventArgs args)
{
    if (_pixels == null) return;
    using (var session = args.DrawingSession)
    {
        var frameBitmap = CanvasBitmap.CreateFromBytes(session,
            _pixels,
            _imageProperties.PixelWidth,
            _imageProperties.PixelHeight,
            DirectXPixelFormat.B8G8R8A8UIntNormalized);

        using (frameBitmap)
        {
            _scaleEffect.Source = frameBitmap;
            _scaleEffect.Scale = new Vector2()
            {
                X = (float)_scaleX,
                Y = (float)_scaleY
            };

            session.DrawImage(_scaleEffect, 0f, 0f);
        }
    }
}

The draw event was triggered by the Invalidate call. We use the GPU to scale the the frame. The scaling it self was calculated when we prepared the frame initially. 

This process keeps going as long as the gif is on screen. 

Edits and improvements

Since the original post I've made a few improvements and the code above has been altered correspondingly. Here's the list of changes so far:

  • Removed unnecessary logic from the UI thread
  • Improved the gif frame by creation by using buffers rather than manual by swapping. (This uses a bit more memory but is close to 3 times faster)

Conclusion

It was a lot of fun to get this working. There's a lot of samples out there, but none who actually renders multiple gifs at the same time.
Hopefully this could help bringing gifs into the wild. There's a lot of people who've helped me get this far and get it working. A huge thanks to Shawn Hargreaves (Win2D), Paul Betts (ReactiveUI and Rx) and Rodrigo Díaz for his tireless efforts and discussions with me.

The code is experimental and very large gifs and cause low framerates when rendering on small low-end devices. Also, the source is placed inside my experimental toolkit, so it might be subject to changes over time.

 

A solution to an 'Unspecified error' when calling GetNavigationState via the SuspensionManager on Windows Phone 8.1

A common approach to communicate between page-navigations is to send data as a parameter when navigating. This parameter can all an object of all kinds - both value and reference types. However there's one gotcha: If you call Frame.GetNavigationState() and you've passed anything other than a simple type like guid, string or numerics you'll get an unspecified error and a COM exception like this:

{System.Runtime.InteropServices.COMException (0x80004005): Unspecified error

GetNavigationState doesn't support serialization of a parameter type which was passed to Frame.Navigate.
   at Windows.UI.Xaml.Controls.Frame.GetNavigationState()
   at xx.Common.SuspensionManager.SaveFrameNavigationState(Frame frame)
   at xx.Common.SuspensionManager.<SaveAsync>d__11.MoveNext()}

I passed the SelectedItem from a List as a parameter to a details page and that caused the exception when I tried to tombstone and save the page state using the SuspensionManager.

I looked up the documentation from MSDN and it's all there in black and white:

Note: The serialization format used by these methods is for internal use only. Your app should not form any dependencies on it. Additionally, this format supports serialization only for basic types like string, char, numeric and GUID types.
— https://msdn.microsoft.com/en-us/library/windows/apps/windows.ui.xaml.controls.frame.getnavigationstate

My solution to this was to serialize my object as json using the Json.NET nuget package and sending the object as a string and deserializing it on the details page.

public void NavigateToDetails(Author author)
{
    string json = JsonConvert.SerializeObject(author, Formatting.Indented);
    NavigationHelper.NavigateTo<AuthorPage>(false, json);
}

And on the details page I simply deserialize it: 

protected override async void OnNavigatedTo(NavigationEventArgs e)
{
     base.OnNavigatedTo(e);
     Author author = JsonConvert.DeserializeObject<Author>(e.Parameter.ToString());
     //// Do something with author
}

This is a simple work around that will work out of the box with the SuspensionManager shipped with Windows Phone 8.1. You might still run into trouble if you serialize huge lists as json and pass that as a parameter, but in those cases I'd say you're most likely doing something wrong. 

Feel free to comment and reach out if you have any comments or additions.

P.S Make sure you follow me on twitter @deanihansen for more news, design tips, articles and how-to's.

Inspirational TextBox styles for Windows Phone & Store

Inspired by a post on text input effect for Web on Codrops I decided to create similar effects in xaml for Windows Store and Phone demonstrating how we can make input fields look great with a few simple animations. 

I've created different effects for the TextBox named Split, Flip, Iconic and Slide.
 

Split

The Split Style keeps the header label between the two lines. Once the textbox gets focus the header transitions upwards and gives room for the actual content. The header stays in place until the textbox loses focus and the text is removed.


Flip

When Flip gets focused the border goes to zero while the background flips up from behind. As long as there's text and focus this stays in place. When removed and focus is lost the background slides backwards out and the border tips back up.


Iconic

Iconic has a nice toned down icon. Once the TextBox gets focused the lights and scales up a bit to bring attention to the input field. Once the field is cleared it tones back down.


Slide

Slide got an overlay per default which is ready to slide out as soon as the TextBox gets focus. As long as there's focus and content it will remain in the right side. If content and focus is cleared is slides back out to the left.


Download the source and sample

The control and styles is made for proof of concept purposes and
it's not recommended for "plug and play" in a production environment.
 

 

Using the control and styles

To create these visual effects I had to create a custom TextBox (BrTextBox.cs). Applying these visual effects is rather easy once you use the custom TextBox. 
Just use the control instead of a normal textbox, set the header and apply style for the effect you want.

 <local:BrTextBox Header="First name" Style="{StaticResource SplitBrTextBoxStyle}"  />
 <local:BrTextBox Header="First name" Style="{StaticResource FlipBrTextBoxStyle}"  />
 <local:BrTextBox Header="First name" Style="{StaticResource SlideBrTextBoxStyle}"  />
 <local:BrTextBox Header="First name" Style="{StaticResource IconicBrTextBoxStyle}"  />

 

Custom control

Creating a custom control was necessary because I had to hooks into the Loaded, GotFocus and LostFocus event. In these events we check on whether we have Text or not. Based on this we set a one of two visual states: Expand or Collapse.

Each visual state will then perform a series of animations to transform the textbox visuals.

Code:

public sealed class BrTextBox : TextBox
{
    private const string StateCollapse = "Collapse";
    private const string StateExpand = "Expand";
    private bool hasFocus;

    public BrTextBox()
    {
        DefaultStyleKey = typeof(BrTextBox);
        GotFocus += BrTextBox_GotFocus;
        LostFocus += BrTextBox_LostFocus;
        Loaded += BrTextBox_Loaded;
        TextChanged += BrTextBox_TextChanged;
    }

    private void BrTextBox_TextChanged(object sender, TextChangedEventArgs e)
    {
        // In case viewmodels clear the string
        if (string.IsNullOrWhiteSpace(Text) && hasFocus == false)
        {
            VisualStateManager.GoToState(this, StateCollapse, true);
        }
    }

    private void BrTextBox_Loaded(object sender, RoutedEventArgs e)
    {
        if (string.IsNullOrWhiteSpace(Text))
        {
            VisualStateManager.GoToState(this, StateCollapse, true);
        }
        else
        {
            VisualStateManager.GoToState(this, StateExpand, true);
        }
    }

    private void BrTextBox_LostFocus(object sender, RoutedEventArgs e)
    {
        hasFocus = false;

        if (string.IsNullOrWhiteSpace(Text) == true)
        {
            VisualStateManager.GoToState(this, StateCollapse, true);
        }
    }

    private void BrTextBox_GotFocus(object sender, RoutedEventArgs e)
    {
        hasFocus = true;

        if (string.IsNullOrWhiteSpace(Text) == true)
        {
            VisualStateManager.GoToState(this, StateExpand, true);
        }
    }
}

 

Animation using the UIThread

Some of the animations (SplitBrTextBoxStyle & SlideBrTextBoxStyle) requires the UIThread for to create the visual transition through animation. For this these the two styles will have the Property EnableDependentAnimation="True" in some of their animations.
This means the animation will use the UIThread rather than the compositor to animate the value change in the properties.
This is done because e.g. the width / height because these properties can't be visually transformed without the UIThread redrawing the control for each frame of the animation.
Ultimately the use of UIThread for animations can lead to a decrease in UI performance while animating and should be avoided at all costs unless you have no other option. 

 

Final words

Creating the control was rather fun and there's a lot of unexplored possibilities. The control as it is right now doesn't contain any other visual states other than Collapse and Expanded. For the same reason it's not ready for a production environment but was created to give you a general idea of the possibilities.

Feel free to reach out to me with your thoughts on the control and the styles. I'd love to hear from you in the comments. 

/Deani

P.S Make sure you follow me on twitter @deanihansen for more news, design tips, articles and how-to's.

Slide in menu control for Windows Phone

One of the most used menu navigational patterns right now is the hamburger icon / slide in menu.

The pattern is nothing something I'm particular fond of. Mainly because the icon doesn't look good, the icon takes up space and it's highly overused.
I've always felt the same way about the appbar. It doesn't look good but it's effective.

I've thought for some days how the concept two concepts could be merged together effectively and while having the full benefits of making it look good and maintaining the vertical scroll functionality. 

The solution is a two sided hidden menu wrapped in a easy to use control.
I present to you: 
SlideInMenuContentControl.
It contains one menu for navigation and one for context.

The control is free to use and available for download.

 

Download source and demo


 

 

Download source with buttons

UPDATE - SAMPLE WITH BUTTONS TO TRIGGER MENU
I've added a sample with buttons to slide out the menu's! Important: The trigger buttons need ManipulationMode="All" in order to keep the focus away from the flipvew. If the ManipulationMode isn't set the flipview keeps focus and the SelectedIndex will get reset.


Using the SlideInMenuContentControl

Using the control is simple. Simply replace the ". . ." with your xaml. 

  <local:SlideInMenuContentControl>
        <local:SlideInMenuContentControl.LeftMenuContent>
            <Grid>
                   ...
            </Grid>
        </local:SlideInMenuContentControl.LeftMenuContent>
        <local:SlideInMenuContentControl.RightMenuContent>
            <Grid>
                   ...
            </Grid>
        </local:SlideInMenuContentControl.RightMenuContent>
        <local:SlideInMenuContentControl.Content>
            <Grid>
                   ...
            </Grid>
        </local:SlideInMenuContentControl.Content>
    </local:SlideInMenuContentControl>


SlideInMenuContentControl configuration options:

Menu state (Both, Right, Left): Property to determine which sides of the menu should be visible in. Default is both.
RightSideMenuWidth: Property to determine the width of the slidein. Less than 200 is not recommended. Default is 250.0 px.
LeftSideMenuWidth: Property to determine the width of the slidein. Less than 200 is not recommended. Default is 250.0 px.
GoToMenuState(ActiveState state): Method for forcing the control to go to a specific menu.


A huge shout out to my friend Daniel Vistisen for helping and providing technical insights about building custom controls. He's one of those guys who can built anything in xaml and C#.
He's one to follow for sure! 

Feel free to tell me about your experience using the control and ask if you have any questions. 

P.S Make sure you follow me on twitter @deanihansen for more controls, design tips, articles and how-to's.

Implementing Image blur and gaussian blur in a Windows universal app

Image blur and gaussian blur can give a nice visual effect if used right . Blurred images for backgrounds can give a nice subtle effect and for the same reason it has been used by designers for years. However it's not supported out of the box on Windows Universal apps platform. Luckily there's atleast  two different nuget packages which can help us get started. 
I tried both the Lumia Imaging SDK 2.0 and Win2D. 

Between the two I see Win2D as the winner because the rendering quality, speed and amount of documentation. The downside is  that you don't have the same layout flexibility out of the box as with the Lumia Image SDK. 
Below you'll find source for both implementations.

Win2D Gaussian Blur

Tap to see the Gaussian Blur rendering. The rendering quality way better than the Lumia Image SDK.

Win2D is an easy-to-use Windows Runtime API for immediate mode 2D graphics rendering. It makes use of the Direct2D engine and utilizes the GPU to create the new blur effect. 
Win2D comes with a great documentation that gets you started in seconds and it works on both Windows Phone and Windows Store. 
The rendering is smooth, crisp and it's really really fast. Only downside is that you have to work the with the Win2D CanvasControl and CanvasBitmap rather than a regular Bitmap or WriteableBitmap.

 

Download demo and sample source

 

Lumia Image SDK 2.0

Tap to see the Blur rendering. The rendering quality not quite as good as the Win2D rendering.

The Lumia Image SDK is a Nokia and now Microsoft owned SDK created to provide easy-to-use filtering and image manipulation capabilities. The Image SDK is known for being fast - especially when working with live camera feeds and it has been a part of the platform for a while. I assume it makes use of the GPU but didn't find any material on the core of the Lumia Image SDK.

The rendering is not nearly as smooth as the Win2D though. If you compare the images it looks like the Lumia Image SDK makes use of a blur algorithm that blurs in squares. Not nearly is neat and good looking as Win2D but on the upside you get to work directly with Bitmaps and Writeable bitmaps with offers layout flexibility compared to Win2D where you have to some of the calculations yourself to get the image in the right size. 

Download source and sample

 

Wrapup

Win2D is the clear winner here though the Lumia SDK is a bit easier to implement but offers less flexibility and quality. A huge shout out to Rodrigo Diaz (@r2d2rigo) for introducing me to Win2D! He's a absolutely brilliant and clever guy. One to follow for sure! 

If you prefer not downloading the source you can find it below this section.

Feel free to contact contact me or write in the comments below if you have any questions or problems getting the sample to work.

P.S Make sure you follow me on twitter @deanihansen for more news, design tips, articles and how-to's.


Code snippets for Lumia Image SDK blur and Win2D Gaussian Blur

Lumia Image SDK implementation:

1) This code snippet is a part of a templated control. The template contains an Image named "imagePresenter":

private async Task RenderImage()
{
    if (!string.IsNullOrWhiteSpace(ImageUrl))
    {
        var imageControl = GetTemplateChild("imagePresenter") as Image;

        if (imageControl != null)
        {
            var client = new HttpClient();

            // or get stream locally
            var stream = await client.GetStreamAsync(ImageUrl);

            var source = new Lumia.Imaging.StreamImageSource(stream, ImageFormat.Undefined);

            var source = new Lumia.Imaging.StreamImageSource(stream, ImageFormat.Undefined);

            var info = await source.GetInfoAsync();

            double displayScaling = DisplayInformation.GetForCurrentView().LogicalDpi / 96.0;


            double pixelWidth = Window.Current.Bounds.Width * displayScaling;

            var scalefactor = pixelWidth / info.ImageSize.Width;

            using (var filterEffect = new FilterEffect(source))
            {
                WriteableBitmap target = new WriteableBitmap((int)pixelWidth, (int)(info.ImageSize.Height * scalefactor));

                var filter = new BlurFilter(45);

                filterEffect.Filters = new IFilter[] { filter };

                using (var renderer = new WriteableBitmapRenderer(filterEffect, target, OutputOption.PreserveAspectRatio))
                {
                    await renderer.RenderAsync();

                    imageControl.Source = target;
                }
            }
        }
    }
}

Win2D Gaussian blur implementation

1) Create the CanvasControl which will host our Image and hook into the two events. Finally set the CanvasControl as Content on a contentpresenter or control.

control = new CanvasControl();
control.Draw += OnDraw;
control.CreateResources += OnCreateResources;
contentPresenter.Content = control;

2) Create and load the image from a URL

private async void OnCreateResources(CanvasControl sender, object args)
{
    if (!string.IsNullOrWhiteSpace(ImageUrl))
    {
        scaleEffect = new ScaleEffect();
        blurEffect = new GaussianBlurEffect();

        image = await CanvasBitmap.LoadAsync(sender.Device,
          new Uri(ImageUrl));

        imageLoaded = true;

        sender.Invalidate();
    }
}

3) On Draw we take the draw session and apply a scale effect to resize to the appropriate siize for the screen and DPI. Afterwards the blur effect is applied and finally the draw is called to render the image onto the Canvas.

private void OnDraw(CanvasControl sender, CanvasDrawEventArgs args)
{
    if (imageLoaded)
    {
        using (var session = args.DrawingSession)
        {
            session.Units = CanvasUnits.Pixels;

            double displayScaling = DisplayInformation.GetForCurrentView().LogicalDpi / 96.0;

            double pixelWidth = sender.ActualWidth * displayScaling;

            var scalefactor = pixelWidth / image.Size.Width;

            scaleEffect.Source = this.image;
            scaleEffect.Scale = new Vector2()
            {
                X = (float)scalefactor,
                Y = (float)scalefactor
            };

            blurEffect.Source = scaleEffect;
            blurEffect.BlurAmount = Blur;

            session.DrawImage(blurEffect, 0.0f, 0.0f);
        }
    }
}

HorizontalScrollViewerStyle and VerticalScrollViewerStyle for Windows apps

Back in the days when Windows Store 8.0 was shipped Microsoft bundled a nice set of standard styles in the file StandardStyles.xaml. While I never used most included styles there was two styles I used repeatedly through out almost all projects and that was HorizontalScrollViewerStyle and VerticalScrollViewerStyle.

The benefits of the two styles was that they disabled all the properties you would expect to be disabled. With the styles your scrollviewer disables zoom and only scrolls the appropriate direction. 

HorizontalScrollViewerStyle

 <Style
        x:Key="HorizontalScrollViewerStyle"
        TargetType="ScrollViewer">
        <Setter
            Property="HorizontalScrollBarVisibility"
            Value="Auto" />
        <Setter
            Property="VerticalScrollBarVisibility"
            Value="Disabled" />
        <Setter
            Property="ScrollViewer.HorizontalScrollMode"
            Value="Enabled" />
        <Setter
            Property="ScrollViewer.VerticalScrollMode"
            Value="Disabled" />
        <Setter
            Property="ScrollViewer.ZoomMode"
            Value="Disabled" />
    </Style>


VerticalScrollViewerStyle

    <Style
        x:Key="VerticalScrollViewerStyle"
        TargetType="ScrollViewer">
        <Setter
            Property="HorizontalScrollBarVisibility"
            Value="Disabled" />
        <Setter
            Property="VerticalScrollBarVisibility"
            Value="Auto" />
        <Setter
            Property="ScrollViewer.HorizontalScrollMode"
            Value="Disabled" />
        <Setter
            Property="ScrollViewer.VerticalScrollMode"
            Value="Enabled" />
        <Setter
            Property="ScrollViewer.ZoomMode"
            Value="Disabled" />
    </Style>

Feel free to let me know in the comments if you have any comments or improvements to the code above.

P.S Make sure you follow me on twitter @deanihansen for more news, design tips, articles and how-to's.