.NET Native, SilverlightSerializer, MakeGenericType and MissingMetadataException

Recently I have ported my app Map Mania to the  Universal Windows Platform, and at the moment of this writing it is being certified. (that, I least I hope). It’s the first UWP app I actually submit to the Store. I have been doing quite some IoT Core stuff and that does not require much submitting. But it had to happen at one time, if only to dogfood my WpWinNl port to UWP, and it was an educating experience.

I learned the hard way that stuff that does work with debug settings, does not necessarily work when you compile stuff ‘for real’. I ran into quite a serious issue with generics. As soon as I turned my app over to the .NET Native toolchain, it even crashed on startup. Mind you, most of the ‘business code’ in this app and large parts of the XAML were still the same as the original Windows Phone7 and 8 app, as was the WpWinNl base library – all ported to UWP, but largely unchanged.

The root cause of the problem turned out to be SilverlightSerializer. This is a serialization framework made by Mike Talbot that I have been using to store app state since 2011. It’s name quite dates it origins. Unfortunately the link to the original article is dead – actually the whole blog seems to have disappeared - but it’s code has been sitting in WpWinNl and it’s predecessors ever since that time. I have ported it to Windows Phone 8, Windows RT, and now to UWP.

Deep down in the core it uses the Type.MakeGenericType method to make a generic type with two type parameters to store types. So if I have ViewModel with a double property, it makes a GetSetGeneric<ViewModel, double>. This, now, does not go down well with .NET Native. Runtime, it will pop up the following error:

Exception thrown: 'System.Reflection.MissingMetadataException' in System.Private.CoreLib.dll

Additional information: Reflection_InsufficientMetadata_EdbNeeded: WpWinNl.Utilities.GetSetGeneric<BeautyOfMaps.Models.TileMapSource,System.Boolean>. For more information, visit http://go.microsoft.com/fwlink/?LinkId=623485

Now, in your app, in  the “Properties” folder, there is a Default.rd.xml file that allows you to instruct the compiler not to ‘optimize things away’, to make sure this code still works. The insidious part is that this code is in my libary, downloaded from NuGet, and since it uses generics there is no ‘'generic’ way for me to instruct the compiler in the libary’s rd.xml not to optimize away code in the app that is using it.

If I wanted to fix this particular problem, I have to add the following line to the app’s rd.xml:

<Type Name="WpWinNl.Utilities.GetSetGeneric{BeautyOfMaps.Models.ViewModel.MainViewModel,System.Boolean}" Dynamic="Required All"/>

And guess what – then you get the next error. Now it it’s complaining about missing

WpWinNl.Utilities.GetSetGeneric<LocalJoost.Maps.TileSources.Google,
LocalJoost.Maps.TileSources.GoogleType>

So it apparently needs a line like this for every class type and property type combination that goes past MakeGenericType. So while you already defined in code what these properties are, you have to declare (again) in XML which ones you want to keep. I have been in contact with the .NET Native team about the apparent lack of logic behind this, but is seems that according to their date very little people actually use this kind of reflection. While this is true without any doubt – Microsoft telemetry does not lie - I think the critical thinking error that was made here is that those people might be using libraries that do – like mine! - and then they are thoroughly in deep… er, manure.

Trying to add manually all class/property type combinations is where madness lies, my friends. I came to about the 6th type and then I was kind of done. I am a developer, so I am lazy by nature, and in stead of actually doing it manually I added a weird property to SilverlightSerializer called RdXmlEntries. It’s usage is as follows:

• Run the app in DEBUG (so not .NET Native)
• Serialize all objects you want to serialize
• Directly behind the last serialization, add this code:

foreach (var entry in SilverlightSerializer.RdXmlEntries)
{
  Debug.WriteLine(entry);
}

• Copy and paste the resulting data in your Default.rd.xml
• Verify your code now runs under .NET Native.

This code is now in WpWinNlBasic version 3.0.5-alpha on NuGet and available for use. You now know why I keep this in the unstable branch – as long as I haven’t had the chance to dogfood all the stuff I have ported and created earlier, I want to thread carefully.

When I showed this solution to the .NET Native team I got feedback that I only needed to provide lines for when the second parameter is a value type, not when it’s a reference type. I have validated that in one case, but I serialize quite a complicated object graph and had spent more than enough time on it already, I did not really feel like experimenting further.

Bottom line – be very, very careful with advanced reflection in UWP apps. Don’t make the same mistake I made – test early under .NET Native and follow guidance provided here to maintain your sanity.

Special thanks to Tom Verhoeff for providing assistance based upon his earlier experience. And sorry, no code sample this time as this is more general guidance that a specific coding issue ;)

Tombstoning MVVMLight ViewModels with SilverlightSerializer on Windows Phone 7

I’ve been down this road before but I thought it wise to revisit this subject, since I see a lot of people in the Windows Phone 7 developer community still struggling with the concept of tombstoming. In my previous attempt to make some universal way of tombstoning ViewModels on Windows Phone 7 I used DataContractSerializer, which is a nice idea but does not work very well with MVVMLight since the ViewModelBase is not serializable. And sometimes you have to go trough a lot of hooplah if stuff you use in your code does not turn out to be serializable after all.There is a better way, I think. So here it is: universal tombstoning for MVVMLight, take two.

Enter SilverlightSerializer by the as far as I am concerned immortal Mike Talbot. Download this thing and store it somewhere in your sources. To use it for tombstoning, I have created the following extension methods:

using System;
using System.IO;
using System.IO.IsolatedStorage;
using System.Windows;
using GalaSoft.MvvmLight;

namespace LocalJoost.Utilities
{
  public static class ApplicationExtensions
  {
    private static string GetIsFile( Type t)
    {
      return string.Concat(t.Name, ".dat");
    }

    public static T RetrieveFromIsolatedStorage<T>(this Application app) 
      where T : class
    {
      using (var userAppStore = 
         IsolatedStorageFile.GetUserStoreForApplication())
      {
        var dataFileName = GetIsFile(typeof(T));
        if (userAppStore.FileExists(dataFileName))
        {
          using (var iss = userAppStore.OpenFile(dataFileName, FileMode.Open))
          {
             return SilverlightSerializer.Deserialize(iss) as T;
          }
        }
      }
      return null;
    }

    public static void SaveToIsolatedStorage(this Application app, 
                                      ViewModelBase model)
    {
      var dataFileName = GetIsFile((model.GetType()));
      using (var userAppStore = 
               IsolatedStorageFile.GetUserStoreForApplication())
      {
        if (userAppStore.FileExists(dataFileName))
        {
          userAppStore.DeleteFile(dataFileName);
        }
        using (var iss = userAppStore.CreateFile(dataFileName))
        {
          SilverlightSerializer.Serialize(model,iss);          
        }
      }
    }
  }
}

Then I go to the App.xaml.cs and modify it as depicted next:

private void Application_Launching(object sender, LaunchingEventArgs e)
{
  LoadModel();
}

private void Application_Activated(object sender, ActivatedEventArgs e)
{
  LoadModel();
}

private void LoadModel()
{
  try
  {
    MyMainViewModel.Instance = this.RetrieveFromIsolatedStorage<MyMainViewModel>();
  }
  catch (Exception){ }
  if( MyMainViewModel.Instance == null) MyMainViewModel.CreateNew(); 
}

private void Application_Deactivated(object sender, DeactivatedEventArgs e)
{
  this.SaveToIsolatedStorage(MyMainViewModel.Instance);
}

private void Application_Closing(object sender, ClosingEventArgs e)
{
  this.SaveToIsolatedStorage(MyMainViewModel.Instance);
}

In red you see my additions to the standard generated App.xaml.cs. You will also need to add at least one “using” statement (using LocalJoost.Utilities) and a second one with the namespace in which you have put MyMainViewModel. Notice the fact that there is an empty try-catch around the RetrieveFromIsolatedStorage call. This is intentional. Since binary deserialization tends to throw exceptions if you have changed the ViewModel’s code, you always want to make sure your application gets served a working ViewModel where ever it needs to come from.

The skeleton of a MainViewModel looks something like this in my household:

public class MyMainViewModel : ViewModelBase
{
  public MyMainViewModel( )
  {
  }  

  private static MyMainViewModel _instance;
  public static MyMainViewModel Instance
  {
    get {return _instance;}
    set { _instance = value; }
  }

  public static void CreateNew()
  {
    if (_instance == null)
    {
      _instance = new MyMainViewModel();
    }
  }
}

If your want certain properties of your ViewModel specifically not to be serialized (because they come from a – in this example omitted – application model) you can mark them with the [DoNotSerialize] attribute that comes with SilverlightSerializer.

You can now always bind to MyMainViewModel.Instance and you will either get a fresh new or a retrieved from isolated storage ViewModel. The fun thing is that although MVVMLight’s ViewModelBase is not serializable, the SilverlightSerializer does in fact serialize it which makes life a whole lot easier.

You can, by the way, also decide to store the ViewModel on the PhoneApplicationService on deactivation and retrieve it there on activation. IMHO always storing on isolated storage gives a more consistent feeling to an application. But that’s just me.

A few important points to conclude this post:

• A ViewModel that is to be serialized on isolated storage should always have a default constructor.
• Properties of the ViewModel that you want to have serialized need to have a getter and a setter (believe me, you can spend some time overlooking this).
• Do not try something clever like this on the static instance property of your ViewModel:

public static MyMainViewModel Instance
{
  get {return _instance ?? (_instance = new MyMainViewModel());}
  set { _instance = value; }
}

If you do this, MyMainViewModel.Instance = this.RetrieveFromIsolatedStorage will first call the getter, thus creating a new instance of MyMainViewModel whatever happens next, then overwrite it with the retrieved version. But if you do something like registering a message in the constructor of MyMainViewModel, the initially created model will stay alive ‘somewhere’ - you will end up with two instances of your supposedly singleton instance running in you App, both listening to the same message and reacting to it - and spend, like me, a not so nice and probably extended period of  time scratching your head and wondering what the hell is going on.

I hope that with this post I once and for all made it easier for the Windows Phone 7 community to tackle something apparently tricky as tombstoning thus improving the quality of what we are all going to put in the App Hub.

Extension methods for tomb stoning the Windows Phone 7 model

As I was (and still am) playing with my MVVM-driven Windows Phone 7 map viewer, I started thinking about tomb stoning and/or serializing stuff into the Windows Phone 7 application state and/or isolated storage. I came up with four extension methods than can be used to store/serialize the complete model in both. Maybe not ideal in all cases, but it was a simple solution for me.

Getting started

I created an assembly LocalJoost.Utilities, in which I first defined a generic interface for the model, which I called – with a flash of inspiration – IModel ;-). It’s implementation is not interesting for the extension methods.

Save to phone application state

using System;
using System.IO;
using System.IO.IsolatedStorage;
using System.Runtime.Serialization;
using System.Windows;
using Microsoft.Phone.Shell;

namespace LocalJoost.Utilities
{
  /// <summary>
  /// Some extensions method that allow serializing and deserializing
  /// a model to and from phone state and/or isolated storage
  /// </summary>
  public static class ApplicationExtensions
  {
    private const string ModelKey = "model";
 
    public static IModel RetrieveFromPhoneState(this Application app) 
    {
      if (PhoneApplicationService.Current.State.ContainsKey(ModelKey))
      {
        return PhoneApplicationService.Current.State[ModelKey] as IModel;
      }
      return null;
    }

    public static void SaveToPhoneState(this Application app, IModel model)
    {
      if (PhoneApplicationService.Current.State.ContainsKey(ModelKey))
      {
        PhoneApplicationService.Current.State.Remove(ModelKey);
      }
      PhoneApplicationService.Current.State.Add(ModelKey, model);
    }
  }
}

Creating a Locator

I wanted my model to be bindable by XAML and be usable from code as well, so I created the following ‘Locator’:

using LocalJoost.Utilities;

namespace LocalJoost.Models
{
  public class Locator
  {
    private static IModel _model;
    private static Locator _locator;

    public IModel Model
    {
      get { return _model; }
      set { _model = value; }
    }

    public static Locator Instance
    {
      get { return _locator ?? (_locator = new Locator()); }
    }
  }
}

Using Locator and Model from XAML

First, add the namespace to the XAML

xmlns:Utilities="clr-namespace:LocalJoost.Utilities;assembly=LocalJoost.Utilities"

Then bind it to your Layout root or any other element of choice

 <Grid x:Name="LayoutRoot"
  DataContext="{Binding Source={StaticResource Locator}, Path=Model}">

Instantiating your model

Interesting caveat – the Locator is instantiated, not your model. When the application runs for the very first time, there will be no model to retrieve. Thus, in the constructor of App.Xaml.cs you need to add:

Locator.Instance.Model = new YourModel();

in which you replace "YourModel" for your own model type.

Storing in / retrieving from phone application state

Using the extension methods is pretty simple then: open your App.Xaml.cs, find two methods called “Application_Activated” and “Application_Deactivated” and modify them as follows:

// Code to execute when the application is activated (brought to foreground)
// This code will not execute when the application is first launched
private void Application_Activated(object sender, ActivatedEventArgs e)
{
  Locator.Instance.Model = this.RetrieveFromPhoneState();
}

// Code to execute when the application is deactivated (sent to background)
// This code will not execute when the application is closing
private void Application_Deactivated(object sender, DeactivatedEventArgs e)
{
  this.SaveToPhoneState(Locator.Instance.Model);
}

And that’s it. Your model now persists to the phone state and survives, for instance, a hit on the search button

Storing in Isolated Storage

This proves to be marginally more difficult. I expanded the class ApplicationExtensions with two more methods and a const:

private const string DataFile = "model.dat";

public static T RetrieveFromIsolatedStorage<T>(this Application app) 
  where T : class
{
  using (var appStorage = IsolatedStorageFile.GetUserStoreForApplication())
  {
    if (appStorage.FileExists(DataFile))
    {
      using (var iss = appStorage.OpenFile(DataFile, FileMode.Open))
      {
        try
        {
          var serializer = new DataContractSerializer(typeof(T));
          return serializer.ReadObject(iss) as T;
        }
        catch (Exception e)
        {
          System.Diagnostics.Debug.WriteLine(e);
        }
      }
    }
  }
  return null;
}

public static void SaveToIsolatedStorage(this Application app, IModel model)
{
  using (var appStorage = IsolatedStorageFile.GetUserStoreForApplication())
  {
    if (appStorage.FileExists(DataFile))
    {
      appStorage.DeleteFile(DataFile);
    }
    var serializer = new DataContractSerializer(model.GetType());
    using (var iss = appStorage.CreateFile(DataFile))
    {
      serializer.WriteObject(iss, model);
    }
  }
}

You now have to find Application_Launching and Application_Closing in App.Xaml.cs and modify them in a similar way:

// Code to execute when the application is launching (eg, from Start)
// This code will not execute when the application is reactivated
private void Application_Launching(object sender, LaunchingEventArgs e)
{
    Locator.Instance.Model = 
       this.RetrieveFromIsolatedStorage<YourModel>();
}

// Code to execute when the application is closing (eg, user hit Back)
// This code will not execute when the application is deactivated
private void Application_Closing(object sender, ClosingEventArgs e)
{
    this.SaveToIsolatedStorage(Locator.Instance.Model);
}

Some notes

• In order to make this work, your need to mark your model class with the [DataContract] attribute, and every member you want to be serialized with [DataMember]
• If you are using MVVMLight (which of course you are ;-) ) and if you are just making simple apps that don’t use separate models but just view models, be aware that your model cannot inherit from ViewModelBase, for the simple reason that this class is not serializable. Oops ;-)
• If your model communicates with the view using attached dependency properties you may run into some timing difficulties. I will soon blog about a work-around for that.

Using extension methods to serialize objects to XML and compress the result - and deserialize again

Elaborating upon the string extension methods I created for compressing and decompressing strings to and from a byte array, it turned out fairly easy to create another set of compress / decompress methods to serialize any type of object to and from a byte array containing compressed XML. I once again took the StringZipExtensions class and added the following methods to compress any old object to a byte array:

/// <summary>
/// XmlSerializes the object to a compressed byte array
/// using the specified encoding.
/// </summary>
/// <param name="objectToCompress">The object to compress.</param>
/// <param name="encoding">The encoding.</param>
/// <returns>bytes array with compressed serialized object</returns>
public static byte[] Compress(this object objectToCompress,
  Encoding encoding)
{
  var xmlSerializer = new XmlSerializer(objectToCompress.GetType());
  using (var stringWriter = new StringWriter())
  {
    xmlSerializer.Serialize(stringWriter, objectToCompress);
    return stringWriter.ToString().Compress(encoding);
  }
}

/// <summary>
/// XmlSerializes the object to a compressed byte array using default
/// UTF8 encoding.
/// </summary>
/// <param name="objectToCompress">The object to compress.</param>
/// <returns>bytes array with compressed serialized object</returns>
public static byte[] Compress(this object objectToCompress)
{
  return Compress(objectToCompress, new UTF8Encoding());
}

Here, once again, an overload using a default UTF8 encoding and a method which uses your own encoding to do the heavy lifting. Then, the methods for decompressing, which - in all modesty - are a rather neat usage of generics, I think:

/// <summary>
/// Decompress an array of bytes into an object via Xml Deserialization
/// using the specified encoding
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="compressedObject">The compressed string.</param>
/// <param name="encoding">The encoding.</param>
/// <returns>Decompressed object</returns>
public static T DecompressToObject<T>(this byte[] compressedObject,
  Encoding encoding)
{
  var xmlSer = new XmlSerializer(typeof(T));
  return (T)xmlSer.Deserialize(new StringReader(
    compressedObject.DecompressToString(encoding)));
}

/// <summary>
/// Decompress an array of bytes into an object via Xml Deserialization
/// using default UTF8 encoding
/// </summary>
/// <param name="compressedObject">The compressed string.</param>
/// <returns>Decompressed object</returns>
public static T DecompressToObject<T>(this byte[] compressedObject )
{
  return DecompressToObject<T>(compressedObject, new UTF8Encoding());
}

Then you can take something like this hugely complex ;-) object Person

public class Person
{
  public int Id { get; set; }
  public string Name { get; set; }
  public DateTime Birthday { get; set; }

  public override bool Equals(object obj)
  {
    var toCompare = obj as Person;
    if( toCompare != null )
    {
      return
        Name.Equals(toCompare.Name) &&
        Id.Equals(toCompare.Id) &&
        Birthday.Equals(toCompare.Birthday);

    }
    return base.Equals(obj);
  }
}

and serialize/compress and decompress/deserialize it like this:

[Test]
public void CompressObjectTest()
{
  var baseLineObject = new Person
  {
  Id = 99, 
  Name = "Tom", 
  Birthday = new DateTime(1969, 06, 03)
  };
  var compressed = baseLineObject.Compress();
  var testObject = compressed.DecompressToObject<Person>();
  Assert.AreEqual(testObject, baseLineObject);
}

Code downloadable here

Storing objects as compressed messages in the Windows Azure Queue

For my current private R&D project I wanted to store 'task set' objects (in code samples below as shown type "Request") on the Windows Azure Queue. To prevent serialization issues I opted for XML serialization, like this:

private void Enqueue(Request tr)
{
  var queueStorage = QueueStorage.Create(
  StorageAccountInfo.GetDefaultQueueStorageAccountFromConfiguration());
  var queue = queueStorage.GetQueue("MyQueue");
  if (!queue.DoesQueueExist())
  {
    queue.CreateQueue();
  }

  var xmlSerializer = new XmlSerializer(tr.GetType());
  using (var stringWriter = new StringWriter())
  {
    xmlSerializer.Serialize(stringWriter, tr);
    queue.PutMessage(new Message(stringWriter.ToString()));
  }
}

I would let the worker role deserialize the Request object and then execute it. It annoyed me to no end to learn that the Azure Queue limits message sizes to 8192 bytes. I could have redesigned my task sets to smaller units, but that would hurt the efficiency of the process I had in mind. Based upon the StringZipExtensions class I blogged about that can serialize and deserialize any old object to and from compressed XML (which you can download here) I created the following extension methods, which enable you to store objects as a GZip compressed set of bytes on the Azure queue and retrieve them again:

using LocalJoost.Utilities.Compression;
using Microsoft.Samples.ServiceHosting.StorageClient;

namespace LocalJoost.Utilities.Azure
{
  public static class MessageQueueExtensions
  {
    /// <summary>
    /// Decompresses the specified queue message.
    /// </summary>
    /// <param name="message">The message.</param>
    /// <returns></returns>
    public static string Decompress(this Message message)
    {
      return message != null ?
       message.ContentAsBytes().DecompressToString() : null;
    }

    /// <summary>
    /// Decompresses the specified queue message.
    /// to an object
    /// </summary>
    /// <param name="message">The message.</param>
    /// <returns></returns>
    public static T Decompress <T>(this Message message) where T:class
    {
      return message != null ?
       message.ContentAsBytes().DecompressToObject<T>() : null;
    }
  }
}

Adhering to my first code sample, you can now simply put an object to the queue like this

private void Enqueue(Request tr)
{
  var queueStorage = QueueStorage.Create(
    StorageAccountInfo.GetDefaultQueueStorageAccountFromConfiguration());
  var queue = queueStorage.GetQueue("MyQueue");
  if (!queue.DoesQueueExist())
  {
    queue.CreateQueue();
  }

  queue.PutCompressedObject(tr);
}

and let the worker role use the Decompress extension method on the Message itself:

var queueStorage =
  QueueStorage.Create(
  StorageAccountInfo.GetDefaultQueueStorageAccountFromConfiguration());
var queue = queueStorage.GetQueue("MyQueue");
if (queue.DoesQueueExist())
{
  var message = queue.GetMessage(600);
  if (message != null)
  {
    var request = message.Decompress<Request>();
    request.Execute();
    queue.DeleteMessage(message);
  }
}

And there you go. It is as simple as that. Although the Window Azure Queue message queue size still is limited to 8192, the amount of data that fits in that space is increased dramatically.