Custom XML Serialization for the .NET Compact Framework

.NET provides a whole slew of utilities for serializing objects into an XML form. But as I wrote in my previous post, .NET Compact Framework has serious problems with this serialization. The good news is that you can leverage all of the existing Attributes and tricks that you think should work (if it weren’t so buggy) and use them in your own serialization scheme.

Get Started

For example I want to know if I should skip a given member? There are a number of different things I can check. Is a Reference type null? Is there and XmlIgnore attribute? Is there a PropertyNameSpecified value set to false? All of those questions can easily be answered using reflection.


///

/// Should the current property be skipped based on rules
/// such as the existence of a propertySpecified value set to false?
///

/// The MemberInfo to check /// The object that contained this member /// true if this member should be skipped
public bool SkipMember(MemberInfo member, object o)
{
object val = null;
if (member.MemberType == MemberTypes.Field)
{
val = ((FieldInfo)member).GetValue(o);
}
else if (member.MemberType == MemberTypes.Property)
{
val = ((PropertyInfo)member).GetValue(o, null);
}

if (null == val)
return true;

string propertyToTest = member.Name + "Specified";

PropertyInfo specifiedProperty = o.GetType().GetProperty(propertyToTest);
if ((null != specifiedProperty && !(bool)specifiedProperty.GetValue(o, null)))
return true;

FieldInfo specifiedField = o.GetType().GetField(propertyToTest, FIELD_BINDING_FLAGS);
if ((null != specifiedField && !(bool)specifiedField.GetValue(o)))
return true;

return member.IsDefined(typeof(XmlIgnoreAttribute), false);
}

I can use a similar “fall-through” strategy to determine the name of the element to write using the XmlElement attribute for example. Now that I know I can answer some basic questions about an Object using the built-in mechanisms that .NET uses for serialization I can get down to serious serialization.

We’re all Object-Oriented programmers these days right? Right!? So to start I decided that the best way to handle this problem was to decompose it into a bunch of simpler problems.

ITagWriter

There are two things that we can write in XML. Either an XML Element or an XML Attribute. So, I created an interface ITagWriter with two concrete implementations to correspond to these two XML types: AttributeTagWriter and ElementTagWriter. These classes allow me to write the structure of the XML Document.


///

/// Interface to implement to write different Xml tags
/// Either Elements or Attributes.
///

internal interface ITagWriter
{
///

/// Write the opening Xml tag with the given name
///

/// The XML Document to write the tage to. /// The name of the tag void WriteStart(XmlWriter doc, string tagName);

///

/// Write the appropriate end tag
///

/// The XML Document to write the tage to. void WriteEnd(XmlWriter doc);
}

IValueWriter

With the ability to write the structure, I then need to be able to write out the values of the various objects and their properties. Just like with the ITagWriter interface, I decided to create an IValueWriter for the various kinds of values that I would need to write. The types I came up with were ObjectWriter, CollectionValueWriter, EnumValueWriter, SimpleValueWriter, and XmlElementValueWriter.


///

/// Interface to implement to write different kinds of values.
///

internal interface IValueWriter
{
///

/// Write the Entry value to the XmlDocument
///

/// The XML Document to write the tage to. /// The meta-information and value to write. void Write(XmlWriter doc, CustomSerializationEntry entry);
}

You’ll notice the CustomSerializationEntry class is the parameter for the IValueWriter.Write() method. This class contains all of the metadata and the value about the various properties of an Object. This alows us an easy way to ask questions about a given property. Is it a Collection? Is it an Enum? Is there a sort order? Basically the idea is to encapsulate all of the things that are interesting from a serialization point of view.

To help manage the interaction I also created a basic TypeLookup class. The job of this class is to determine what type of ITagWriter and IValueWriter to use for a given CustomSerializationEntry instance. This allows us to centralize that decision making in a single class. The centralized knowledge keeps the individual writer implementations much simpler. They just need to ask for the correct writer and then call the methods defined in the interface. They don’t need to care what type they are writing. All hail power of encapsulation and abstraction!

Start Serializing

I bootstrap the serialization by creating an ObjectWriter to handle the outermost object. From there, the ObjectWriter takes over, constructing CustomSerializationEntry objects for each of the serialized object’s properties. The type of the property determines the type of IValueWriter that is used to write the property value.


///

/// Serialize an object using the given
/// xmlRoot as the root element name.
///

/// /// ///
public string Serialize(object o, string xmlRoot)
{
StringBuilder sb = new StringBuilder();
using (XmlTextWriter writer = new XmlTextWriter(new StringWriter(sb)))
{
writer.Formatting = Formatting.Indented;

XmlWriter xmlDoc = XmlWriter.Create(writer);
WriteRootElement(xmlDoc, o, xmlRoot);
}

return sb.ToString();
}

private static void WriteRootElement(XmlWriter doc, object o, string rootElement)
{
doc.WriteStartDocument();

ObjectWriter writer = new ObjectWriter(new TypeLookup());
writer.Write(doc, o, rootElement);

doc.WriteEndDocument();
}

The ObjectWriter itself creates a CustomSerializationEntry for all the properties that should be written. It then loops over the properties. Notice how it uses the TypeLookup (lookup) to ask for the proper value writer for each of the properties.

// ...
public void Write(XmlWriter doc, object o, string elementName)
{
doc.WriteStartElement(elementName);
IEnumerable entries = GetMemberInfo(o);
foreach (CustomSerializationEntry currentEntry in entries)
{
lookup.GetValueWriter(currentEntry).Write(doc, currentEntry);
}
doc.WriteEndElement();
}
// ...

Conclusion

OK, so I left out a lot of details! But If I gave you all of the answers it wouldn’t be any fun now would it. I hope you can see how decomposing the problem of serialization turns it a series of relatively simple problems that you can answer. So, if I can do this in about 500 lines of code, how come Microsoft can’t implement a decent XML Serializer for the .NET Compact Framework.

.NET Compact Framework Serialization Bugs

XML Serialization in the .NET Compact Framework seems to be buggy enough that it is generally not useful if you need to serialize a class that conforms to a schema of some sort. If all you need to do is serialize and deserialize representations of a class you are probably fine. But if you need to use the serialized data to interoperate with a service (for example) it likely will not work.

Enums

I wrote previously about Problems with Enum Serialization on the .NET compact framework. To summarize: The XmlEnum attribute allows you to change the value that is serialized. One of the reasons to do this is to limit the valid values in the document. Of course, Enums have naming restrictions such as not being able to have spaces in them. So, one of the main reasons you would do this would be to put spaces in the name that was serialized.

The problem is that the XmlEnum under the .NET Compact Framework truncates the value at the first space. So, in the example below your Enums would serialize to “Some” and “Another” instead of the correct “Some Value” and “Another Value”.


public enum Foo {
[XmlEnum("Some Value")
Some,
[XmlEnum("Another Value")]
Other
}

Controlling Serialization of Value Types

.NET has reference types and value types. Reference types are stored on the heap and can be null. Value types are stored on the stack and can not be null. What is a value type and what is a reference type is not always obvious. int and double are value types but so is DateTime. string is a reference type though.

The serialization code is smart enough to not serialize null values. But what about those value types? They can not be null, so how do you determine if they should be serialized or not? There are two ways you can do it.

  1. DefaultValue
  2. PropertySpecified

DefaultValue

One way is to specify a default value. If the property has that default value it will not be serialized.

Example:

[XmlAttribute("age"), DefaultValue(-1)]
public int Age;

I have not found any problem with this on the .NET Compact Framework, but I haven’t used it extensively.
DefaultValue really does not make sense for every case. What about a value where negative, positive and zero all make sense? What about a boolean value where true and false are both meaningful and different from null? Valid values are really a business concept and many business concepts will not constrain them in such a way that you can specify a DefaultValue to control serialization, so this is not always useful.

See: more information on DefaultValue.

PropertySpecified

.NET serialization also allows you to specify a control value to check to see if a property should be serialized. The form of the control value is propertynameSpecified. It is a boolean value. If the value is false, then the property it controls will not be serialized. If the value, is true then it will be serialized.

Example:

[XmlIgnore]
public bool FooSpecified = false;

[XmlElement("foo)]
public int Foo {
get { return this.foo; }
set {
this.foo = value;
this.FooSpecified = true;
}
}

This is fine in the full framework. The problem is with the .NET Compact Framework. When the serializer comes across a propertynameSpecified value that is false, the serialization of that class stops. This means if you have 5 properties and the second property has a control value set to false, only the first value will be serialized!

Coming Soon

In a future post, I will write about how you can relatively easily write your own XML Serializer for the .NET Compact Framework in about 500 lines of code.

Update: follow-up post on Writing a Custom XML Serializer has been posted.

More .NET Compact Framework Woes

I posted previously on a Bug in the .NET Compact Framework with the XmlEnum Attribute with whitespace in the name. Well I’ve run into some other interesting “features”.

The first thing to realize is that the things that work on the Full framework don’t work on the Compact Framework.

What Works With Serialization

First thing is the good news: Arrays work everywhere. They work on the Compact Framework and the Full Framework. The downside of course is that Arrays are very inconvenient. You have to manually resize them yourself for example.

The second thing that works are regular ILists. When you map them a collection will be used. Of course one of the big improvements in .NET 2.0 was the introduction of Generics. Generics allow you to have strongly-typed collections without manually implementing them for each specific type.

More Generics Problems

Xml Serialization works if you use IList. But there are problems with Generics interfaces. I guess there are more than problems. Short story is that you can not use IList for XML serialization, it plain just doesn’t work.

Here’s where the difference between the Full Framework and the Compact Framework come into play. On the Full Framework, you can map to a concrete collection wether it’s generic or not. So List will work. Unfortunately this does not work in the Compact Framework.

[XmlElement(Name="foo")]
public List<string> Foos 
{
     get { return this.foos }
     set { this.foos = value; }
}

You end up with an exception:

Two mappings for string[].

Stack Trace:
at System.Xml.Serialization.TypeContainer.AddType()
at System.Xml.Serialization.TypeContainer.AddType()
at System.Xml.Serialization.XmlSerializationReflector.AddIXmlSerializableType()
at System.Xml.Serialization.XmlSerializationReflector.AddType()
at System.Xml.Serialization.XmlSerializationReflector.FindType()
at System.Xml.Serialization.XmlSerializationReflector.FindType()
.....

So What Do You Do About It

You basically have 2 options:

  1. Use Arrays
  2. Create your own custom, strongly-typed collections classes

Arrays

The advantage is that this is very simple and requires no extra code. The downside as I said above is that you have to write code to manually do array resizing.

public T[] AppendItem<T>(T[] theArray, T newItem) {
    T[] newArray = new T[theArray.Length + 1];
    Array.Copy(theArray, newArray, theArray.Length);
    newArray[newArray.Length - 1] = newItem;
    return newArray;
}

Custom, Strongly-Typed Collections

If you don’t want to use Arrays and deal with manually resizing them, you can build your own Collections classes for each of your types.

Create your strongly-typed collection:

[Serializable]
[EditorBrowsable(EditorBrowsableState.Advanced)]
public class EmployeeCollection : ArrayList {
    public Employee Add(Employee obj) {
        base.Add(obj);
       return obj;
    }
 
   public Employee Add() {
       return Add(new Employee());
   }
 
   public void Insert(int index, Employee obj) {
       base.Insert(index, obj);
   }
 
   public void Remove(Employee obj) {
       base.Remove(obj);
   }
 
   new public Employee this[int index] {
       get { return (Employee) base[index]; }
       set { base[index] = value; }
   }
}

Then use that collection in your class to map:

[XmlRoot("company")]
class Company {
    private EmployeeCollection employees;
 
    [XmlElement(Type=typeof(Employee),ElementName="employee",IsNullable=false)]
    [EditorBrowsable(EditorBrowsableState.Advanced)]
    public EmployeeCollection Employees {
        get { return this.employees; }
        set { this.employees = value; }
    }
}

Pick Your Poison

The problems with Generics collections in the .NET Compact Framework seem like yet another bug. So, pick your poison and choose a workaround. Whichever seems simpler to you. Hope that helps someone out.

Using SQL Compact Edition Under ASP.NET

What is SQLCE?

SQL Compact Edition is the “low end” version of a SQL database solution from Microsoft. It is a single-file, application managed, database implementation. It doesn’t have all of the bells and whistles of the high end database solutions. This is great when you realize the next lowest version, SQL Express is over a 100MB install.

The beta of this software was called SQL Everywhere Edition (SQLEV). Microsoft decided that they didn’t like the name so they went with SQL Compact Edition (SQLCE ). The name Compact Edition is a bit of a misnomer. It can be used on the Compact Framework, but it can also be used on the full-framework anywhere a single-file, zero-install database might be needed. Well almost Everywhere at least. There is an explicit check when running on the Full Framework to make sure that you are not using it in an ASP.NET application. SQLCE is not a very scalable solution. It has some inherent limitations with concurrent connections for example. This is fine though if you go back to “what are you using this for”? An embedded, single-file database. Well I ran into a case where I need a small web-service to be running where an embedded database makes a lot of sense. I’m using the HttpListener class to run my own Http.sys server without using IIS. This still counts as ASP.NET to the SQLCE code though.

Force SQLCE to run under ASP.NET

Steve Lasker posted blog entry on how to use SQLCE under ASP.NET using the pre-release version of the SQLEV. Under SQLEV you set a flag that would tell the SQL Connection, “yes I know this isn’t supported, but let me do it anyway”:

AppDomain.CurrentDomain.SetData("SQLServerEverywhereUnderWebHosting", true)

As you can see the name of the product is right there in the key. Well they changed the name of the product and so they changed the name of the key. So, if you were using the beta for development and are now switching over to the release version of SQLCE , you will need to change the key:

AppDomain.CurrentDomain.SetData("SQLServerCompactEditionUnderWebHosting", true)

That should allow you to use the database under ASP.NET. Now you can revel in the fact that you are swimming in unsupported waters!

Special Thanks
I found this using the great .NET disassembler Lutz Roeder’s .NET Reflector. You should check it out. It can be a great way to track down details of an API implementation when the abstraction is a bit leaky.