Taking meta-programming beyond crazy

Firstly, a brief note that protobuf-net is up to r580 now, on both nuget and google-code; mainly small tweaks while I build up enough energy to tackle a few larger pieces (some knotty interface / dynamic / base-class improvements are next on the list)

Pushing both ends

Aimed to time with the .NET 4.5 release (and perhaps more notably, the .NETCore and .NETPortable profiles), I’ve recently spent a lot of time on meta-programming, culminating in the new precompiler that allows these slimmed down and highly restrictive frameworks to still have fast serialization (static IL, etc).

While there, I set myself a silly stretch goal; for the main purpose simply of to see if I could do it – which was: to get the whole shabang working on .NET 1.1 too. This would give me a fair claim to supporting the entire .NET framework. So, for a bit of reminiscing – what does that need?

Generics

Generics were introduced in .NET 2. v1 of protobuf-net made massive usage of generics; so much so that it actually killed the runtime on some platforms (see here, here and here). So removing most of the generics was already a primary design goal in v2.

Perhaps the most significant problem I hit here was trying to decide on a core collection type for the internal state. As it turns out, there’s no free lunch here; there is no collection type that is common to all frameworks – some don’t have ArrayList. In the end, I wrote my own simple collection – not just for this, but also because I wanted a collection that was thread-safe for iterations competing with appends (iterators only read what existed when it started iterators).

Language

You’d be amazed what you miss when you try to design something to compile on down-level compilers. For a dare, go into project-properties, the “Build” tab, and click “Advanced…” – and change the “Language Version” to something like ISO-1 (C# 1.2) or ISO-2 (C# 2.0) – see what breaks. Obviously you expect generics to disappear, but you also lose partial methods, partial classes, iterator blocks, lambdas, extension methods, null-coalescing, static classes, etc – and just some technically legal syntax that the early compilers simply struggle with. protobuf-net is configured to build in ISO-2 in the IDE, but with #if-regions to rip out the last few generics in the .NET 1.1 build. Writing iterator blocks… not fun.

Framework

There’s a silly number of variances in the core BCL between different frameworks; even things like string.IsNullOrEmpty or StringBuilder.AppendLine() aren’t all-encompassing. I ended up with a utility class with a decent number of methods to hide the differences (behind yet more #if-regions). But by far the craziest problem: reflection. And protobuf-net, at least in “Full” mode (see here for an overview of “CoreOnly” vs “Full”), uses plenty of reflection. Oddly enough, the reflection in .NET 1.1 isn’t bad – sure, it would be nice to have DynamicMethod, but I can live without it. Getting this working on .NET 1.1 was painless compared to .NETCore.

Aside / rant: how much do I hate “.GetTypeInfo()” on .NETCore? With the fiery rage of 2 stars slowly crashing into each-other. Oh, I’m sure that the differences to Type / TypeInfo make perfect sense for application-developers in .NETCore, who probably should be limiting their use of reflection, but for library authors: this change really, really hurts. The one things that lets me keep civil about this change is that in “CoreOnly” + “precompiler” we do all the reflection work up-front using the regular reflection API, so for me at least most of this ugly is just a cruel artefact. But still: grrrrrrrrrrrrrr.

Opcodes

There are a number of opcodes that simply don’t exist back on 1.1; if I’ve done my compare correctly, this is: Unbox_Any, Readonly, Constrained, Ldelem and Stelem. The good news is that most of these exist only to support generics, and are pretty easy to substitute if you know that you aren’t dealing with generics.

Metadata Version

.NET 1.1 uses an earlier version of the metadata packaging format than all the others use. This is yet another thing that the inbuilt Reflection.Emit can’t help, but - my new favorite metaprogramming tool to the rescue: IKVM.Reflection supports this. I have to offer yet another thanks to Jeroen Frijters who showed me the correct incantations to make things happy: beyond the basics of IKVM, the key part here is a voodoo call to IKVM’s implementation of AssemblyBuilder:

asm.__SetImageRuntimeVersion("v1.1.4322", 0x10000);

The 0x10000 here is a magic number that specifies the .NET 1.1 metadata format. For reference, 0x20000 is the version you want the rest of the time. As always, IKVM.Reflection seems to have considered everything; it is the gold standard of assembly writing tools. Awesome job, Jeroen. I jokingly half-expect to find that Roslyn has a a reference to IKVM.Reflection ;p

Putting the pieces together

But! Once you’ve dealt with all those trivial problems; it works. I’m happy to say that protobuf-net now has “CoreOnly” and “Full” builds, and support from “precompiler”. So if you still have .NET 1.1 applications (and I promise not to judge you… much), you can now use protobuf-net with as many optimizations as it is capable of. Which is cute:

C:\SomePath>AnotherPath\precompile.exe Net11_Poco.dll -o:MyCrazy.dll -t:MySerializer

protobuf-net pre-compiler
Detected framework: C:\Windows\Microsoft.NET\Framework\v1.1.4322
Resolved C:\Windows\Microsoft.NET\Framework\v1.1.4322\mscorlib.dll
Resolved C:\Windows\Microsoft.NET\Framework\v1.1.4322\System.dll
Resolved protobuf-net.dll
Adding DAL.DatabaseCompat...
Resolved C:\Windows\Microsoft.NET\Framework\v1.1.4322\System.Xml.dll
Adding DAL.DatabaseCompatRem...
Adding DAL.OrderCompat...
Adding DAL.OrderLineCompat...
Adding DAL.VariousFieldTypes...
Compiling MySerializer to MyCrazy.dll...
All done

C:\SomePath>peverify MyCrazy.dll

Microsoft (R) .NET Framework PE Verifier  Version 1.1.4322.573
Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.

All Classes and Methods in MyCrazy.dll Verified

In case it isn’t obvious, “Net11_Poco.dll” is a .NET 1.1 dll created in Visual Studio 2003; “precompiler” has then detected the 1.1-ness, bound IKVM to the .NET 1.1 framework, and compiled a protobuf-net custom serializer for that model, as a legal .NET 1.1 dll.

Questionable sanity

Another way of reading all this is: I’ve possibly now crossed the line between “eccentric” and “batshit crazy”. I don’t have a need to use .NET 1.1, but I would be overjoyed if someone else gets some genuine usage out of this. Mainly, I just wanted to learn some things, challenge myself, and take a bit of professional pride in doing something fully and properly – just because: I can.