Philip Mateescu

django-permissivecsrf

2013-01-14T00:00:00-06:00

I bet you're here because you've run into the puzzling Django error

CSRF verification failed. Request aborted.

Reason given for failure:
    Referer checking failed - http://<domain>/ does not match https://<domain>/.

I'll double down that this happens when you try to make your login form post to HTTPS, perhaps to https://<domain>/accounts/login/.

If I'm mostly right, you've run into the problem I've solved. If I'm not right and you somehow ended up on this page, I'd love to hear about it.

Short version

In the CSRF middleware, Django does an extra check when a request comes over HTTPS to ensure it comes from the same site (same origin check).

If any portion of your site benefits from HTTPS, you probably should run your entire site over HTTPS. You can use django-sslify to force your website to operate in HTTPS mode all the time.

Furthermore, if you want to disable CSRF checking for your own views, there are other methods you can use, for example the @csrf_exempt decorator.

However, if the views are not under your control and you are comfortable n trading some security for not really that much convenience, you can use django-permissivecsrf to work around this error.

Use the instructions in the README file on GitHub or on PyPI to get this project up an running. Mostly it consists of installing django-permissivecsrf and adding 'permissivecsrf.middleware.PermissiveCSRFMiddleware' to your MIDDLEWARE_CLASSES entry.

Enjoy.

Long version

The gist of why this happens is explained in point #4 of the How it works section of the Django documentation on Cross Site Request Forgery (emphasis mine):

4) In addition, for HTTPS requests, strict referer checking is done by CsrfViewMiddleware. This is necessary to address a Man-In-The-Middle attack that is possible under HTTPS when using a session independent nonce, due to the fact that HTTP 'Set-Cookie' headers are (unfortunately) accepted by clients that are talking to a site under HTTPS. (Referer checking is not done for HTTP requests because the presence of the Referer header is not reliable enough under HTTP.)

In other words, because the HTTPS headers are encrypted, the HTTP-Referer header is resilient against MITM attacks, so it can be safely used to check and make sure the CSRF cookie is originated by the same site that served the page and that the referring page has also been served over HTTPS, which means that page has also been protected against header injections.

The same check could be made on HTTP calls as well, but since HTTP headers are not encrypted, they could be easily faked and thus the check would be a useless placebo.

This explanation is also present, in comment form, in this f92a21daa7 commit by spookylukey aka Luke Plant, and further detailed by him in a reply to a complaint about the strictness of CSRF Referer check on the django-developers maillist.

How django-permissivecsrf works

The Django CSRF middleware performs an extra-check if the request is over HTTPS to ensure that the request came from the same site, i.e. that the referrer (HTTP-Referer header) matches the current site, and that the schema of the referrer is also HTTPS.

In other words, in ensures that the call to https://example.com/account/login came from another page of https://example.com/. As such, if you put your login form on your non-secure homepage, http://example.com/, but use a secure target for your form's action attribute, <form action="https://example.com/account/login" method="POST">, Django's check will fail because::

'http://example.com/' != ('https://%s/' % request.get_host())

However, Django will not perform the CSRF check at all if the request object has an attribute _dont_enforce_csrf_checks set to True. That's what PermissiveCSRF relies on: if the request came from the same site, regardless the schema, it sets _dont_enforce_csrf_checks to True, thus telling the Django CSRF middleware to skip the CSRF check for that request.

This only happens if:

DEBUG == True. Your production server should always be HTTPS;
The HTTP-Referer header is present;
The request is for an HTTPS URL (i.e. request.is_secure() == True);
and the referrer uses HTTP.

In all other cases it defers to Django for normal processing.

Bottom line

There's only one thing to take away from all this: in production use HTTPS (see django-sslify). Period.

Miniature Minifier Makefile

2012-09-30T00:00:00-05:00

A solution that uses only make and curl:

# c/o: http://wonko.com/post/simple-makefile-to-minify-css-and-js
# Patterns matching CSS files that should be minified. Files with a -min.css
# suffix will be ignored.
CSS_FILES = $(filter-out %.min.css,$(wildcard \
    css/*.css \
    css/**/*.css \
))

# Patterns matching JS files that should be minified. Files with a -min.js
# suffix will be ignored.
JS_FILES = $(filter-out %.min.js,$(wildcard \
    js/*.js \
    js/**/*.js \
))

# Commands
CSS_MINIFIER = curl -X POST -s \
    --data-urlencode "input@CSS_TMP" \
    http://www.cssminifier.com/raw

JS_MINIFIER = curl -s -X POST \
    --data-urlencode "js_code@JS_TMP" \
    http://marijnhaverbeke.nl//uglifyjs 

CSS_MINIFIED = $(CSS_FILES:.css=.min.css)
JS_MINIFIED = $(JS_FILES:.js=.min.js)

# target: minify - Minifies CSS and JS.
minify: minify-css minify-js

# target: minify-css - Minifies CSS.
minify-css: $(CSS_FILES) $(CSS_MINIFIED)

# target: minify-js - Minifies JS.
minify-js: $(JS_FILES) $(JS_MINIFIED)

%.min.css: %.css
    @echo '  Minifying $< ==> $@'
    $(subst CSS_TMP,$(<),$(CSS_MINIFIER)) > $@
    @echo

%.min.js: %.js
    @echo '  Minifying $< ==> $@'
    $(subst JS_TMP,$(<),$(JS_MINIFIER)) > $@
    @echo

# target: clean - Removes minified CSS and JS files.
clean:
    rm -f $(CSS_MINIFIED) $(JS_MINIFIED)


# target: help - Displays help.
help:
    @egrep "^# target:" Makefile

You can also find the latest version on GitHub part of my LeaseMilesTracker project.

How Does It Work?

Surprisingly simple.

We make a list of all CSS files, filtering out the *.min.css files;
we ask cssminifier.com to make us a minified version;
then we save the latter to a file ending in min.css.

Same with the JavaScript files, but using the service provided by UglifyJS

The obvious targets are minify-css, minify-js, and minify which is dependent on both. The less obvious targets are %.min.css and %.min.js whose purpose is to cause the re-minification of the CSS or JS files, if the normal files are newer than the minified version.

Finally, clean will remove all the minified files, that is all *.min.css and *.min.js files.

Why Would I Use This?

Ryan Grove's "Simple makefile to minify CSS and JS", the original inspiration for my Makefile, is an excellent solution if you're using the YUI compressor, but YUIc requires Java (and so does Google's closure-compiler) and I was interesting in having this not only work without Java, but more important work on a system that would require no special tools beyond what you'd find standard on most *nix distros.
After all this is 2012 and we're supposed to be using RESTful APIs and what not.

Oh, and also Mountain Lion doesn't come with Java installed.

Kendo Grid Select Editor

2012-09-11T00:00:00-05:00

I'm going to assume you're reading this post because you'd like to find out how to use a regular HTML SELECT as an editor in a Kendo grid, and that you already know why you want to do that - you just had a hard time figuring out how to.

If you just want to see the code and don't care about the explanation, here is a jsFiddle to play with, also a gist for your forking pleasure.

If you wonder why you'd want to do it, read the Why would I want to do that? section that come back for the how.

I see two primary scenarios for using a SELECT aka drop-down list.

Simple Field - Extra Info

This would be the case where, for example, your model contains a field to store the user login (e.g. foo), but when displaying the drop-down to select the user you'd like to also list their name (e.g. foo - Foo Frye).

Let us assume you have the mapping of login to full name in an array of objects, and for genericality's sake let's assume it has a format similar to the following:

var nameList = [
  { Login: 'foo', Description: 'foo - Foo Frye' },
  ...
]

When you create your grid and specify the columns, you have the chance to pass in a function that would create a custom editor instead of the text box the grid gives you by default:

$('#grid').kendoGrid({
  ...
  dataSource: {
    data: [
      { ..., Login: 'foo', ... }
    ]
  }
  columns: [
    ...
    {
      field: 'Login',
      editor: function(container, options) {
        var s = $('<select ' +
                  ' data-bind="source: listSource.list, ' +
                    'value: ' + options.field + '" ' + 
                  ' data-text-field="Description"' + 
                  ' data-value-field="Login"' + 
                  '/>')
        options.model.listSource = kendo.observable({list: nameList})
        s.appendTo(container)
      }
    }
    ...

What's going on here?

We create a drop-down $('<select/>')
Using the data-bind attribute we tell Kendo to use a specific source for the drop-down and to write the selected value to the underlying model's field (here options.field == 'Login').
We tell it to use the Description property of each element in the source for the text portion, and the Login field for the value. This would be as if we had an <option value="foo">foo - Foo Frye</option>.
We create a kendo.observable containing the list of names.
We append the HTML select to the container, which, if you're curious, is the actual grid cell (TD).

There are only two tricky parts here and they both gravitate towards the use of the source directive: options.model is the object from your data source that correspond to the grid row; the source: listSource.list will look for a property called listSource on the grid row/object/model and expect it to
a) be a kendo.observable and
b) contain a sub-object named list.

Complex Field

What if your Login field was a complex object, maybe similar to one of the entries in nameList? If this was the case, how would you go about displaying it in a SELECT, but have it write back to your field the same complex object, not just a simple string.

The code looks similar, in a way even simpler:

$('#grid').kendoGrid({
  ...
  dataSource: {
    data: [
      { Complex: { Login: 'foo', Description: 'foo - Foo Frye' } },
    ]
  }
  columns: [
    ...
    {
      field: 'Complex',
      editor: function(container, options) {
        var s = $('<select ' +
                  ' data-bind="source: listSource.list, ' +
                    'value: ' + options.field + '" ' + 
                  ' data-text-field="Description"' + 
                  '/>')
        options.model.listSource = kendo.observable({list: nameList})
        s.appendTo(container)
      }
    }
    ...

The most significant difference from the simple example is the absence of the data-text-value attribute. This causes the entire underlying record to be written back into the Complex model field.

That's it. If there was a simpler way to do this, I couldn't find it.

Why Would I Want to Do This?

If you use Telerik's excellent Kendo stack, you already have access to a nice drop-down editor that is part of the suite. The Kendo UI demos even show you how to use the Kendo DropDownList as a custom editor, and use it you should for it has a plethora of features.

For all its niceness, the DropDownList is built upon styled UL/LI elements, and that means it is missing a few features that a normal drop-down would have, chiefly the ability to use keyboard navigation and the two significant advantages that come with it: using keys to trigger and navigate the list (Alt-Down on Windows and Spacebar on Mac), and type-to-select when focused (for example typing Tex or TT to select Texas in a list of states).

That and the fact that it doesn't play 100% nice with Twitter Bootstrap, in the sense that the styles don't quite match. Oh, and on a mobile browser you don't get the native picker.

Fortunately, the Kendo grid allows you to specify a custom editor for a column; unfortunately, the Kendo stack tends to make use of and wrap your SELECT in a kendo.observable type of object. I suspect that this is the main reason why examples of using normal HTML inputs with the Kendo grid tend to be scarce on the web.

Templates for GoodNotes

2012-04-28T00:00:00-05:00

Recently I've been testing a few apps for taking notes and I have settled on GoodNotes.
It's a really good looking app and the developers are dedicated to it. How dedicated? They had a retina version out within days after the iPad 3 release.

After a few weeks of giving up paper and exclusively using the app, I've created a few templates for the type of notes I find myself taking most frequently.

While these templates have been build with GoodNotes in mind, you can take the SVG files and with very few tweaks target them towards other note taking apps.

Read on for an explanation of the design and thoughts that went into each template, how to import them into GoodNotes, and how you can help improve them, or jump to the end to see where you can get them.

The Templates

The template are geared towards writing and designing rather than viewing (read-only) so they account for the space occupied by UI elements. They also favor unobtrusive guiding elements so that they provide support for writing but not distract from what is being written or drawn.

960 Grid System

Two templates support the 960 Grid system (also employed by my favorite: Twitter Bootstrap) in 12 column format:

960px Portrait - has longer columns for designing in portrait mode;
960px Landscape - has wider columns (33% wider), better suited for drawing in landscape.

These two has been inspired by hellopanos's templates for Penultimate, which I found too dark and thus distracting. Since the iPad has a great screen a lighter color is very visible and let's the design take center stage while still providing alignment guidance. Space between columns is proportional to the width of the columns.

Since the design is performed within the constraint of the columns, I have chosen to indicate the outer/side border of the 960px grid with two faint lines. This way they are not distracting yet provide the visual indication of what the boundaries are.

iPad

Three iPad templates present an iPad that is scaled about 73% from the actual size (in pixels). FWIW, I even got the corner radius accurate and then I realized it doesn't really matter that much.

The screen portion displays the status bar (20px scaled by 0.73) and if you squint hard enough you'll see marker lines for the 44pt navigation and 49pt tab bars. They are very faint so that they don't distract from designing your interface, but still present if you want to make use of them.

Meeting Notes

Inspired by a Penultimate template, it contains a top section to record the date/time and place of the meeting, the subject/topic/reason for meeting, and the participants. This section doesn't offer a great deal of space, but if you need more than that you're probably doing something wrong or at least ineffectual.

The template devotes most of its body to the meeting notes and is concluded by a checklist-like section to record action items.

I gave each action item what I thought to be enough space, but you might want to consider writing in those sections using the GoodNotes magnifier mode.

Each section contains almost transparent text to indicate its purpose. I've thought it would be nice to provide such a hint, but I made it light enough so that it allows to write on top of it.

Task List

The task list template is geared toward being able to capture tasks quickly. That means that the space for each of the 14 lines is oversized so that you don't have to use the magnifier mode.

The lines are a bit darker than the other templates as they intend to provide more guidance for writing. There is white space at the top of the templates, enough to write a title for the task list, but not so much that it's distracting it if you don't.

If you look at the image by itself you might think there's too much space at the bottom, but remember that GoodNotes has an area at the bottom where it displays the toolbar for bringing up the palm rest or magnifier. The template accounts for that.

ToDo List

The to-do template also has 14 rows of checkboxes but in two-column format as it's designed for shorter entries. You can use the magnifier mode if you need to capture more details.

Since to-do lists are more likely to have a title, the template provides more emphasis by separating the title area from the checklist through a darker line, while at the same time the individual to-do entries have lighter supporting lines to allow more focus on the written text.

How to Import the Templates

Using the images in GoodNotes is simple. Get the templates onto the device, then from within a notebook click the + button in the toolbar, click Import, select your image and go to town on it.

However, if you find yourself using one or more of the templates with some frequency, it might be worth going through the following five-step process to make them part of the template gallery.

Click on the + button in the toolbar and choose Import
Select your image from one of the image sources, depending on how you made available the templates to GoodNotes (in this case, I copied them to the app using iTunes).
Repeat step 1: touch the + in the toolbar, but select Other Templates this time. You will be presented with the template selection dialog.
Click the + button in the top left corner of this dialog, then select Current Template.
Give your template a recognizable name. Optionally, touch Edit then use the sizing chevron to move your template(s) in a better position.

Tech Notes

One of the decisions I made early on was to optimize these templates towards input (writing/drawing), which meant having to account for GoodNotes's toolbar and palm-rest/magnifier. I could've started with a 768 x 1024 image to mimic the iPad's screen, but I've noticed that GoodNotes moves the template down so that its top is right below the toolbar, effectively clipping its bottom part and thus reducing the effective area.

For this reason, the templates are actually 768 x 972, given that GoodNotes's toolbar is 52 pixels high. The chevron that triggers the palm-rest/magnifier is 35 pixels high, so I designed all templates so that their bottom 40 pixels remains clear. I could've cropped the image and target 768 x 937, but I thought it looked better if the "paper" continued all the way to the bottom of the screen.

To create each template, I have started with a handcrafted SVG file and tweaked it for hours (mostly because I had no familiarity with SVG) until I got the desired result.

I then exported the images into PNG using OS X's qlmanage and ImageMagick's convert and loaded them into GoodNotes to check out the result and make sure the templates were comfortable to use.

Converting to PNG could've (should've) been a simple process because ImageMagick can convert SVG images, alas the feature-set support is limited, and occasionally, incorrectly implemented.

It would've been a great deal easier to create them in an image editor and, indeed, the first attempt was in Pixelmator and it took about 1/3 to half of the time it took to create the SVGs.

However, since the SVG files are text they are effectively the source code for the templates, something I can certainly appreciate as a software engineer.

As a bonus, you need but a text editor to quickly change their dimensions and create templates for another note taking app (e.g. Penultimate).

Where to Get Them

The templates are on github and the SVG files are within the source folder of that repository. If you're interested only in the PNG files, I've uploaded a zip file that contains them all.

The templates, both PNG and SVG files, are released under a Creative Commons BY-NC-SA license, which means you are free to share them and change (remix) them, but you cannot sell them, you must give proper attribution, and should you alter them you may distribute the resulting work only under the same license (or similar).

That being said, have fun hacking them. Looking forward to it.

Asymmetric Sparkbars

2012-03-22T00:00:00-05:00

I am creating some sparkbars, sparkline's poorer cousin, for a project I'll talk about soon, and I wanted to represent in a single chart both additions and subtractions.

As it happens, there's an entire range of Unicode characters devoted to blocks: U+2580 to U+259F, of which U+2581, "Lower one eighth block", through U+2588, "Full block", seem perfect for rendering text sparklines.

The smart way to handle this task would've been to use jQuery.sparkline, which makes use of the canvas element to draw some pretty impressive charts, but I stubbornly decided to stick with text.

I am starting with two data sets, one representing tasks completed per day - progress, the other being tasks created per day, for the same interval, and I will use a spark() function provided by node-textspark to convert each set into discrete values, returned to me as Unicode characters.

My starting point is these two sparklines:

I'd love for the top of the second set to be glued to the bottom of the first set and thus give a good visual of what happened for each day. For that to happen I'll be using a different color and I will make use of negative space to represent the data points.

First I'll set the background color of the second series to red and the foreground color to white, effectively transforming the negative space, previously white - now red, into sparkbars that seem to hang from the top.

Unfortunately, the block characters stop at the baseline and thus the space between the baseline and the beard line (not making this up) is visible and colored red (from the background).

To make that line go away, I'll set the height on the div containing the second series to the same height as the font, then set the overflow property to hidden so that it clips the bottom part, below the baseline.

Lastly, I'll set a negative top margin for the second div so that it gets closer to top series:

I chose to leave a pixel between the series as I think it looks just a tad better than if they had no space between them:

The ending HTML is deceptively simple and scales well when the browser zooms in.

<!DOCTYPE html>
<html>
    <head>
        <style type="text/css">
            .inverse .inner { background-color: red; color: white; }
            .sparkline-group { font: 'Courier New'; font-size: 18px; }
            .sparkline-group .sparkline.inverse {
                margin-top: -7px;
                height: 18px;
                overflow: hidden;
            }
        </style>
    </head>
    <body>
        <div class="sparkline-group">
            <div class="sparkline"><span id="spark1" class="inner"></span></div>
            <div class="sparkline inverse"><span id="spark2" class="inner"></span></div>
        </div> 
        
        <script src="spark.js" type="text/javascript"></script>
        <script type="text/javascript">
            var spark1 = [1,2,5,9,0, 5,4,8,9,1, 3,0,6,4,9, 1,2,3,9,5], 
                spark2 = [9,9,8,7,5, 4,8,9,6,6, 4,9,4,7,9, 1,6,7,4,9];
            document.getElementById('spark1').innerHTML = spark(spark1);
            document.getElementById('spark2').innerText = spark(spark2);
        </script>
    </body>
</html>

That's it.

Json.NET and NHibernate Serialization Blues

2012-02-27T00:00:00-06:00

After spending some irritating hours with Microsoft's JavaScriptSerializer, I've switched over to Json.NET and I've been happier. That matters. A lot.

I have better control over serialization, for example I can specify how to handle circular references (ReferenceLoopHandling), or whether to include nulls (NullValueHandling) and/or default values (DefaultValueHandling); it also gives me more injection points.

Yet it exhibits one frustrating behavior when one of the objects is masked by an NHibernate proxy, as it starts to go off the rails and soon the JSON code is riddled with __interceptors, and EventListeners, and various other critters that have no purpose being sent across the wire.

(Yes, arguably I shouldn't be directly serializing these NHibernate objects, but the DTOs I would use would look remarkably similar to the NH objects they would attempt to mock.)

The problem comes with the chaining of strategies a JsonSerializer uses to determine how to serialize a particular strategies. This chain is captured by the DefaultContractResolver in the CreateContract(Type) method (which thankfully and wisely is also virtual): it tries in succession to figure out if the type is a primitive, if it's decorated with Json*Attributes, if it's a dictionary, etc, and eventually it just gives up and creates a plain JsonObjectContract.

However, in the middle of that chain is a test for whether the type is ISerializable, and unfortunately the NHibernate proxies happen to have that attribute, and that is what causes the serializer to go down a rabbit hole.

My solution is to force it to use a JsonObjectContract when it encounters one of these proxies.

public class IgnoreSerializableJsonContractResolver : DefaultContractResolver
{
    protected override JsonContract CreateContract(System.Type objectType)
    {
        /* Behavior in base we're overriding:
        if (typeof(ISerializable).IsAssignableFrom(objectType))
            return CreateISerializableContract(objectType);
        //*/

        if (objectType.IsAutoClass 
              && objectType.Namespace == null 
              && typeof(ISerializable).IsAssignableFrom(objectType)) {

            return base.CreateObjectContract(objectType);
        }

        return base.CreateContract(objectType);
    }
}


var serializer = new JsonSerializer();
serializer.ContractResolver(new IgnoreSerializableJsonContractResolver());

I am not sure whether the condition is too restrictive or may generate false positives, but all the NH proxies I've encountered seem to exhibit the same behavior: they are auto-generated classes, without a namespace.

JSON Entity Converter

2012-02-20T00:00:00-06:00

Whether in the context of my friend Curtis's BORAX proposal, or my own, we will eventually need to serialize our domain objects and ship them to the browser.

Our best solutions is probably some sort of Domain Transfer Object (DTO), but for a second let's assume we take the quick route and turn our domain objects directly into JSON mush.

Is ASP.NET MVC 3 we can use the JsonResult class, but with it comes a problem: the JavaScriptSerializer, which JsonResult uses, cannot handle circular references (and rightfully so since JSON doesn't have references).

Let me illustrate using the following simplified domain classes (you can find the full code in this Gist):

public class Entity {  // Layer Supertype
    public int Id { get; set; }
}

public class Company : Entity { 
    public string Name { get; set; }
    public Contact MainContact { get; set; }
}

public class Contact : Entity {
    public string Name { get; set; }
    public Company WorksFor { get; set; }
}

When serializing an object of type Company, the JavaScriptSerializer will choke on Contact.WorksFor since it points back to the Company that the contact is associated with.

Before we talk about how we'll fix it, let's talk about the output. What I think I'd like to have when serialize a company to JSON is:

{
    "Id" : 1,
    "Name" : "Moof",
    "MainContact" : {
        "Id" : 100,
        "Name" : "Clarus",
        "WorksFor" : {
            "TypeName" : "Company",
            "Id" : 1
        }
    }
}

In other words, if one of the object we try to serialize has a property of type Entity, we'll just output its Id and assume the consumer of the JSON will know to request /refdata/{entitytypes}/{id} to resolve it, if needed.

Of course, this will not avoid circular references entirely, but merely provide an example on how to work with the class that will solve our particular problem: JavaScriptConverter.

JavaScriptConverter, of the System.Web.Script.Serialization clan, allows us to inject specific type handlers into JavaScriptSerializer and have it defer to our converter when encountering those types in the serialization process. And it gets better: our converter won't have to provide actual JSON, it merely has to return a dictionary of name-value pairs and the JavaScriptSerializer will take care of the actual transformation.

Here's how we could implement the required Serialize method:

public override IDictionary<string, object> Serialize(object obj, JavaScriptSerializer serializer) {
    var ret = new Dictionary<string, object>();
    string name; object value;
    
    var props = obj.GetType().GetProperties(BindingFlags.Instance | BindingFlags.Public | BindingFlags.GetProperty);
    foreach (PropertyInfo prop in props) {
        name = prop.Name;
        value = prop.GetValue(obj, null);
        if (value != null && typeof(Entity).IsAssignableFrom(prop.PropertyType))
            // real gist code: ret.Add(name, _reducer((Entity) value)); //
            ret.Add(name, new Dictionary<string, object> { 
                { "TypeName", value.GetType().name },
                { "Id", ((Entity) value).Id }
            });
        else
            ret.Add(name, value);
    }
    
    return ret;
}

(Of course the actual implementation is a bit more elegant, I hope)

We can now serialize objects such as these:

var co = new Company { Id = 1, Name = "Moof Inc." };
var clarus = new Contact { Id = 1, Name = "Clarus" };
var mark = new Contact { Id = 2, Name = "Mark" };
co.AddContact(clarus);
co.AddContact(mark);
co.MainContact = clarus;

var converter = new EntityConverter(new[] { typeof(Contact) });
var serializer = new JavaScriptSerializer();
serializer.RegisterConverters(new[] { converter });
serializer.Serialize(co).Dump();

And get the following JSON representation:

{
    "Id": 1, 
    "Name": "Moof Inc.",
    "Contacts": [
        {
            "Id": 1, 
            "Name": "Clarus", 
            "WorksFor": {
                "Id": 1, 
                "TypeName": "Company"
            }
        }, 
        {
            "Id": 2, 
            "Name": "Mark", 
            "WorksFor": {
                "Id": 1, 
                "TypeName": "Company"
            }
        }
    ], 
    "MainContact": {
        "Id": 1, 
        "Name": "Clarus", 
        "WorksFor": {
            "Id": 1, 
            "TypeName": "Company"
        }
    } 
}

The full code - designed to work with the most excellect LINQPad, shows three attempts (four if we include the failed circular reference error) and with them the various ways the EntityConverter can control the JSON output.

If you want to use the code in Visual Studio, add a reference to System.Web.Extensions and import the System.Web.Script.Serialization namespace.

Setting up FxCop under CruiseControl.NET

2012-02-14T00:00:00-06:00

After spending up half a day trying to get FxCop v1.36 to play nice with CruiseControl.NET and MSBuild, I thought it was worth spending an extra hour or two documenting it.

There are two ways we can get FxCop running under CCNet: the easy way and (what I think it's) the right way.

Either way, we start by creating an FxCop project using the FxCop GUI; it makes it easy to add our assemblies and to select the rules we'd like enforced.

The easy way has us using an <Exec/> task within the MSBuild file:

<Target Name="FxCop" DependsOnTargets="Build;RunFastTests">
  <Delete Files="$(ProjectRoot)\fxcop.xml" />
  <Exec Command="&quot;$(FxCopPath)\FxCopCmd.exe&quot;
    /dic:&quot;$(ProjectRoot)\src\CodeNameDictionary.xml&quot;
    /project:$(ProjectRoot)\src\Default.FxCop /out:$(ProjectRoot)\fxcop.xml" 
    ContinueOnError="true">
  </Exec>
</Target>

In this version, we're telling FxCop command line tool to use a custom dictionary - more on this later, the project file we created earlier, and to output the results to an XML file.

When being run, FxCop tends to load the previous results, which would cause load errors if we renamed classes that previously had errors, so we'll <Delete/> the output file before running the task.

The nicer, more explicit way, has us use the supplemental tasks provided by the MSBuild Community Tasks Project, in particular the <FxCop/> task (grab at least version 1.3.0.528, part of the nightly builds - v1.2 doesn't properly support this task).

<Target Name="FxCop" DependsOnTargets="Build;RunFastTests">
  <Delete Files="$(ProjectRoot)\fxcop.xml" />
  <FxCop
    ToolPath="$(FxCopPath)\FxCopCmd.exe"
    ProjectFile="$(ProjectRoot)\src\Default.FxCop"
    CustomDictionary="$(ProjectRoot)\src\CodeNameDictionary.xml"
    FailOnError="false"
    AnalysisReportFileName="$(ProjectRoot)\fxcop.xml"
  />
</Target>

What you won't find in the MSBuild Community Tasks documentation is the requirement to use the ToolPath to point to the right version of FxCop. As of now, the <FxCop/> task looks explicitly for FxCop v1.32 and for a registry key that no longer exists when it tries to build the full command line:

...
if (string.IsNullOrEmpty(ToolPath)) {
  string fxCopPath = Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles);
  fxCopPath = Path.Combine(fxCopPath, @"Microsoft FxCop 1.32");
  try {
    using (RegistryKey buildKey = Registry.ClassesRoot.OpenSubKey(@"FxCopProject\shell\Open\command")) {
      if (buildKey == null) {
        Log.LogError("Could not find the FxCopProject File command in the registry. Please make sure FxCop is installed.");
      } else {
        fxCopPath = buildKey.GetValue(null, fxCopPath).ToString();
        ...

The problem with the registry look up is that FxCop v1.36 uses a key called FxCop.Project.9.0, all which means that the <FxCop/> task fails to find the executable. That is why we have to use the ToolPath, inherited from its Task parent, to help it find it.

Alright, now the promised dictionary clarification.

If our namespaces, classes, etc. include non-lexical words, for example Flickr.Configurator, FxCop will announce
that our code breaks a naming rule - CA1704:IdentifiersShouldBeSpelledCorrectly - and will tell us to:

Correct the spelling of 'Flickr' in assembly ExternalDependencies.dll'.

Correct the spelling of 'Configurator' in assembly ExternalDependencies.dll'.

(Yes, Configurator is not an English word either - us developers have a lot of those.)

We'll fix this problem by adding a custom FxCop dictionary containing the words we want to mark as correct:

<Dictionary>
  <Words>
    <Recognized>
      <Word>Flickr</Word>
      <Word>Configurator</Word>
    </Recognized>
  </Words>
</Dictionary>

We'll then spell this dictionary using the CustomDictionary property of the <FxCop/> task.

We could and should also add the dictionary into the FxCop project file to make sure the GUI doesn't complain about those naming rules being broken either. Find the <CustomDictionaries/> node within the project file, Default.FxCop in the examples above, and add a <CustomDictionary/> node:

<CustomDictionaries SearchFxCopDir="True" SearchUserProfile="True" SearchProjectDir="True">
  <!-- Tells FxCop to find it in the same location as the Default.FxCop file -->
  <CustomDictionary Path="CodeAnalysisDictionary.xml" />
</CustomDictionaries>

CCNet captures all the output of a build into a giant result file using <merge> tasks. We'll follow the instructions on the CCNet website to merge in the generated fxcop.xml file:

<tasks>
  <merge>
    <files>
      <file>fxcop.xml</file>
      <!-- Other files to merge for your build would also be included here -->
    </files>
  </merge>
</tasks>

Finally, to be able view the results of the FxCop analysis in CCNet's web dashboard, we'll need to include the XSLT files, which extract and pretty up the entries merged in from fxcop.xml, into the dashboard.config file, typically located within C:\Program Files\CruiseControl.NET\webdashboard\ folder:

...
<buildPlugins>
  <buildReportBuildPlugin>
    <xslFileNames>
      <!-- other xsl file, such nunit and msbuild -->
      <xslFile>xsl\fxcop-summary_1_36.xsl</xslFile>
      <xslFile>xsl\fxcop-report_1_36.xsl</xslFile>
    </xslFileNames>
  </buildReportBuildPlugin>
  <buildLogBuildPlugin />
  <!-- These create two menu entries on the left, when viewing a project's build page -->
  <xslReportBuildPlugin description="FxCop Summary" actionName="FxCopSummary" xslFileName="xsl\fxcop-summary_1_36.xsl"></xslReportBuildPlugin>
  <xslReportBuildPlugin description="FxCop Report" actionName="FxCopReport" xslFileName="xsl\fxcop-report_1_36.xsl"></xslReportBuildPlugin>
</buildPlugins>

That's it. Next time you build you should see the FxCop details when you click the "FxCop Report" link.

The only one thing to note is that violating FxCop rules, no matter their level or the value of ContinueOnError, will not mark the build a broken. If you want FxCop failures to break the build, there is one further step you need to take.

New (Old) Web App Architecture

2012-01-21T00:00:00-06:00

We've been doing this for a while, more or less, in various shapes and forms, so it's nothing new. We've been doing it as mash-ups and we've been doing it with AJAX and we've done it with web services, but we've only done mash-ups when we couldn't control the source of data, and we've done AJAX only to speed things up, and web services - well, they're traditionally XML based, and by now we've mostly agreed that is somewhat ugly.

When we could control the source of data, we've reverted back to some server-side framework producing the UI code, and we worked hard to make that combination good, just like we worked hard before to make integrating web services easy.

Why a New Way?

What we have right now - with Rails, and Django, and ASP.NET MVC, and PHP, and you name it - is better that everything we ever had before, but there are a few things that irk me about the current state of affairs:

If I want to change your UI (and I don't mean cosmetically), it's almost guaranteed that I have to learn your framework. At a minimum, I'll have to learn your templating language, but my point is that I cannot do much if I only know HTML and JavaScript. Conversely you will not be able to make use of my hypothetical UI skills unless I also happen to know your framework.
It's laborious to get your project to run. Sometimes it's easy, most often it's not; I'll run into dependency issues, versioning issues, versioning of dependencies issues, and project setup issues when all I wanted to do was to make a minor UI change.
I need to run your whole stack to be able to see and change your UI: database, modules (gems, eggs, dlls), services, etc. If any of your dependencies is truly external (say that SAP system you get your list of clients from) you most likely can kiss your off-lining goodbye, unless you were smart enough to provide a stub service that still serves relevant data.
If your project provides a UI but doesn't provide a separate API, it's very difficult or at least impractical for me to consume your data. This is more difficult to accept in an enterprise where we're all supposed to be working towards the same goal. And building a separate API slows you down, and increases your maintenance costs because now you have two pieces of software you need to maintain, unless you were smart from the beginning and made your app consume your API.
There's a clear boundary between the client - the web browser - and the server, with a good deal of data being sent back and forth, yet you mostly never write tests for the communication between browser and server. You don't do that for the same reason: it's difficult or impractical to consume what you send (GET), slightly easier to consume what you receive (POST, PUT, DELETE).
In some setups it takes time for a model change to propagate to the UI, more so with compiled languages. Thor forbid if you need to make a DB change - now we have to wait for the DBA(s).
It's hard to scale your app because building your UI often needs all the models it uses to have been loaded in the same thread/process. Most of the time, your apps, or at least the layers building the UI, are not designed or cannot handle different models coming from different sources.
You are building your UI sequentially. Slowest buffalo wins: you have to wait for your slowest model to load before you can send your page to the browser. Not that async building would always make a difference - sometimes you just have to wait for that drop-down to populate before you can successfully edit that record - but in a sync world it doesn't matter if the relevant data is fast and the irrelevant data loads slowly, because it all boils down to everything having to finish loading before you produce your UI.

As web developers we cringe at the thought of putting markup in the database, or having stored procedures generate HTML, but we have no problems with code generating HTML. I'm not saying it's hypocrisy, but it's the first word that comes to mind.

If you're a desktop developer, imagine your services spewing UI controls instead of just typical web service XML. Imagine your UIToolbar, NSButton, ContextMenuStrip, or your GridBagLayout being the result of a web service call; I don't mean a UI control that you create in your UI layer based on a web service call, but an actual object that you only deserialize and put on screen at (x,y). I get a shiver just writing this, but in the world of web apps it's something perfectly natural.

Why? Because we have a good templating language? Because we've always done it this way? Because everything comes from a server anyway?

Between mash-ups and web services, we've already solved a great deal of those issue. We just need to apply them rigorously and consistently.

Proposal

I propose that we build the app UI using static HTML (as in not produced by other code) and that we use only JSON to exchange business data, by means of the four HTTP verbs (GET,POST,PUT,DELETE), essentially turning your Rails/Django/ASP.NET MVC/PHP app into a lean-mean JSON producing/consuming machine.

Loading the UI for displaying or editing an entity would take the following steps:

The browser requires from the web server http://example.com/clients/1. It receives a static HTML page representing the layout required to display the client and JavaScript code that knows how to perform the next step.
The JavaScript code (e.g. jQuery) loads, perhaps from another server, http://data.example.com/clients/1, the JSON representation of Client(1).
The JSON values are displayed into the static layout (for example, using Knockout.JS).
Saving is performed by POSTing the updated JSON representation back to the originating data URL (http://data.example.com/clients/1).

If we consider that most pages fall into a few larger categories: display entity list, display one entity, edit one entity, etc. the first step can be further generalized by having it serve a more generic static template for the category and having it load more detailed template for the entity:

The browser navigates to http://example.com/one_entity.html#clients/1. It receives a static HTML page representing the generic layout for displaying one entity and JavaScript code that knows how to load the detailed template, and how to load the entity data.
The JavaScript code loads http://example.com/templates/client.html and inserts it into a known placeholder node.
The JavaScript code then loads JSON from http://data.example.com/clients/1 and populates the template received in step two.

This latter formula allows us to focus on the details of what differs between various entity displays and maintain the same layout.

Advantages

With this approach:

If I want to change your UI, all I have to learn is your API, which is as little as looking at what JSON objects you send me.
I don't need to get your project to run: all I need are the static HTML files. I can serialize your JSON replies to individual files and serve them as such. Sure, I won't be able to send them back, but at least I can navigate most of your app.
I don't need to run your whole stack: all I need is a web server.
You don't need to provide a separate API - your web services are now your API. I can integrate with your API just as easy as you can.
We both can now write and run tests for that client-server boundary. By preserving the JSON you send me at one point in time I can make sure that any changes you made in your active code don't cause surprise midway through my development cycle. You too can serialize known replies to make sure that code changes elsewhere have not caused changes in the API.
UI changes are in almost all cases as easy as a page refresh. You can make your model and DB changes later - as long as you send me the JSON structure we agreed on my UI will behave correctly.
You can scale your app through sharding at domain level, different JSON sources coming from different servers, and I can make sure I will configure my URL(s) to account for that. That also means that if I want to get clients from that SAP server you don't have to write a bridge, I'll just use theirs (which hopefully provides a JSON interface).
Because asynchronous tasks is something JavaScript is really good at, I can load my interface asynchronously: I can load the list of clients from server 1, the list of countries from server 2, and current market prices for lolcats from server 3 without doing anything I wouldn't normally do in JavaScript. Actually making things synchronous in JS is sometimes the more complicated approach (ask NodeJS programmers).

On top of that:

What you cache (JSON) became considerably smaller and any web server can already deal incredibly efficiently with static HTML.
You can send an implementation team to a client site that doesn't have to know much beyond HTML and JavaScript, and let's be honest - that skill set comes in considerably cheaper.
You can finally get that HTML5 guru that loathes working with ASP.NET. Because he won't have to. And he'll get to use his Mac (yay!).

Between libraries like jQuery and Knockout.JS (or Backbone.JS) we already have all the tools to make it happen. Chances are you already use them with your app - why not go all the way.

Notice that I was careful not to say the app is RESTful - that's because my proposal doesn't address the fourth guiding principle of Fielding's cannon. That's something for you to decide (and I think you should do it).

Here's a simple example, using Knockout.JS, to see what I'm talking about: http://github.com/philipmat/webmvc/.

.Net Interfaces Are Not Classes

2011-10-14T00:00:00-05:00

Well, duh! Right?

To be honest, if you know that already, and who doesn’t, then this post does not have much to offer, except a tale of DataBinding exceptions, TypeConverters, and a few interesting tid-bits I picked up along the way, along with what I think it’s a bug in a core .Net class.

Throughout this post I will use two interface and two classes implementing them to make my point:

interface IBase {
  string BaseMethod();
}
interface IDerived : IBase {
  string DerivedMethod();
}

class Base: IBase {
  public string BaseMethod() { return "BaseMethod"; }
}
class Derived : Base, IDerived {
  public string DerivedMethod() { return "DerivedMethod"; }
}

Rocket surgery, innit?

data binding and interfaces

Good OOP code monkeys have designed and programmed against abstractions, e.g., interfaces, before Uncle Bob articulated the SOLID principles, and I like to believe that even before the advent of the Dependency Inversion principle.

It only makes sense that we apply the same principle, programming against interfaces, to MVVM design, as it shows in this overly simplistic example:

class MyForm : Form {
  public IBase Base { get; set; }
  public IDerived Derived { get ; set; }

  protected override void OnLoad(EventArgs e) {
    this.Text += ":" + (Base == null 
                          ? "Base is null" 
                          : Base.BaseMethod());
    this.Text += ":" + (Derived == null 
                          ? "Derived is null" 
                          : Derived.DerivedMethod());
  }
}
class MyViewModel  {
  public IBase Base { get { return new Base(); } }
  public IDerived Derived { get { return new Derived(); } }
}

I’m using Windows.Forms because WPF would’ve been too verbose for such a tiny task, but the concept translates almost word for word.

MVVM was designed to make use of binding, DataBinding to be more specific:

void Main()
{
  MyForm viewBase = new MyForm { Text = "Base" }, 
         viewDerived = new MyForm { Text = "Derived" };
  var viewModel = new MyViewModel();

  viewBase.DataBindings.Add("Base", viewModel, "Base", 
                            false, 
                            DataSourceUpdateMode.OnPropertyChanged);
  viewDerived.DataBindings.Add("Derived", viewModel, "Derived", 
                               false, 
                               DataSourceUpdateMode.OnPropertyChanged);

  viewBase.Show();
  viewDerived.Show();
}

Running that code (here is the script to use with LINQPad) we’d expect to get two windows with titles “Base:BaseMethod”, and respectively “Derived:DerivedMethod”, yet instead we get two big, fat, FormatExceptions that make as much sense as a fish with a bicycle (full stack trace here and also notice how early in the control’s lifecyle this error happens):

System.FormatException: Cannot format the value to the desired type.
   at System.Windows.Forms.Binding.FormatObject(Object value)
   ...
   at System.Windows.Forms.Control.UpdateBindings()
   ...
   at System.Windows.Forms.Control.CreateControl()
   at System.Windows.Forms.Control.WmShowWindow(Message& m)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
   at System.Windows.Forms.Form.WmShowWindow(Message& m)
   at System.Windows.Forms.Form.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Cannot format ?! And format what – a Base into an IBase? How is that possible?

the devil is in the code

Very puzzled, I pulled out a decompiler, JetBrains’ dotPeek, took a look at Binding.FormatObject(Object), and found these interesting lines:

//-  propertyType is typeof(IBase)
//-  value is our new Base()
//-  so is obj1
if (propertyType == typeof (object))
  return value;
if (obj1 != null && 
    (obj1.GetType().IsSubclassOf(propertyType) || 
     obj1.GetType() == propertyType))
  return obj1;

I’m pretty sure this would’ve jumped out at you, dear reader, because you’re so much smarter, but in my head it didn’t make any sense; I thought “of course Base is a child of IBase – what’s your problem?”.

Yes, Base is a child of IBase, for very fuzzy values of child, but it is not a subclass of IBase: Base implements IBase but does not inherit from IBase.

Given the direction of the assignment, I believe this is a bug: IsSubclassOf is an unnecessary restriction, they should have used propertyType.IsAssignableFrom(obj1.GetType()) instead, which would’ve taken care of this mess and allow for binding onto an interface to happen as expected.

Well, let’s debate that later, how do we fix it?

Obviously we could change MyForm.Base to be of type Base and MyForm.Derived to be of either type Derived or Base. That would work, and it’s a reasonable fix if you happen to dislike abstractions.

You could also change both properties to be of type object, but that just means you have OOP in general.

The next few instructions in Binding.FormatObject contain another solution:

TypeConverter converter = 
    TypeDescriptor.GetConverter(value != null 
                                ? value.GetType() 
                                : typeof (object));
if (converter != null && converter.CanConvertTo(propertyType))
  return converter.ConvertTo(value, propertyType);

You can read more about the TypeConverter, while I create one for the Derived class and then decorate it using the TypeConverterAttribute:

class DerivedConverter : TypeConverter {
  private readonly Type typeD = typeof(Derived);
  public override bool CanConvertTo(ITypeDescriptorContext context, Type destinationType) {
    if (destinationType.IsAssignableFrom(typeD)) 
      return true;
    return base.CanConvertFrom(context, destinationType);
   }
   
  public override object ConvertTo(ITypeDescriptorContext context, CultureInfo culture, 
         object value, Type destinationType) {  
    if (destinationType.IsAssignableFrom(typeD)) 
      return value;
    return base.ConvertTo(context, culture, value, destinationType);
  }
}

[TypeConverter(typeof(DerivedConverter))]
class Derived : Base, IDerived {
  public string DerivedMethod(){return "DerivedMethod"; }
}

If you rerun the script with the TypeConverter in place, you’ll only get one exception for IBase, while the IDerived example performs as expected.

Kind of silly, isn’t it, to use a separate class to convert an instance into the interface it implements anyway.

It was at this point that I did what any sane developer would do when faced with this snafu: bitch on Twitter. Rory Primrose (@roryprimrose – you should follow him) replied with an example of his own, this one around interface inheritance.

interface inheritance

His assumption was that calling Type.GetMethods on a derived interface would also return the methods of its parent(s).

I had to try and cook some of my own code and here are the surprising results:

BindingFlags 
  inst_pub = BindingFlags.Instance | BindingFlags.Public,
  flat_inst_pub = BindingFlags.FlattenHierarchy | inst_pub,
  decl_inst_pub = BindingFlags.DeclaredOnly | inst_pub;
Action<string, Array> logm = (s, a) => a.Length.Dump(s);

Derived d = new Derived();
Type iB = typeof(IBase),
iD = typeof(IDerived),
cB = typeof(Base),
cD = d.GetType();

logm("Instance | Public", cD.GetMethods(inst_pub));                   //=> 6
logm("Flatten | Instance | Public", cD.GetMethods(flat_inst_pub));    //=> 6
logm("Declared | Instance | Public", cD.GetMethods(decl_inst_pub));   //=> 1

logm("Instance | Public", iD.GetMethods(inst_pub));                   //=> 1
logm("Flatten | Instance | Public", iD.GetMethods(flat_inst_pub));    //=> 1?!

cD.IsSubclassOf(cB).Dump("Derived subclass of Base?");                //=> True
(d is Base).Dump("Derived is Base");                                  //=> True
cD.IsSubclassOf(iD).Dump("Derived subclass of IDerived?");            //=> False
(d is IDerived).Dump("Derived is IDerived");                          //=> True

iD.IsAssignableFrom(cD).Dump("Derived assignable to IDerived?");       //=> True
iD.IsSubclassOf(iB).Dump("IDerived subclass of IBase?");               //=> False?!
iB.IsAssignableFrom(iD).Dump("IDerived assignable IBase?");            //=> True

The classes behave as expected (2 methods + 4 from Object), but I expected FlattenHierarchy with IDerived to return two methods: its own DerivedMethod and its parent BaseMethod.

Probing further, shows not only that classes are not subclasses of the interfaces they implement, but neither are interfaces subclasses of their parents.

Much sooner that I, Rory asked the question at the core of his post:

Does interface inheritance exist?

Much to my amazement, the answer is no, at least according to reflection. This is completely not what I had assumed.

when in doubt

… look up the specs.

In highsight, it should all have been obvious. My mistake lies in thinking of interfaces as purely virtual/abstract classes (there’s some C++ roots showing – only pointing that out to get some street cred).

However, that shows fault immediately because, even if it was so – even if interfaces were purely virtual classes, the CLI does not support multiple class inheritance (but you knew that already, didn’t you), yet it supports multiple interface inheritance using the exact same syntax.

The standard, ECMA-335, explicitly states in 8.9.9 Object type inheritance that:

[…] all object types shall either explicitly or implicitly declare support for (i.e., inherit from) exactly one other object type.

If I were to nitpick though, I would point out that the C# standard, ECMA-334, is a bit inconsistent in usage: within 8.9 Interfaces you will read “Interfaces can employ multiple inheritance”, which is contradicted by ECMA-335 8.10 Member inheritance:

While interface types can be derived from other interface types, they only “inherit” the requirement to implement method contracts, never fields or method implementations.

This is a CLI restriction: while ordinary C++ does support multiple inheritance, Managed C++ doesn’t. This blog post shares Managed C++ team’s decision and includes this grain of insight about Eiffel’s support of multiple inheritance on the CLI:

[…] the CLI does not, for example, support private inheritance, value inheritance (that is, the inheritance of implementation but not of type), or multiple inheritance (MI). While a language can choose to support these aspects of inheritance, that support requires a mapping onto the existing CLI object model because there is no direct support.

The Eiffel language under CLI, for example, choose to provide an MI mapping […]

conclusion

What did we learn?

In truth, there’s no true inheritance in the CLI when it comes to interfaces – that is reserved for classes; in practice, we can consider interface inheritance exists as long as we use those relationships for typing rather than reflection.

If you perform runtime type inspection with a goal to see if casting is possible, use Type.IsAssignableFrom, rather than Type.IsSubclassOf. I cannot think of a good use case where you’d explicitly want IsSubclassOf and that would not betray your abstractions, but it’s late and I cannot think well during normal hours, let alone late into the night.

When you need to list the members an interface exposes, including the members of its parents, you have to perform recursive inspection using GetInterfaces() and then query those results in turn for members.

If DataBinding throws wrenches in your spokes because the target type is an interface, either replace it with an abstract class or use a TypeConverter in combination with a TypeConverterAttribute.

Three Ways To Indent JSON

2011-09-01T00:00:00-05:00

Python comes with a JSON encoder and decoder you can use to prettify messy JSON code.

On the Command Line

If you have an unindented JSON file:

cat ugly.json | python -mjson.tool > pretty.json

(May I recommend the wonderful DTerm to run commands from almost any app you might be in?)

From within VIM

Using filter commands, !, replace the text with its formatted version, filtered through the same Python command:

:%!python -mjson.tool

Since it's VIM, you can replace as little or as much as needed; see Use filter commands to process text for more information.

With a Service

Create an Automator service and use the Run Shell Script action to process the input text (make sure to select the Output replaces selected text checkbox), once again using Python:

Since Automator automatically saves services to the ~/Library/Services/ directory, it'll become available to you instantly, and it'll allow you to go from this:

to this:

.Net DI Container Speed Redux

2011-08-19T00:00:00-05:00

Following my previous .Net DI container speed test, a few people (thanks @slaneyrw, @roryprimrose, @JamesChaldecott) pointed out that the abysmal performance of Unity in both singleton loaded and transient loaded scenarios was largely due to checking for object registration before requesting. For example @roryprimrose detailed that :

@philipmatix The perf problem in Unity is actually in the IsRegistered call (35,347ms), not the Resolve (580ms).

A discussion on CodePlex with Chris Tavares and Grigori Melnik further denoted that, from Unity's perspective, the use of IsRegistered is intended for debugging your apps only.

With that in mind, I've decided to add tests that show the performance of containers when no registration checks were performed. Following the numbers and charts I'll also provide my thoughts on why I believe both cases have merit.

Spoiler alert: Unity redeems itself, and although not quite the speediest container, it put out some pretty damn good numbers.

The rest of this post is broken down as following:

performance without registration checks - in which we look at the numbers the containers yielded out when no registration checks were performed;
the registration check debate - in which we talk about whether IsRegistered should be performed at all;
alternatives to registration checks - we run a few scenarios that deal with optional dependencies - eschewing registration checks, and see the performance of those alternatives;
conclusion - where I stop thinking.

Performance without Registration Checks

The following table displays the improvements that took place once the registration check was removed. I expected more impressive gains; I guess all containers but Unity have really efficient registration checks and as a result have only posted gains that hovered mostly between 1.1x to 1.3x, with a few weird numbers in between.

               Sgl             Trans           Sgl - L           Trans - L
                10k    100k     10k   100k      10k     100k      10k     100k 
Autofac         1.3     5.7     1.0    1.0      1.0      1.1      1.0      1.1
CastleWindsor   1.1     4.2     1.0    1.1      1.2      1.1      1.0      1.0
Ninject         1.0     2.1     1.0    1.0      1.1      0.9      1.0      1.0
Spring.Net      1.3    22.0     1.0    1.0      1.2      1.3      1.0      1.0
StructureMap    1.0     1.2     1.2    1.1      1.1      1.3      1.0      1.1
Unity           3.3     3.7     2.7    2.9     78.2    169.9     68.0    148.2

Of course, the big news here is that Unity, unencumbered by registration checks, posted speeds that were from 68 to 169 times faster.

To make it clear, the change in code was:

// Old and busted
if (container.IsRegistered<IDummy>()) 
    return container.Resolve<IDummy>();

// New hotness
return container.Resolve<IDummy>();

Another way to look at these numbers is to compare the contribution of each container to the total time it took each scenario to run. Of course, Unity's trimmed out since it caused the chart to go nuts (to give you an idea this portion you see here is approximately 7 inches - Numbers' measurements - out of 50 inches that was the size of the entire chart).

If it wasn't obvious, "w/" indicates with registration checks and "w/o" means without registration checks.

An interesting detour for me was to try and get at least a superficial understanding of why Unity's IsRegistered performed so poorly. As an extension method it doesn't do much - it simply iterates over the unityContainer.Registrations picking up the one whose type matches the requested type. This is where things differ between Unity and almost all other containers: whereas those containers build their registries at the point where you register the types, Unity builds its list on each call to Registrations.

Indeed, each call to Registrations builds an IDictionary<Type, List<string>>, which is then filled in with types and names retrieved from the NamedTypesRegistry, here represented by registeredNames:

private void FillTypeRegistrationDictionary(IDictionary<Type, List<string>> typeRegistrations)
{
  if(parent != null)
  {
    parent.FillTypeRegistrationDictionary(typeRegistrations);
  }

  foreach(Type t in registeredNames.RegisteredTypes)
  {
    if(!typeRegistrations.ContainsKey(t))
    {
      typeRegistrations[t] = new List<string>();
    }

    typeRegistrations[t] =
      (typeRegistrations[t].Concat(registeredNames.GetKeys(t))).Distinct().ToList();
  }
}

Finally, on return, this dictionary is then transformed by a LINQ query into an IEnumerable<ContainerRegistration>.

It's obvious that having this code executed 100,000 times, more so when loaded with a significant number of classes, will not win any performance awards.

If you are wondering, like I did, this was a conscious design decision, as indicated by Chris Tavares, who said:

Yeah, IsRegistered is not optimized at all - in fact, it's almost deliberately suboptimal [...]

(I left out a very important second part of his response, which I'd like to use in the registration check debate section).

Fair enough, it's a decision I respect, and my intent is not to provide criticism, but a mere heads-up to fellow developers, given a great deal of people cheered the introduction of IsRegistered in Unity 2.0. Since I am quite fond of Unity, I'd very much like to see it brought on to perform on parity with the other containers, in all aspects.

The Registration Check Debate

In the same reply that I've extracted the quote above, Mr. Tavares asked a very good question: Why do you call IsRegistered at all?, and expressed an opinion I share: if you're using it a lot you're most likely doing DI wrong.

In broader terms, all these containers are truly IoC containers. IoC is an approach to handling object dependencies, a meta-pattern if you will, whereas Dependency Injection - which itself comes in about three flavors - is an implementation pattern. The other common IoC pattern is Service Locator.

I believe that even if Martin Fowler didn't coin the two terms, he gave them the definition that people usually link to. You should read his article, but a very simplistic view would say that when you use DI you end up with all your dependencies injected in your classes, ready for use, whereas with SL you use a dedicated class to construct your dependencies right in the places in code where you need them. There are various use cases for either, but the general feeling, at least my feeling, is that DI presents a cleaner approach.

For all its cleanliness, Dependency Injection does require that you start without prior restraints. Not quite the case when dealing with a good mess of code where the High Cohesion, Low Coupling principle wasn't even known to the developers, let alone employed in an effective way.

It is from this direction I came and, as Michael Feathers points in his seminal book Working Effectively with Legacy Code, Service Locator is an effective tool to help break tightly coupled classes (think of SL, if you will, as the gateway drug to DI). I think in part due to this type of situation, and I'm sure due in part to developers avoiding the DI because it requires a good deal more discipline, the presence of a SL in code is perceived as a good indication of code smell.

I got similar comments from other people. @dot_NET_Junkie was stronger in his statement, calling it an anti-pattern. Although not in such intense terms, I do share his feeling, but sometimes it's not entirely fair. Given my use-case, decoupling classes so that I can eventually get to pure-DI, it would be akin to calling a crutch an anti-pattern when all you're trying to do is get a little mobility waiting for your broken leg to mend.

At the end of the day, all that these containers provide is dependency resolution, how you employ them, whether pure DI or most likely a combination of DI and SL, it's up to you. What didn't escape me is the irony of the Service Locator being present even in the purest of DI cases: after all, how do you get that very first object, the root of your entire running hierarchy, out of the DI container? You ask for it, you get it, and there's a name for that: service location. Glad we've established that; now we are just haggling about the price..

Alternatives to Registration Checks

There's another interesting use case for a service locator, at least one in which it is considerably easier to employ it versus dependency injection: optional plugin points. What all these containers we looked at have in common is that when requesting an object that the containers cannot construct they will throw an exception: Unity throws a ResolutionFailedException, Ninject throws an ActivationException, etc. Given exceptions tend to be somewhat expensive in .Net, never mind the silliness of considering the absence of an optional component an exceptional case, it sounds reasonable that we'd check whether a component is registered before requesting it from the container.

But don't trust me when I say exceptions are expensive - let's look at some numbers. In the following benchmark I am trying to retrieve an object that is not registered and guard against it missing, first using IsRegistered and then by handling the exception thrown by the container.

public IDummy Run_IR(Type t, string name) {
    if (k.IsRegistered(t, name))
        return ((IDummy) k.Resolve(t, name));
    return null;
}

public IDummy Run_Ex(Type t, string name) {
    try {
        return (IDummy) k.Resolve(t, name);
    }  catch (ResolutionFailedException) {
        return null;
    }
}

The testing methodology differs from the previous cases, too. Instead of loading the containers with 222 interfaces (666 classes) from the start, I will load them progressively, starting with 1 registered interface (3 named classes) and going up increments of 20. Furthemore, as I do this because of my curiosity around Unity, I will only test it against Autofac (no other reason than it was the first one alphabetically). If you're interested in the source code, you'll find it in the ex_vs_isreg branch on github; the only relevant classes are VariableLoadAutofacRunner and VariableLoadUnityRunner.

What I've learned? ResolutionFailedException is CRAZY expensive in Unity. Took me about two days to tweak the tests because I thought I coded an infinite loop. Here's a useless chart:

Look at the values behind the chart, and yes, you're reading it right - it took 43 minutes to run a single loop of 10k queries:

# Regs      Autofac               Unity
         5x10k  1x10k       5x10k       1x10k
         IsReg     Ex       IsReg          Ex       
1            4    508          70     198,241
21           2    378         803     205,340
41           2    376       1,571     205,370
61           2    375       2,296     210,353
81           2    378       3,061     212,034
101          2    378       3,876     212,431
121          2    372       4,712     219,162
141          2    373       5,277     219,637
161          2    565       6,397     223,136
181          2    375       6,774     223,365
201          2    375       8,012     226,826
221          2    374       8,720     232,349
Run time    26   4827      51,569   2,588,244

You can see that although exceptions in Autofac are about 150x more expensive than IsRegistered they don't even hold a candle to Unity's.

So exceptions are out, what else? Unity provides a built-in way to handle optional dependencies by the way of an OptionalDependencyAttribute that you decorate your injection points with. If Unity can find a dependency it will inject it, otherwise it will pass in null. The attribute is not the only way, as Chris Tavares wrote:

Optional dependencies can be configured the same way as any other dependency - through attributes, the API, or the config file. As such, you don't have to put attributes in your code anywhere. Just resolve an object with the optional dependencies, and that object will get them injected (or not) if the container has a registration for it.

I have no reason to doubt him, but I was hard pressed to find either examples or documentation on how to handle them without the attribute. I'm sure that if you ask, either him or Grigori Melnik will happily provide such examples.

How does the OptionalDependency attribute handle? Quite well.

In the table below, the first column contains the progressive number of interfaces registered (x3 for objects), the second is the 10x10k run time for straight Resolve<IDummy>, i.e. where the classes had no dependencies, the third column contains the numbers for the dummies having an external optional dependency but with none registered to satisfy it, whereas in the third column there is a dependency to satisfy the constructors of the dummies.

# Reg     Straight  Optional,  Optional, 
                    None Reg   One Reg
1           33         36         34
21          30         32         32
41          33         39         38
61          37         45         45
81          40         54         53
101         46         64         62
121         51         75         73
141         52         87         85
161         58        100        103
181         62        114        110
201         78        131        124
221         76        162        142

Very respectable numbers provided you are either ok with peppering your code with [OptionalDependency] (ugh), or use a config file (ugh), or you figure out how you can configure it through the Unity API (yay!).

There's one more way - of course I left the best for last: the most elegant approach to handling optional components is to employ the Null Object pattern, in other words to provide a default, no-op implementation for all optional components. That way, you can use the container with no registration checks and get the excellent performance all containers provide, and when you have a specific implementation you simply override the existing registration in the container. As a supplemental benefit, the Null Object pattern saves you one of those ugly but required null checks.

Conclusion

The best use case for any of the six containers we looked at, now and a few weeks ago, Autofac, Castle.Windsor, Ninject, Spring.Net, StructureMap, and Microsoft Unity, and probably the best use case for Dependency Injection in general, is one where all your dependencies are satisfied. If you find yourself in this fortunate position, 4 out of the 6 containers produce results so close it's not worth giving it a second thought - go with what you have or know. Even in the Ninject case, if you're familiar with it, stick with it. Best you use any container than go without. If you're forced to use Spring.Net, I'm sorry; I hope you reap the benefits of the rest of the framework.

If you find yourself employing a Service Locator pattern and Unity, examine whether you have a great deal of optional dependencies or, for that matter, any reason to perform IsRegistered checks. Best you don't, whether you use Unity, or any other container. If you do need to deal with optional dependencies, go over the alternatives to registration checks - your best bet is the Null Object pattern, it's a technique that works with all containers. It's an awesome pattern in general, and one that applies greatly to almost all situations, not just DI, and yet, sadly, I don't see employed enough.

The source code for my tests is on github. I have now created several branches for the various experiments we have performed: the original, using IsRegistered is in the with_isregistered branch; the direct Resolve without IsRegistered is in the without_isregistered branch; the attempts to measure the impact of exceptions in case of failed resolutions - for Autofac and Unity only - is in the ex_vs_isreg branch; finally, the examination of Unity's performance with the OptionalDependencyAttribute is in the ex_vs_isreg_vs_opt branch (badly named because it doesn't try to measure the performance of exception handling).

.Net DI Container Speed Test

2011-08-08T00:00:00-05:00

You probably don't care why I've done this, but if you don't even care about most of the details of this test, let me lay it on you short and sweet: Autofac, Castle.Windsor, and StructureMap put out some excellent, consistent numbers; Spring.Net is middle of the road; Ninject is consistently the slowest of the pack by several orders of magnitude; and finally Unity showed such a weird behavior that made me question both the validity of my approach and the sanity of its developers (mostly kidding, ctavares is awesome). Note: there's an update which examines some of the problems with Unity; you'll probably want to read it.

I have elected to run against the latest versions of the six major containers of the .Net world: Autofac v2.4.5.724, Castle.Windsor v2.5.3, Ninject v2.2.0.0, Spring.Net v1.3.1, StructureMap v2.6.1, and Microsoft Unity v2.1.0.0 using the .Net 4.0 framework. I expect the results to be similar if you were to use 3.5; I could be wrong, but I don't think any of them uses exclusive 4.0 features in areas that would make an impact on these results.

methodology
results
retrieving singletons
transient objects
retrieving singletons - loaded containers
transient objects - loaded containers
my take

Methodology

My tests (source code available on github) concern themselves almost exclusively with the speed with which objects are retrieved from the containers. Memory footprint was not even a secondary concern and, to be honest, I don't even know how to measure it with any kind of precision. The code contains a modest attempt at memory measurements - if you believe it to be correct, there's a setting in app.config to print it out as you run the tests.

I believe there are two scenarios that are relevant to observing the performance of these containers: retrieving the same instance each Resolve<T>() call, i.e., singleton scope, and retrieving a new instance on each call a.k.a. transient scope. I don't think either solution is what you would see in real life situations; most likely you'd encounter a mix of the two, with transient objects being injected with singletons and transients alike, but I couldn't come up with a recipe that wouldn't be easily dismissible on account of lacking balance in proportion of singleton to transient objects. At least taking extremes only leaves you to defend from one direction.

Resolving singletons tests the container's raw retrieval speed and its registration internals. Creating new objects on each call tests the container's build plan and the strategies it uses to match up objects.

All containers allow more advanced object lifestyles, e.g., single instance per thread or per HTTP request, but I don't have enough confidence in my abilities (I'm also lazy) to come up with a clear strategy to produce measurements that are a) accurate, b) relevant, and c) somewhat equivalent across containers. However, I think the singleton and transient stories are somewhat indicative of other usage scenarios.

In both cases my program will load and configure all the containers then perform 10 runs of a loop requesting the same object a large number of times (10k and 100k) then average those 10 runs. I will refer to these two as the 10k run and respectively the 100k run. So that's four combinations so far: singleton 10k, singleton 100k, transient 10k, transient 100k.

Finally, to add a fake real world flavor to the test, I will repeat the four test scenarios with the containers loaded with some significant number classes in order to simulate a container under some pressure and make 10k and, respectively, 100k requests of a number of objects randomly selected. I don't think any container performs any kind of heuristic analysis of the requests with intent to provide a quicker resolution path for those objects requested more/most often, though I wish they did.

I am running this in a virtual machine that I will restart before each of the eight scenarios and then I will run the test program three times. I will post the numbers from both the first and third run, but will favor the third run for figures and charts.

Results

Individual results are boring, but are presented in the next sections for your perusal, so let us take a look at pretty charts presenting the overall situation.

I had to force a max value on the Y-axis (milliseconds) seeing how Unity completely blew the scales. As such, please pay attention to the scale on the left and occasionally to the number on top of the bars.

The chart of timings by task gives us a focused view and paints the relative behavior of containers within those groups:

Notice how Autofac, Windsor, and StructureMap behaved consistently well, Ninject behaved consistently badly (although that's an exaggeration), whereas Spring.Net and Unity were highly irregular in their numbers.

Plotting the same numbers by container shows the areas of strength and weakness of each participant, but also how the containers behaved overall relative to each other:

We also get to see quirks, such as four out of six containers showing a dip in the 100k singleton loaded test relative to their other numbers.

All containers are roughly on parity syntactically and feature-wise - one can easily move from one to the next with little training, the only significant differentiator that I observed was elegance of configuration, but that's hardly something to hang your project onto.

Let us look at individual numbers in ugly tables next. Feel free to jump to my take if this is not of interest to you.

Retrieving Singletons

I believe this is the most common case in construction of DI trees and it's after all just an optimization of the transient scenario. It's suitable for classes whose context remains unchanged through the execution of the program, be they complex or not. If you rely (solely? largely?) on the constructor flavor of dependency injection, chances are everything behaves as if it was in singleton mode anyway. However, if you use a Service Locator approach, whether exclusively or not, the difference between singleton and transient becomes something you have to pay more attention to.

Retrieving singletons is a case of configuring the containers to indicate that once an object is created, the same instance should be returned for each subsequent Resolve call.

// Castle.Windsor registration
k = new WindsorContainer();
k.Register(
        Component.For<IDummy>().ImplementedBy<SimpleDummy>().LifeStyle.Singleton
        );

// Retrieval
if (k.Kernel.HasComponent(typeof(IDummy)))
    k.Resolve<IDummy>().Do(); // Do is void Do() {}

The results of the runs:

:               10k - 1st  10k - 3rd  100k - 1st  100k - 3rd
------------------------------------------------------------
Autofac                15         14         688         635
Castle.Windsor         22         23         807         790
Ninject               123        121       2,616       2,558
Spring.Net             17         16       1,982       1,935
StructureMap           21         21         212         205
Unity                  93         98         992         996

         Median        22         22         900         893
            Min        15         14         212         205
            Max       123        121       2,616       2,558

The median across containers for Resolve()-ing the same object ten thousand times is around 20 milliseconds, with four out of six containers being close around that number. Of the two offenders, Ninject was 8.6 times slower than the fastest container, Autofac, and Unity close on its heels at 7x the speed. The 100k run only served to make the gap more obvious with Ninject clocking in a whooping 12.5x slower. Surprisingly, the second fastest container in the 10k run, Spring.Net was now the second slowest (9.4x slower).

Transient Objects

The transient scope is most common with classes that control access to limited resources: database connection, web service requests, etc., or whose context changes on every call. Or if you just want to be safer at the expense of a bit performance. My gut feeling is that in most cases these type of classes will have either creation or use costs that outweigh the container's, but nevertheless I thought it makes for an interesting test case because it tells us how efficient each container's build plan is. I don't expect any of the containers to use the very slow Activator.CreateInstance, but there are various strategies that can be employed to create new objects and calculate their dependencies, all which can use different lifestyles.

As the test focuses on object creation I chose the route of an object with no dependencies, rather than the more complex but more realistic case of a class that gets injected with other dependencies, which themselves might be dynamically resolved at that point.

// Autofac configuration - defaults to transient lifetime
var builder = new ContainerBuilder();
builder.RegisterType<SimpleDummy>().As<IDummy>();
k = builder.Build();

// Retrieval
if (k.IsRegistered<IDummy>())
    k.Resolve<IDummy>().Do();

The results are, as expected, a good deal slower than the singleton scope scenario, except for StructureMap, which captured the crown once more:

:               10k - 1st  10k - 3rd  100k - 1st  100k - 3rd
------------------------------------------------------------
Autofac                64         64         710         634
Castle.Windsor         80         80         845         797
Ninject               255        253       2,575       2,678
Spring.Net            189        185       1,856       1,874
StructureMap           21         21         202         192
Unity                 109         99       1,008         956

         Median        95         90          927        877
            Min        21         21          202        192
            Max       255        253        2,575      2,678

I expected the 100k test to repeat the story of the singleton and to increase in the same proportion from the 10k figures, yet instead the numbers came close to those of the 100k singleton run, for all containers.

Ninject sticked to its guns and managed again to be 12x slower than the fastest container in the 10k race - StructureMap, and almost 14x slower in the 100k run. Worthy mention of suckage: Spring.Net, which managed slowness factors of 8.8 and 9.8 relative to StructureMap.

Retrieving Singletons - Loaded Containers

Based on what I've seen of their respective source code, each container takes different approaches to storing class registrations and as such when a resolution is performed, each container will use a different strategy to either retrieve an existing instance or create a new one (Unity calls them build plans, which I like a lot). I believe this test will attempt to show how well each container's approach scales when the container has a good deal of objects registered and created.

I didn't know what was supposed to be a realistic number of classes to saddle a container with, so I've picked pretty numbers. I've loaded each container with 666 classes registered for 222 interfaces (three name discriminators for each interface). I made then 10k and respectively 100k random requests, pairs of interface/discriminator, to resolve those registrations.

// StructureMap - registering named objects
ObjectFactory.Initialize(x => {
    x.For<IDummy0>().Singleton().Use<SimpleDummy0>().Named("0");
    x.For<IDummy0>().Singleton().Use<SimpleDummy1>().Named("1");
    x.For<IDummy0>().Singleton().Use<SimpleDummy2>().Named("2");
    x.For<IDummy1>().Singleton().Use<SimpleDummy3>().Named("0");
    ...
    x.For<IDummy221>().Singleton().Use<SimpleDummy664>().Named("1");
    x.For<IDummy221>().Singleton().Use<SimpleDummy665>().Named("2");

    });  

// Retrieval of named objects - SM doesn't have an IsRegistered
IDummy d;
if ((d = (ObjectFactory.TryGetInstance(t, name) as IDummy)) != null)
    d.Do();

On one hand I expected the figures to be a bit slower, but not by much. The containers should have efficient ways to sift through the registrations to find the proper resolutions.

:               10k - 1st  10k - 3rd  100k - 1st  100k - 3rd
------------------------------------------------------------
Autofac                25         23         166         163
Castle.Windsor         35         39         342         328
Ninject               128        133       1,053       1,040
Spring.Net             59         36         223         229
StructureMap          114        118         289         348
Unity               6,012      5,945      62,278      59,292

         Median        87         79         316         338 
            Min        25         23         166         163
            Max     6,012      5,945      62,278      59,292

What the!? What just happened? Are those Unity numbers real? 258.5 and respectively a whooping 363.8 times slower than Autofac. By contrast, it makes the usual suspect, Ninject, look like Formula-1 car.

Unfortunately it is true. I was so stupefied by the result that I had to try two different approaches on three different machines. They all posted similar numbers.

Transient Objects - Loaded Containers

You can probably anticipate how quickly I moved on to the next test. Unity has put an okay fight in the transient test, recording numbers on par with the other two usually fast containers: Autofac and Windsor.

Alas, even as the test was in progress (and it took forever and three cookies), I could tell there was no redemption. And sure enough:

:               10k - 1st  10k - 3rd  100k - 1st  100k - 3rd
------------------------------------------------------------
Autofac                73         74         806         645
Castle.Windsor         99         98       1,164         941
Ninject               248        247       2,683       2,338
Spring.Net            230        226       2,346       2,157
StructureMap          118        116         324         312
Unity               6,077      5,918      60,142      58,971

         Median       174        171       1,755       1,549 
            Min        73         74         324         312
            Max     6,077      5,918      60,142      58,971

I'm not going to dwell on those numbers. I don't know how to explain them. I'll just quickly mention Ninject posted only 3.3x in the 10k, but jumped back to 7.5x in the 100k. Spring.Net was the second slowest in the 100k, being 6.9 times slower than StructureMap.

My Take

From seeing these numbers, working with the containers (not just for the purpose of this test), and being somewhat aware of what tells them apart: my favorite is Autofac, which is small, incredibly powerful, and yet very elegant to use. In stark contrast to Spring.Net, I think I spent the least amount of time trying to figure out how to use Autofac - mostly everything seemed obvious. Even the more advanced features, some of which the other containers are either barely now getting them or still dreaming of them, are very straightforward and easy to pick up.

StructureMap is fast (see, I made it italic so it seems fast even as it stands still), but its documentation is a few versions behind, which makes it hard at times to figure out what's the right approach to a given problem. It might be friendlier to introduce in legacy code, given it provides from the get go a Service Locator through its ObjectFactory static root, but given your level of comfort or abstraction needs you might choose to roll your own Service Locator. At any rate, thanks to the Common Service Locator project, you can abstract your DI container quite well. You might find StructureMap suitable if you need a very fast container when loaded with a lot of classes, but if that was a very important criteria I would definitely devise more suitable tests before jumping in.

Castle.Windsor is pretty much the veteran, yet it managed to stay nimble. It has one of the smartest guys in the industry behind it. It also has a few quirks here and there, and it's not the most elegant to work with (name registration and retrieval in particular). I honestly cannot conjure a good reason to pick Windsor over Autofac, unless you are using the rest of the Castle framework.

At the opposite side of the spectrum: Spring.Net. A very powerful framework, especially if you travelled here from the land of Java, its DI/IoC container is just horrible to work with. I've always left it till the very last and punished myself for eating too many brownies by forcing me to complete the wrapper classes for the tests. I am aware of the CodeConfig project and its attempt to bring some sanity and an alternative to the hideous XML configuration files that Spring.Net otherwise employs (don't even try to configure generics). It not only has spotty and slow performance, it's actually quite unpleasant to work with.

Microsoft's Unity is not a bad DI container; it used to be meh... at version 1.2, but starting with 2.0 - which took a very long time to come to light - it's quite nice to work with. That is until you see these numbers and that's seriously off-putting, no matter how many injection and extension features it offers. If you think about picking it up because it integrates well with ASP.Net MVC/Prism/EF/etc, well, so does Autofac.

Finally, Ninject - cool name, awesome website, bad performance. I would use it if the alternative was Spring.Net - but frankly can't see why anyone would voluntarily choose it. Except if you want to be different, which sounds to me like a perfectly good reason.

Alright, we're done. The source code is on github and once you read it you'll probably want to contact me and laugh at something stupid I did in there. Or call me names. Or maybe just to talk about those hard to believe Unity numbers.

Quickly Move Messages In Mail.app

2011-07-16T00:00:00-05:00

This trick uses OS X's superb search in menus feature and since all your mail folders are available as menus entries under Message > Move To, it makes it really quick and easy to perform this operation.

If you use Gmail, remember that labels are available as folder so think of this as a less elegant way to label your messages.

Press Cmd-? or Cmd-Shift-/ to access the Help > Search menu. ¹
Type in the name of your folder.
Press arrow down once for Move To or twice for Copy To and hit return.

1. You might need to enable the shortcut in System Preferences > Keyboard > Keyboard Shortcuts > Application Shortcuts > All Applications > Show Help menu.

Two Significant New Features In Lion

2011-06-14T00:00:00-05:00

A great deal of smart people blogged and spoke on podcasts about all the new features in OS X 10.7 aka Lion.

But there are two features that albeit not life-saving improve the quality of life quite considerably and have been at the forefront of Mac ~~critique~~ switcher pain since... well, OS X.

#1: Resize from Any Edge

Instead of just from the bottom right corner, you can now resize from any edge and any corner too:

The small resize chevron in the bottom right corner is also gone and when combined with the disappearing scrollbars the effect is very pleasant. It's as if windows have finally found their purpose: to display content, not widgets.

I don't need to say more about this one. As an owner of a large LCD that
displays multiple windows at the same time, thank you! Thank you so unpossibly very much.

#2: Merge Folder Content

When you copy or move a folder and the target already contains a folder with the same name, the Finder gives you the option to either replace the target folder or cancel the operation:

Another way to think about it is to view folders being equivalent to files: if you were to move a file and if the target folder already contained a file with the same name, would you expect that the operating system merges the two files? #rationalization

It may make sense to geeks but to most people switching from other OS's, where they have learned to click absently minded on a torrent of badly worded and poorly button-ed dialogs, this was not just merely annoying but down-right harmful. You learn fast though, as the saying goes.

Lion changes this behavior and offers to merge the folder rather than replace its content:

I like the Keep Both button because merge is such a geeky word, but on the other hand I could argue that perhaps both is more ambiguous since according to the dictionary:

both |bōθ|
adjective & pronoun
used to refer to two people or things, regarded and identified together

Using this definition one would expect that both folders would keep their identity, which is obviously not the case. Perhaps it's just a matter of perspective and whether one sees "the thing" as being the folder's content rather than folder itself.

Why Wait So Long?

This is the one question that's been nagging me regarding these two changes, but I think I'll let smarter people answer.

Apple could've done both a good while ago - as far as I can tell there's nothing earth-shattering in Lion without which they couldn't have introduced these features; after all the radars have been filed, and complaints have been voiced for quite a long time.

Or, perhaps, a more interesting question would be "what can we read/infer/speculate from this change?"

I cannot come up with anything that would tie the merge feature into their getting rid of the file system as we know if, and I'm even more puzzled by the resize feature because it seems counter-intuitive to me that Apple would spend time on a feature that's been, aside from having the menu at the top and the window control buttons to the left, a seemingly core visual feature that people identified with OS X (and complained bitterly about).

One more for the road:

256 Terminal Colors

If anything, a 256-color terminal means vim can look as good in the console as MacVim looks.

The screenshot above shows the new terminal theme called "Silver Aerogel". Can't say I'm crazy about it. Black text on gray doesn't offer a very good contrast, but the blur on the window makes it stand out.

Word Count Service For OS X

2011-05-24T00:00:00-05:00

I recently needed a quick way to count the words in selected text on a web page. It seemed like an OS X service would be the perfect ticket, moreso since Snow Leopard shows in the Services menu only those services appropriate for the current context.

Smarter people have tackled this need before, but none of their solutions seemed to work for me, ironically, in Snow Leopard. I suspect that either something changed ever so slightly in AppleScript, or that those scripts were designed to work with a specific utility (ThisService - which by the way it seemed pretty awesome) rather than standalone.

Either way, here's my solution:

on run {input, parameters}
  display alert "Counted " & (count words of (input as string)) & ¬
    " words." message "The selected text had " & (count characters of (input as string)) & " characters."
  return input
end run

Here's a screenshot of it counting the words of this post (how self-referential):

Two quick notes: both John Gruber's and Nima Yousefi's scripts use on process(_str) - I had no luck getting that to work and instead I've decided to go with Automator's suggestion of on run {input, parameters}. As a result of my choice, it seems I was also required to cast the input to a string ((input as string)), otherwise count words of would consistently and unwantedly return 0.

ipy - Completion For Python Interactive

2011-05-12T00:00:00-05:00

Bash has awesome command and file completion. I like that. Ruby people are just spoiled with irb.

Python can do that too.

Completion is handled by the rlcompleter module, aided by the readline library, and it is all set up with two lines of code.

Example:

>>> import rlcompleter, readline
>>> readline.parse_and_bind("tab: complete")
>>> readline. <TAB PRESSED>
readline.__doc__          readline.get_line_buffer(  readline.read_init_file(
readline.__file__         readline.insert_text(      readline.set_completer(
readline.__name__         readline.parse_and_bind(
>>> readline.

Useful, but you want to have that set up automatically (if you can remember or can be bothered to type those liens every time you start python, you have my deepest respects, yet I'm also slightly suspicious of your sanity). You have two options:

The PYTHONSTARTUP environment variable, which if pointing to the name of a readable file the Python commands in that file are executed before the first prompt is displayed in interactive mode.
Running python -i [script], which will cause the interpreter to remain in interactive mode after evaluating the script specified on the command line. You can use it in combination with an alias command.

Start by saving those two lines to a file. I'll call mine ~/.python_init.py:

import rlcompleter, readline
readline.parse_and_bind("tab: complete")

The PYTHONSTARTUP route sets this environment variable in your personal initialization file, .bashrc for the bashful of us (why you want bashrc instead of bash_profile):

export PYTHONSTARTUP=~/.python_init.py

The alias route defines an alias in your personal initialization file for python -i ~/.python_init.py:

alias ipy='python -i ~/.python_init.py'

Now you can start your python interpreter using 50% fewer characters!:

$ ipy
>>> import re
>>> re.<TAB>
re.DEBUG              re.__doc__            re._compile_repl(
re.DOTALL             re.__file__           re._expand(
re.I                  re.__format__(        re._pattern_type(
re.IGNORECASE         re.__getattribute__(  re._pickle(
re.L                  re.__hash__(          re._subx(
re.LOCALE             re.__init__(          re.compile(
...

Either works the same, but I would think that PYTHONSTARTUP, being a set and forget option, would serve you better.

Want to know more? Read the rlcompleter documentation. The readline docs are interesting, too.

Jekyll vs. Hyde - A Comparison Of Two Static Site Generators

2011-05-07T00:00:00-05:00

It seemed fitting, when I first thought about it, that my first blog post should discuss the engine I chose to power my website. After spending days writing this for you, I'm reasonably confident we both would've been better served if I went drinking with my imaginary friends and you just pretended you liked reading it. Anyway...

There are several static site generators/engines you can choose from and your choice is most likely bound by either what features your hosting provider offers or by language preference. On the Ruby planet Jekyll, Webby, and nanoc are the most popular choices.

Robert Rees has an excellent comparison of the Ruby engines, which he calls "Semi-static CMSs". His article was written in June 2009, but you'll find it very much pertinent as two years later these three engines are still firing on all pistons. Largely based on Robert's review, I have selected Jekyll to represent the Ruby side.

With Python you have Hyde and that's pretty much it.

PHP fares about the same: Stacey seems to power quite a good deal of small websites. But on the horizon, on account of his creator, a higher profile engine might end up being Marco Arment's Second Crack, whenever he decides to open source it. I've decided not to include PHP in this comparison because I'm prejudiced to believe that PHP folks are more in the dynamic website camp, given how easy PHP is to get up and go with.

With a static website generator you will spend most of your time producing the content and about the same fidgeting with the layout. Ironically, Jekyll and Hyde lend themselves quite well to a head to head comparison on both features. Not only do both support Markdown and Textile for writing the content, their templating languages are very similar too: Hyde uses the Django templating language, whereas Jekyll uses Liquid, whose inspiration was, surprise!, the Django templating language.

While researching these engines I have put together an example website built using Jekyll and then Hyde on github. While this post does make some references to this sample, you might find it a good counterpart as it covers some of the elements I struggled to figure out.
To get this sample site, execute git clone https://github.com/philipmat/jekyll_vs_hyde in the directory of your choice or simply use github to view the code.

Installation
Structure
Advanced Configuration and Options
The Content and the Templating Language
Miscellaneous Features and Considerations
Conclusion

Installation

The best way to install Jekyll is via RubyGems:

gem install jekyll

This will also install all required dependencies: directory_watcher, liquid, open4, maruku and classifier. More information can be found here, but one interesting, if not ironic, bit is that Jekyll relies on Pygments, a Python project, to provide code syntax highlighting. The installation instructions for Pygments, also detailed on the Jekyll install page, are dependent on your OS, but are usually a variant of:

sudo easy_install Pygments

I recommend that you also generate at this point the stylesheet required by Pygments to style your code:

python -c "from pygments.formatters import HtmlFormatter; print HtmlFormatter().get_style_defs('.highlight')" > highlight.css

Hyde's setup, on the other hand, is more involved. You will start by getting the source code, followed by installing the requirements (Django, pyYAML, and markdown), and probably complete by installing Pygments.

cd /opt/local/src
git clone https://github.com/lakshmivyas/hyde.git
cd hyde && pip install -r requirements.txt

Since you're getting intimate with that terminal anyway, you might want to consider installing markdown2, a faster and more complete implementation of Markdown.

A note on Hyde installation: if you plan to follow along Hyde's own instructions, step #2 of the documentation offers to generate a "basic website" for you, and then processes this skeleton to generate your website. Unfortunately, this "basic website" turns out to be quite comprehensive, and its processing will fail because it requires a great deal of Python modules which were not listed in requirements.txt. At this point you have several options:

You follow along with me or use my truly basic skeleton which I promise it'll work or your money back.
You leave the "basic website" as is and prod it for structure and other tricks. Just be aware that what was generated is anything but basic.
You proceed to discover and install all the requirements and then generate and run the website (hats off to you, you man of genius).

Structure

Both languages tend to follow a similar structure, with Jekyll favoring convention-over-configuration, whereas Hyde relies entirely on configuration. Both will read pretty much all files in the structure and either through process or straight copy will generate your static website in dedicated output folder.

Jekyll

The more prominent elements of a typical Jekyll structure include a _layouts folder that stores the templates to build the site, a _posts folder that contains the blog content, and a _site folder where the generated website will be placed. Also typically present are a configuration file in YAML format, _config.yml, a root index.html, and most likely some sort of media folder for your .css, .js, and images.

./
  _config.yml
  index.html
  _layouts/
      base.html
      post.html
  _posts/
      2011-04-11-lore_ipsum_sit_amet.markdown
      2011-04-21-consectetur_adipisicing_elit.markdown
  media/
      main.css
      main.js
  _site/
      ...

This configuration is merely a preference. Between command line arguments and the _config.yml file, Jekyll allows you to overwrite most of its options, which can be quite extensive.

Within this structure, Jekyll will attempt to process all .html, .markdown, and .textile files that contain a specific header, called YAML Front Matter; it will ignore those files that do not have this header or those that start with an underscore (including folders). All other files will be copied to the _site folder, preserving the folder hierarchy in which they were located.

Jekyll also mandates that the files within the _posts folder are in a YEAR-MONTH-DATE-blog_title.MARKUP format. You have control over the permalinks it will generate, but the names of the files are fixed.

Hyde

Hyde helpfully offers to generate a "basic website" for newcomers - which, as mentioned before, doesn't work, but gives you a basic idea of what Hyde looks like and a settings.py file to work with. You need to comment out a good deal of (not initially useful) lines in settings.py to get you started.

Hyde compartmentalizes its files a bit more than Jekyll:

layout - files used as templates for the content (set using LAYOUT_DIR in settings.py);
content - where the content resides (set with CONTENT_DIR). These files will be processed by Hyde and the templates in layout will be applied according to each file's individual preference, similar but not exactly like Jekyll's YAML Front Matter;
media - images, stylesheets, etc (MEDIA_DIR). These are files that are copied (but may be pre-processed) to the output folder preserving their hierarchical structure;
deploy - the folder to which Hyde outputs the static website (DEPLOY_DIR). Its Jekyll equivalent is the _site folder.

Advanced Configuration and Options

Aside from the caveat regarding Hyde's generated "basic website" and settings.py, neither engine requires any further configuration to start working on your website, but both offer more advanced options, and in both cases most of these options are configurable through settings.py in Hyde's case and _config.yml in Jekyll's.

You get common features such as:

exclude files from processing (e.g. .htaccess) or even entire folders (e.g. drafts);
specify what port to run the development server on;
indicate a different output folder;
alternative rendering engines (e.g. a different Markdown engine);
Growl notifications;
syntax highlighting for code snippets using Pygments.

Some of the other options come from a combination of usage and configuration. For example, Hyde allows you to use a different Markdown engine, markdown2, by using a {% markdown2 %} tag in the content or templates, whereas Jekyll interprets all .markdown files as Markdown syntax but the actual translation engine can be changed in _config.yml using the markdown: [engine] setting, allowing you to switch from the default maruku to RDiscount or another similar engine.

This touches on the first and perhaps largest difference between the two engines: Hyde makes use of template tags for processing, while Jekyll relies on the file extension to determine the type of markup for the content present within it. As a result, Hyde allows you to use multiple engines within the same content file: you can freely mix Markdown and Textile within a single file, if that's something you'd want to do, but at the expense of having your content sprinkled with Django tags.

Hyde also makes heavy use of pre-processors and post-processors when transforming the content and support files and allows you, among others, to minify your JavaScript files (using JSMin) or to optimize them using the Closure Compiler, to code your CSS using SASS or Less CSS or Stylus, or even generate thumbnails for your images.
Of course, you'd have to install the appropriate Python packages for these pre-processors, for example you would need the PIL - Python Imaging Library to make the thumbnails pre-processor work. These pre-processors are enumerated in specific lists within settings.py, e.g:

MEDIA_PROCESSORS = {
    '.css':('hydeengine.media_processors.TemplateProcessor',
            'hydeengine.media_processors.CSSmin',),
    ...

but they are then configured separately using global options specific to each preprocessor (e.g. THUMBNAIL_MAX_WIDTH).

In contrast, Jekyll allows its options to be configured in a hierarchical, and easier to read, fashion:

maruku:
  use_tex:    false
  use_divs:   false

Jekyll's answer to Hyde's pre-processors, to some degree, are a quite extensive collection of plugins, for example Jason Graham's Less CSS Converter or Michael Levin's sitemap.xml generator.

Oh, and cherry on the cake, both have support for Growl - built-in Hyde and through a plugin in Jekyll, although they both generate the files so fast I don't see why you would need it.

The Content and the Templating Language

Hyde uses the Django templating language while Jekyll uses the similar, but less complex/powerful, Liquid.
If you're familiar with Django, you'll be at home with Liquid. If you're very familiar with Django, you'll notice pretty fast that Liquid is missing one very significant feature: blocks. We'll talk about this later - I just wanted to get it out of the way.

A Simple Page

In both cases, you will pair a content file with one or more template files.
Let's start with Jekyll and a very simple "about" page, which will start out as an ./about.markdown file and end up in ./_site/about.html.

---
layout: page
title: About
---

This sample shows how to build a very simple 
[Jekyll](https://github.com/mojombo/jekyll) powered website.

The first section of the about.markdown file, between the --- markers is the YAML Front Matter. Jekyll requires it on all files you'd like to have processed, otherwise the file will be ignored and copied to the output as is. The layout property tells Jekyll what template to use when processing this file, so in this case Jekyll expects there is a ./_layouts/page.html file, which in my example looks something like this:

---
layout: base
---
<div class="content">
<h1>{{ page.title }}</a>

{{ content }}

</div>

Three quick things as this is a comparison more than an introduction to the templating language: the {{ page.title }} gets replaced by the title parameter from about.markdown's YAML Front Matter section; the {{ content }} will contain the processed body of the about.markdown; and, finally, because page.html contains it's own YAML Front Matter section, it will be further processed using ./_layouts/base.html.
base.html will then look very similar:

<html>
<head>
    <title>{{ page.title }} - Jekyll Sample</title>
</head>
<body>
    {% include nav.html %}
    {{ content }}
</body>
</html>

The interesting bit is the {% include nav.html %} which does what you think it does, except it will require that the file is located at ./_includes/nav.html.

Let's look at the Hyde equivalent, but we'll do so in reverse order. I promise you it'll make more sense.
base.html looks just like the Jekyll equivalent:

<html>
<head>
    <title>Jekyll vs. Hyde - A Comparison Of Two Static Site Generators - Hyde Sample</title>
</head>
<body>
    {% include "nav.html" %}
    {% block content %}page content goes here{% endblock%} %}
</body>
</html>

The difference from the Jekyll version is the {% block content %}{% endblock %}.
Next template in chain: page.html.

{% extends "base.html" %}
{% block content %}
<div class="content">
<h1>{{ page.title }}</h1>
{% block article %}{% endblock %}
</div>
{% endblock %}

In Django, blocks in child files overwrite blocks in the parent they inherit from (the {% extends "base.html%} specifies the parent of a template). As such, the content block in the page.html template will replace the similar block in the parent base.html. Finally, let's examine Hyde's version of the "about" page, ./content/about.html:

{% extends "page.html" %}
{% hyde
    title: "About"
    created: 2011-04-26 00:25:00
%}

{% block article %}
{% markdown %}
This sample shows how to build a very simple 
  [Hyde](https://github.com/lakshmivyas/hyde) powered website.
{% endmarkdown %}
{% endblock %}

Notice how the content block of page.html replaces the parent's, and the article block of about.html replaces the similar block defined in page.html. With Hyde a child page can overwrite any of the blocks defined in any its ancestors; contrast with Jekyll where a child can only overwrite the parent's {{ content }} declaration.
Your probably figured out that the {% hyde %} block is the rough equivalent of YAML Front Matter header.

Dynamic Page - The Blog

To create a blog page, that is to generate a dynamic page that essentially includes other pages, Jekyll and Hyde offer similar mechanisms, but they go a different way about it. Jekyll is more opinionated and requires that your blog posts are a) stored within the _posts folder and b) that they follow a very specific naming convention: YEAR-MONTH-DATE-blog_title.MARKUP, where MARKUP would be either markdown or textile. If you do so, you will be able to use a construct in your templates that will iterate over the posts in reverse order by the date within the file name. For example, to output the list of the last 3 posts you would write:

{% for post in site.posts limit:3  %}
    <h1>{{ post.title }}</h1>
    {{ post.content }}
    <hr/>
{% endfor  %}

site.posts (or paginator.posts if you enable pagination in _config.yml) contains all your posts regardless of the structure you have in place inside the _posts folder. I think this is nice because it allows you to organize your posts however you think it's best for you.
Variables added to the YAML Front Matter are available to you through the post variable. I am making use of this feature in the example website to show how one could provide an abstract of a post through the post.abstract atom.

In contrast, Hyde doesn't really care about any particular structure. From within any file you can enumerate its siblings and children and even access its ancestor (parent folder). The templating guide has a good deal more information, but you'll deal with the following variables:

The page variable, which refers to the current page; for example, within index.html, page refers to the index.html file;
page.node refers to the directory in which page exists (./content/ in the case of ./content/index.html);
page.node.walk provides a list of all the children, depth first, of the node, the page itself included as the first node.
page.node.walk_pages provides the same except it excludes the folders.

In practice, you could use Hyde, with its multiple "dynamic" sources, to build any kind of websites (for example a documentation website), whereas Jekyll, with it's single "dynamic" source, seems heavily slanted towards blogging.
However, this comes at a price: for simple types of websites working with Hyde tends to require more effort. Examine the equivalent of the Jekyll index.html posted above:

{% extends "base.html" %}
{% block article %}
    {% for node in page.node.walk %}
    {% if node.name == "posts" %}
        {% for post in node.walk_pages %}
        {% if not post.listing and not post.exclude %}
            <h1>{{ post.title }}</h1>
            {% render_article post %}
        {% endif %}
        {% endfor %}
    {% endif %}
    {% endfor %}
{% endblock %}

Considerably more involved, I would think. Also one of the reasons I started writing this post and providing the sample website: the lack of Hyde documentation beyond laconic enumeration of the features. Some of the documentation mentions inexistent examples, and overall it just makes it unnecessarily hard to figure out even some of the basic details on how you are supposed to be using the engine.

Both engines support tags or categories for posts, and they define them the same way: in the YAML Front Matter for Jekyll and the {% hyde %} block for Hyde. You can access these categories in Jekyll through the site.categories variable, which collects all the categories defined in all posts. Hyde requires that you first configure a site pre-processor which will enable category access through the categories variable.

Part of the templating language, both Jekyll and Hyde allow you to include other files (my example includes a nav.html file) using the {% include FILE %}; less obvious is that Jekyll looks for these files in the _includes folder, whereas Hyde looks for it in the content folder.

Read through Hyde's templating guide to discover more about what the page, page.node, and site variables offer. The Jekyll equivalent can be found on the template data wiki page.

Miscellaneous Features and Considerations

Both engines come with a built-in web server that allows you to quickly check how the generated content will look like. Jekyll's is based on WEBrick and Hyde's on CherryPy, so you will need to have the respective gem or egg installed. One of the features that makes Jekyll stand out is that it combines the server with a directory watcher which will continuously regenerate the files that have changed (run jekyll --auto --server), whereas for Hyde you will need to run the regen command each time you change a file by running python /path/to/hyde.py -g.
In practice, this is an incredible time saver and Jekyll processes only the files that change so it is very, very quick. To give you an idea how fast this process is, consider that I have MacVim set to save the files whenever the window loses focus; I can Cmd-Tab to the browser window and by the time I get to issue the refresh page command, the changed file has already been processed.

To get the same functionality, as I was working through the sample website for Hyde, I ended up backgrounding the webserver (started with python /path/to/hyde.py -w), and executing the regen command each time I would need to see the output. Unfortunately, Hyde seems to reprocess the entire website; it still is fast because the site is small, but it's something to consider when selecting your engine.

When it comes to extensibility, Hyde offers a good deal of built-in site, media, and content pre-processors and the vast library of Django snippets, whereas Jekyll relies on plugins (a good collection of simple plugins written/compiled by Jose Gonzalez) to augment its functionality and to provide supplemental tags for Liquid.

Another factor to consider is whether you have an existing blog. If it runs on Wordpress, you're in luck: both Jekyll and Hyde offer migrators / importers. The edge goes to Jekyll whose migrator list is quite extensive: Drupal, Movable Type, Typo 4, TextPattern, Blogger/Blogspot, and all else failing, you have good ol' CSV.

And finally, deploying your site to its final location is an exercise in simply copying the output folder, perhaps using a tool like rsync to speed it up. Hyde has an interesting project in Clyde which is an web editor for your pages, which would allow you to edit your site untethered from the computer on which you normally edit your site. As of right now, Clyde is not fully complete or stable, but that might be something that would sway your vote, so I thought you should know.

Conclusion

Jekyll is considerably easier to get up and run with, has plenty of examples, a strong presence in github behind it, and where it's lacking it can be extended with plugins. That Liquid is such a close implementation of the Django templating language is both a blessing and a curse: a blessing because you can look at existing Django examples, and a curse because not all those examples will work, and because you will not be able to take advantage of the large library of filters and tags available for Django.

Hyde on the other hand benefits immediately from the power of the Django templating language, the considerate flexibility that comes with it, as well as most of the tags and filters available through the Django snippets library. It takes more effort to get it up and develop your initial wireframe, and there is not much help if you get off the beaten path, but those costs might be offset if you require the more advanced features Hyde provides, including the one possibly killer features in the Clyde web editor, once it's complete and stable.

If you're considering running your website off github's infrastructure, choosing Jekyll is pretty much a no-brainer. If you run it on your own infrastructure, I hope this post and the accompanying sample gave you enough information to make an informed guess about the direction you want to pursue.

I the end I chose Jekyll. Either way, have fun.