PhilipMat

django-permissivecsrf

Mon, Jan 14 2013

I bet you're here because you've run into the puzzling Django error

CSRF verification failed. Request aborted.

Reason given for failure:
    Referer checking failed - http://<domain>/ does not match https://<domain>/.

I'll double down that this happens when you try to make your login form post to HTTPS, perhaps to https://<domain>/accounts/login/.

If I'm mostly right, you've run into the problem I've solved. If I'm not right and you somehow ended up on this page, I'd love to hear about it.

Short version

In the CSRF middleware, Django does an extra check when a request comes over HTTPS to ensure it comes from the same site (same origin check).

If any portion of your site benefits from HTTPS, you probably should run your entire site over HTTPS. You can use django-sslify to force your website to operate in HTTPS mode all the time.

Furthermore, if you want to disable CSRF checking for your own views, there are other methods you can use, for example the @csrf_exempt decorator.

However, if the views are not under your control and you are comfortable n trading some security for not really that much convenience, you can use django-permissivecsrf to work around this error.

Use the instructions in the README file on GitHub or on PyPI to get this project up an running. Mostly it consists of installing django-permissivecsrf and adding 'permissivecsrf.middleware.PermissiveCSRFMiddleware' to your MIDDLEWARE_CLASSES entry.

Enjoy.

Long version

The gist of why this happens is explained in point #4 of the How it works section of the Django documentation on Cross Site Request Forgery (emphasis mine):

4) In addition, for HTTPS requests, strict referer checking is done by CsrfViewMiddleware. This is necessary to address a Man-In-The-Middle attack that is possible under HTTPS when using a session independent nonce, due to the fact that HTTP 'Set-Cookie' headers are (unfortunately) accepted by clients that are talking to a site under HTTPS. (Referer checking is not done for HTTP requests because the presence of the Referer header is not reliable enough under HTTP.)

In other words, because the HTTPS headers are encrypted, the HTTP-Referer header is resilient against MITM attacks, so it can be safely used to check and make sure the CSRF cookie is originated by the same site that served the page and that the referring page has also been served over HTTPS, which means that page has also been protected against header injections.

The same check could be made on HTTP calls as well, but since HTTP headers are not encrypted, they could be easily faked and thus the check would be a useless placebo.

This explanation is also present, in comment form, in this f92a21daa7 commit by spookylukey aka Luke Plant, and further detailed by him in a reply to a complaint about the strictness of CSRF Referer check on the django-developers maillist.

How django-permissivecsrf works

The Django CSRF middleware performs an extra-check if the request is over HTTPS to ensure that the request came from the same site, i.e. that the referrer (HTTP-Referer header) matches the current site, and that the schema of the referrer is also HTTPS.

In other words, in ensures that the call to https://example.com/account/login came from another page of https://example.com/. As such, if you put your login form on your non-secure homepage, http://example.com/, but use a secure target for your form's action attribute, <form action="https://example.com/account/login" method="POST">, Django's check will fail because::

'http://example.com/' != ('https://%s/' % request.get_host())

However, Django will not perform the CSRF check at all if the request object has an attribute _dont_enforce_csrf_checks set to True. That's what PermissiveCSRF relies on: if the request came from the same site, regardless the schema, it sets _dont_enforce_csrf_checks to True, thus telling the Django CSRF middleware to skip the CSRF check for that request.

This only happens if:

  • DEBUG == True. Your production server should always be HTTPS;
  • The HTTP-Referer header is present;
  • The request is for an HTTPS URL (i.e. request.is_secure() == True);
  • and the referrer uses HTTP.

In all other cases it defers to Django for normal processing.

Bottom line

There's only one thing to take away from all this: in production use HTTPS (see django-sslify). Period.

Miniature Minifier Makefile

Sun, Sep 30 2012

A solution that uses only make and curl:

# c/o: http://wonko.com/post/simple-makefile-to-minify-css-and-js
# Patterns matching CSS files that should be minified. Files with a -min.css
# suffix will be ignored.
CSS_FILES = $(filter-out %.min.css,$(wildcard \
    css/*.css \
    css/**/*.css \
))

# Patterns matching JS files that should be minified. Files with a -min.js
# suffix will be ignored.
JS_FILES = $(filter-out %.min.js,$(wildcard \
    js/*.js \
    js/**/*.js \
))

# Commands
CSS_MINIFIER = curl -X POST -s \
    --data-urlencode "input@CSS_TMP" \
    http://www.cssminifier.com/raw

JS_MINIFIER = curl -s -X POST \
    --data-urlencode "js_code@JS_TMP" \
    http://marijnhaverbeke.nl//uglifyjs 

CSS_MINIFIED = $(CSS_FILES:.css=.min.css)
JS_MINIFIED = $(JS_FILES:.js=.min.js)

# target: minify - Minifies CSS and JS.
minify: minify-css minify-js

# target: minify-css - Minifies CSS.
minify-css: $(CSS_FILES) $(CSS_MINIFIED)

# target: minify-js - Minifies JS.
minify-js: $(JS_FILES) $(JS_MINIFIED)

%.min.css: %.css
    @echo '  Minifying $< ==> $@'
    $(subst CSS_TMP,$(<),$(CSS_MINIFIER)) > $@
    @echo

%.min.js: %.js
    @echo '  Minifying $< ==> $@'
    $(subst JS_TMP,$(<),$(JS_MINIFIER)) > $@
    @echo

# target: clean - Removes minified CSS and JS files.
clean:
    rm -f $(CSS_MINIFIED) $(JS_MINIFIED)


# target: help - Displays help.
help:
    @egrep "^# target:" Makefile

You can also find the latest version on GitHub part of my LeaseMilesTracker project.

How Does It Work?

Surprisingly simple.

  1. We make a list of all CSS files, filtering out the *.min.css files;
  2. we ask cssminifier.com to make us a minified version;
  3. then we save the latter to a file ending in min.css.

Same with the JavaScript files, but using the service provided by UglifyJS

The obvious targets are minify-css, minify-js, and minify which is dependent on both. The less obvious targets are %.min.css and %.min.js whose purpose is to cause the re-minification of the CSS or JS files, if the normal files are newer than the minified version.

Finally, clean will remove all the minified files, that is all *.min.css and *.min.js files.

Why Would I Use This?

Ryan Grove's "Simple makefile to minify CSS and JS", the original inspiration for my Makefile, is an excellent solution if you're using the YUI compressor, but YUIc requires Java (and so does Google's closure-compiler) and I was interesting in having this not only work without Java, but more important work on a system that would require no special tools beyond what you'd find standard on most *nix distros.
After all this is 2012 and we're supposed to be using RESTful APIs and what not.

Oh, and also Mountain Lion doesn't come with Java installed.

Kendo Grid Select Editor

Tue, Sep 11 2012

I'm going to assume you're reading this post because you'd like to find out how to use a regular HTML SELECT as an editor in a Kendo grid, and that you already know why you want to do that - you just had a hard time figuring out how to.

If you just want to see the code and don't care about the explanation, here is a jsFiddle to play with, also a gist for your forking pleasure.

If you wonder why you'd want to do it, read the Why would I want to do that? section that come back for the how.

I see two primary scenarios for using a SELECT aka drop-down list.

Simple Field - Extra Info

This would be the case where, for example, your model contains a field to store the user login (e.g. foo), but when displaying the drop-down to select the user you'd like to also list their name (e.g. foo - Foo Frye).

Let us assume you have the mapping of login to full name in an array of objects, and for genericality's sake let's assume it has a format similar to the following:

var nameList = [
  { Login: 'foo', Description: 'foo - Foo Frye' },
  ...
]

When you create your grid and specify the columns, you have the chance to pass in a function that would create a custom editor instead of the text box the grid gives you by default:

$('#grid').kendoGrid({
  ...
  dataSource: {
    data: [
      { ..., Login: 'foo', ... }
    ]
  }
  columns: [
    ...
    {
      field: 'Login',
      editor: function(container, options) {
        var s = $('<select ' +
                  ' data-bind="source: listSource.list, ' +
                    'value: ' + options.field + '" ' + 
                  ' data-text-field="Description"' + 
                  ' data-value-field="Login"' + 
                  '/>')
        options.model.listSource = kendo.observable({list: nameList})
        s.appendTo(container)
      }
    }
    ...

What's going on here?

  1. We create a drop-down $('<select/>')
  2. Using the data-bind attribute we tell Kendo to use a specific source for the drop-down and to write the selected value to the underlying model's field (here options.field == 'Login').
  3. We tell it to use the Description property of each element in the source for the text portion, and the Login field for the value. This would be as if we had an <option value="foo">foo - Foo Frye</option>.
  4. We create a kendo.observable containing the list of names.
  5. We append the HTML select to the container, which, if you're curious, is the actual grid cell (TD).

There are only two tricky parts here and they both gravitate towards the use of the source directive: options.model is the object from your data source that correspond to the grid row; the source: listSource.list will look for a property called listSource on the grid row/object/model and expect it to
a) be a kendo.observable and
b) contain a sub-object named list.

Complex Field

What if your Login field was a complex object, maybe similar to one of the entries in nameList? If this was the case, how would you go about displaying it in a SELECT, but have it write back to your field the same complex object, not just a simple string.

The code looks similar, in a way even simpler:

$('#grid').kendoGrid({
  ...
  dataSource: {
    data: [
      { Complex: { Login: 'foo', Description: 'foo - Foo Frye' } },
    ]
  }
  columns: [
    ...
    {
      field: 'Complex',
      editor: function(container, options) {
        var s = $('<select ' +
                  ' data-bind="source: listSource.list, ' +
                    'value: ' + options.field + '" ' + 
                  ' data-text-field="Description"' + 
                  '/>')
        options.model.listSource = kendo.observable({list: nameList})
        s.appendTo(container)
      }
    }
    ...

The most significant difference from the simple example is the absence of the data-text-value attribute. This causes the entire underlying record to be written back into the Complex model field.

That's it. If there was a simpler way to do this, I couldn't find it.

Why Would I Want to Do This?

If you use Telerik's excellent Kendo stack, you already have access to a nice drop-down editor that is part of the suite. The Kendo UI demos even show you how to use the Kendo DropDownList as a custom editor, and use it you should for it has a plethora of features.

For all its niceness, the DropDownList is built upon styled UL/LI elements, and that means it is missing a few features that a normal drop-down would have, chiefly the ability to use keyboard navigation and the two significant advantages that come with it: using keys to trigger and navigate the list (Alt-Down on Windows and Spacebar on Mac), and type-to-select when focused (for example typing Tex or TT to select Texas in a list of states).

That and the fact that it doesn't play 100% nice with Twitter Bootstrap, in the sense that the styles don't quite match. Oh, and on a mobile browser you don't get the native picker.

Fortunately, the Kendo grid allows you to specify a custom editor for a column; unfortunately, the Kendo stack tends to make use of and wrap your SELECT in a kendo.observable type of object. I suspect that this is the main reason why examples of using normal HTML inputs with the Kendo grid tend to be scarce on the web.

Templates for GoodNotes

Sat, Apr 28 2012

Recently I've been testing a few apps for taking notes and I have settled on GoodNotes.
It's a really good looking app and the developers are dedicated to it. How dedicated? They had a retina version out within days after the iPad 3 release.

After a few weeks of giving up paper and exclusively using the app, I've created a few templates for the type of notes I find myself taking most frequently.

While these templates have been build with GoodNotes in mind, you can take the SVG files and with very few tweaks target them towards other note taking apps.

Read on for an explanation of the design and thoughts that went into each template, how to import them into GoodNotes, and how you can help improve them, or jump to the end to see where you can get them.

The Templates

The template are geared towards writing and designing rather than viewing (read-only) so they account for the space occupied by UI elements. They also favor unobtrusive guiding elements so that they provide support for writing but not distract from what is being written or drawn.

960 Grid System

Two templates support the 960 Grid system (also employed by my favorite: Twitter Bootstrap) in 12 column format:

  • 960px Portrait - has longer columns for designing in portrait mode;
    960px portrait

  • 960px Landscape - has wider columns (33% wider), better suited for drawing in landscape.
    960px landscape

These two has been inspired by hellopanos's templates for Penultimate, which I found too dark and thus distracting. Since the iPad has a great screen a lighter color is very visible and let's the design take center stage while still providing alignment guidance. Space between columns is proportional to the width of the columns.

Since the design is performed within the constraint of the columns, I have chosen to indicate the outer/side border of the 960px grid with two faint lines. This way they are not distracting yet provide the visual indication of what the boundaries are.

iPad

Three iPad templates present an iPad that is scaled about 73% from the actual size (in pixels). FWIW, I even got the corner radius accurate and then I realized it doesn't really matter that much.

iPad template

The screen portion displays the status bar (20px scaled by 0.73) and if you squint hard enough you'll see marker lines for the 44pt navigation and 49pt tab bars. They are very faint so that they don't distract from designing your interface, but still present if you want to make use of them.

Meeting Notes

Inspired by a Penultimate template, it contains a top section to record the date/time and place of the meeting, the subject/topic/reason for meeting, and the participants. This section doesn't offer a great deal of space, but if you need more than that you're probably doing something wrong or at least ineffectual.

The template devotes most of its body to the meeting notes and is concluded by a checklist-like section to record action items.

Meeting Notes template

I gave each action item what I thought to be enough space, but you might want to consider writing in those sections using the GoodNotes magnifier mode.

Each section contains almost transparent text to indicate its purpose. I've thought it would be nice to provide such a hint, but I made it light enough so that it allows to write on top of it.

Task List

The task list template is geared toward being able to capture tasks quickly. That means that the space for each of the 14 lines is oversized so that you don't have to use the magnifier mode.

Task List template

The lines are a bit darker than the other templates as they intend to provide more guidance for writing. There is white space at the top of the templates, enough to write a title for the task list, but not so much that it's distracting it if you don't.

If you look at the image by itself you might think there's too much space at the bottom, but remember that GoodNotes has an area at the bottom where it displays the toolbar for bringing up the palm rest or magnifier. The template accounts for that.

ToDo List

The to-do template also has 14 rows of checkboxes but in two-column format as it's designed for shorter entries. You can use the magnifier mode if you need to capture more details.

ToDo template

Since to-do lists are more likely to have a title, the template provides more emphasis by separating the title area from the checklist through a darker line, while at the same time the individual to-do entries have lighter supporting lines to allow more focus on the written text.

How to Import the Templates

Using the images in GoodNotes is simple. Get the templates onto the device, then from within a notebook click the + button in the toolbar, click Import, select your image and go to town on it.

However, if you find yourself using one or more of the templates with some frequency, it might be worth going through the following five-step process to make them part of the template gallery.

  1. Click on the + button in the toolbar and choose Import
    Import image into GoodNotes

  2. Select your image from one of the image sources, depending on how you made available the templates to GoodNotes (in this case, I copied them to the app using iTunes).
    Select the template to import

  3. Repeat step 1: touch the + in the toolbar, but select Other Templates this time. You will be presented with the template selection dialog.
    Go to Template dialog

  4. Click the + button in the top left corner of this dialog, then select Current Template.
    Add current template to template list

  5. Give your template a recognizable name. Optionally, touch Edit then use the sizing chevron to move your template(s) in a better position.
    Name template and choose position

Tech Notes

One of the decisions I made early on was to optimize these templates towards input (writing/drawing), which meant having to account for GoodNotes's toolbar and palm-rest/magnifier. I could've started with a 768 x 1024 image to mimic the iPad's screen, but I've noticed that GoodNotes moves the template down so that its top is right below the toolbar, effectively clipping its bottom part and thus reducing the effective area.

For this reason, the templates are actually 768 x 972, given that GoodNotes's toolbar is 52 pixels high. The chevron that triggers the palm-rest/magnifier is 35 pixels high, so I designed all templates so that their bottom 40 pixels remains clear. I could've cropped the image and target 768 x 937, but I thought it looked better if the "paper" continued all the way to the bottom of the screen.

To create each template, I have started with a handcrafted SVG file and tweaked it for hours (mostly because I had no familiarity with SVG) until I got the desired result.

I then exported the images into PNG using OS X's qlmanage and ImageMagick's convert and loaded them into GoodNotes to check out the result and make sure the templates were comfortable to use.

Converting to PNG could've (should've) been a simple process because ImageMagick can convert SVG images, alas the feature-set support is limited, and occasionally, incorrectly implemented.

It would've been a great deal easier to create them in an image editor and, indeed, the first attempt was in Pixelmator and it took about 1/3 to half of the time it took to create the SVGs.

However, since the SVG files are text they are effectively the source code for the templates, something I can certainly appreciate as a software engineer.

As a bonus, you need but a text editor to quickly change their dimensions and create templates for another note taking app (e.g. Penultimate).

Where to Get Them

The templates are on github and the SVG files are within the source folder of that repository. If you're interested only in the PNG files, I've uploaded a zip file that contains them all.

The templates, both PNG and SVG files, are released under a Creative Commons BY-NC-SA license, which means you are free to share them and change (remix) them, but you cannot sell them, you must give proper attribution, and should you alter them you may distribute the resulting work only under the same license (or similar).

That being said, have fun hacking them. Looking forward to it.

Asymmetric Sparkbars

Thu, Mar 22 2012

I am creating some sparkbars, sparkline's poorer cousin, for a project I'll talk about soon, and I wanted to represent in a single chart both additions and subtractions.

As it happens, there's an entire range of Unicode characters devoted to blocks: U+2580 to U+259F, of which U+2581, "Lower one eighth block", through U+2588, "Full block", seem perfect for rendering text sparklines.

The smart way to handle this task would've been to use jQuery.sparkline, which makes use of the canvas element to draw some pretty impressive charts, but I stubbornly decided to stick with text.

I am starting with two data sets, one representing tasks completed per day - progress, the other being tasks created per day, for the same interval, and I will use a spark() function provided by node-textspark to convert each set into discrete values, returned to me as Unicode characters.

My starting point is these two sparklines:

Progress and regress

I'd love for the top of the second set to be glued to the bottom of the first set and thus give a good visual of what happened for each day. For that to happen I'll be using a different color and I will make use of negative space to represent the data points.

First I'll set the background color of the second series to red and the foreground color to white, effectively transforming the negative space, previously white - now red, into sparkbars that seem to hang from the top.

Progress and regress - in complimenting colors

Unfortunately, the block characters stop at the baseline and thus the space between the baseline and the beard line (not making this up) is visible and colored red (from the background).

To make that line go away, I'll set the height on the div containing the second series to the same height as the font, then set the overflow property to hidden so that it clips the bottom part, below the baseline.

Progress and regress - clipping below the baseline

Lastly, I'll set a negative top margin for the second div so that it gets closer to top series:

Progress and regress - in complimenting colors

I chose to leave a pixel between the series as I think it looks just a tad better than if they had no space between them:

Progress and regress - in complimenting colors

The ending HTML is deceptively simple and scales well when the browser zooms in.

<!DOCTYPE html>
<html>
    <head>
        <style type="text/css">
            .inverse .inner { background-color: red; color: white; }
            .sparkline-group { font: 'Courier New'; font-size: 18px; }
            .sparkline-group .sparkline.inverse {
                margin-top: -7px;
                height: 18px;
                overflow: hidden;
            }
        </style>
    </head>
    <body>
        <div class="sparkline-group">
            <div class="sparkline"><span id="spark1" class="inner"></span></div>
            <div class="sparkline inverse"><span id="spark2" class="inner"></span></div>
        </div> 
        
        <script src="spark.js" type="text/javascript"></script>
        <script type="text/javascript">
            var spark1 = [1,2,5,9,0, 5,4,8,9,1, 3,0,6,4,9, 1,2,3,9,5], 
                spark2 = [9,9,8,7,5, 4,8,9,6,6, 4,9,4,7,9, 1,6,7,4,9];
            document.getElementById('spark1').innerHTML = spark(spark1);
            document.getElementById('spark2').innerText = spark(spark2);
        </script>
    </body>
</html>

That's it.

Blog Archive →