Monthly Archives: March 2009

reverse() chicken and egg problem

I wound up in a chicken and egg situation today using Django’s syndication framework and the reverse helper. The problem was that immediately after starting the development server, Django would throw a NoReverseMatch exception on the first client visit, followed by AttributeError on all subsequent visits.

It all started so innocently… I had wanted a set of urls for my application like this:

So I put the following in the application’s urls.py:

# myapp/urls.py
from django.conf.urls.defaults import *
from views import arrivals_list, departures_list
from feeds import LatestArrivals, LatestDepartures


feed_dict = {'a': LatestArrivals, 'd': LatestDepartures}


urlpatterns = patterns('',
    (r'^a/$', arrivals_list, {}, 'arrivals'),
    (r'^d/$', departures_list, {}, 'departures'),
    (r'^(?P<url>[ad])/feed/$', 'django.contrib.syndication.views.feed', {'feed_dict':feed_dict}),
)

That covers my URL wishes, and because I have named the URL patterns I can use that name in templates with the {% url %} template tag and in Python code using the reverse helper.

So naturally the feed classes in feeds.py look like this:

# myapp/feeds.py
from django.contrib.syndication.feeds import Feed
from django.core.urlresolvers import reverse
from django.utils.feedgenerator import Atom1Feed
from models import Tx


class LatestArrivals(Feed):
    """Produces an Atom feed of recent arrival tickets."""
    feed_type = Atom1Feed
    title = 'Arrivals'
    link = reverse('arrivals')
    subtitle = 'Most recent arrivals'

    def items(self):
        return Tx.objects.arrivals()[:10]


class LatestDepartures(Feed):
    """Produces an Atom feed of recent departure tickets."""
    feed_type = Atom1Feed
    title = 'Departures'
    link = reverse('departures')
    subtitle = 'Most recent departures'

    def items(self):
        return Tx.objects.departures()[:10]

Note I used reverse on the link attribute of each class so that I can define the URL in one place, the urls.py module, and a change there will be reflected in the feed’s link too.

But this doesn’t work! When Django imports my urls.py module, it imports LatestDepartures and LatestArrivals, and they in turn use reverse to find the named URL patterns – except those names aren’t defined until after urlpatterns has been defined in urls.py so Django throws an exception and never imports my urls.py module.

You could work around this either by defining your syndication feeds in an entirely different urls.py module. But you can also split up urlpatterns within the same module and import the feed classes after their named URL patterns have been defined.

Here’s the working urls.py module:

from django.conf.urls.defaults import *
from views import arrivals_list, departures_list


urlpatterns = patterns('',
    (r'^a/$', arrivals_list, {}, 'arrivals'),
    (r'^d/$', departures_list, {}, 'departures'),
)


from feeds import LatestArrivals, LatestDepartures
feed_dict = {'a': LatestArrivals, 'd': LatestDepartures}


urlpatterns += patterns('',
    (r'^(?P<url>[ad])/feed/$', 'django.contrib.syndication.views.feed', {'feed_dict':feed_dict}),
)

Using an object for Django’s ChoiceField choices

I had another thought about per-instance choices for forms.ChoiceField. Instead of overriding the __init__ method of your form class, you could use an object with an __iter__ method that returns a fresh iterable each time it is called.

from django import forms


class LetterChoices(object):
    """Return a random list of max_choices letters of the alphabet."""
    def __init__(self, max_choices=3):
        self.max_choices = max_choices

    def __iter__(self):
        import string, random

        return iter((l, l) for l in random.sample(string.ascii_uppercase, self.max_choices))


class LetterForm(forms.Form):
    """Pick a letter from a small, random set."""
    letter = forms.ChoiceField(choices=LetterChoices())

I don’t know if I prefer that style to having a simple function – having to instantiate the class seems wrong to me, I’d much rather use any callable as the choices argument.

Django test database runner as a context manager

In my last post I mentioned it might be an idea to wrap up the Django test database setup / teardown in a context manager for use with Python’s with statement. Here’s my first stab, which seems to work.

from contextlib import contextmanager


@contextmanager
def test_db_connection():
    """A context manager for Django's test runner.

    For Python 2.5 you will need
        from __future__ import with_statement
    """

    from django.conf import settings
    from django.test.utils import setup_test_environment, teardown_test_environment
    from django.db import connection

    setup_test_environment()

    settings.DEBUG = False    
    verbosity = 0
    interactive = False

    old_name = settings.DATABASE_NAME
    connection.creation.create_test_db(verbosity, autoclobber=not interactive)

    yield connection

    connection.creation.destroy_test_db(old_name, verbosity)
    teardown_test_environment()

All of this requires Python 2.5 or later.

So with that snippet you could write a test something like so:

import unittest


class MyTestCase(unittest.TestCase):
    def test_myModelTest(self):
        with test_db_connection():
            from myproject.myapp.models import MyModel

            obj = MyModel()
            obj.save()
            self.assert_(obj.pk)

… and just as with Django’s manage.py test command the objects would be created within the test database then destroyed when the with test_db_connection() block is finished.

Everything’s going to be hunky dory.

Creating a Django test database for unit testing

I needed to run tests involving a Django application but without using the manage.py test management command. So I need my own test suite that sets up the test database and drops it after, leaving my real database untouched.

As of Django 1.0.2 the default behaviour for the test runner is the run_tests function in django.test.simple. Here is the bones of that function with the required setup and teardown calls.

from django.conf import settings
from django.test.utils import setup_test_environment, teardown_test_environment


verbosity = 1
interactive = True

setup_test_environment()
settings.DEBUG = False    
old_name = settings.DATABASE_NAME

from django.db import connection
connection.creation.create_test_db(verbosity, autoclobber=not interactive)

# Here you run tests using the test database and with mock SMTP objects

connection.creation.destroy_test_db(old_name, verbosity)
teardown_test_environment()

Hmmm… Wouldn’t this be a good candidate to be wrapped up for use with Python 2.5’s with statement?

sys.exit(1) versus SystemExit(1)

I used to write Python scripts and have the option parsing go something like this…

if __name__ == "__main__":
    import sys, getopt

    opts, args = getopt.getopt(sys.argv[1:], 'h', ['help'])

    for opt, val in opts:
        if opt in ('-h', '--help'):
            usage(sys.argv)
            sys.exit(1)  # Exit to shell with non-zero result

And then I finally started writing tests for my code, at which point I decided I need to raise SystemExit(1) rather than sys.exit(1) because I imagined Python’s unittest module would get bypassed whenever my code called sys.exit(1).

Except of course I was wrong. sys.exit throws SystemExit in turn, so it comes to the same thing from the point of view of unittest. Failing to read documentation is a very bad habit.

But I prefer throwing the exception myself. You don’t have to import sys if you don’t need it, and it feels prettier (if I had more Python experience I might say more Python-ic).

I used to smoke, drink and dance the hoochie-coo too.

Per-instance choices for Django’s forms.ChoiceField

I keep forgetting the details of how one customizes the choices in a forms.ChoiceField per instance. The forms.ChoiceField documentation says the required argument has to be an iterable but then moves straight to the next section.

Fortunately this was covered long ago by James Bennett when Django’s newforms module was introduced. See Getting dynamic model choices in newforms on Django Snippets.

The following form example has a field for picking a letter of the alphabet (works for Django 1.0). The choices are limited to 3 letters only, picked at random and different for each form instance:

from django import forms


def letter_choices(max_choices=3):
    """Return a random list of max_choices letters of the alphabet."""
    import string, random

    for l in random.sample(string.ascii_uppercase, max_choices):
        yield (l, l)


class LetterForm(forms.Form):
    """Pick a letter from a small, random set."""
    letter = forms.ChoiceField(choices=letter_choices())

    def __init__(self, *args, **kwargs):
        super(LetterForm, self).__init__(*args, **kwargs)
        self.fields['letter'].choices = letter_choices()

So that works. The LetterForm class uses the helper function letter_choices to provide the random choices, which actually returns a generator object rather than a list or tuple of choice pairs. I am relying on Django’s base ChoiceField class calling list() on the choices when each form is instantiated, so having letter_choices return a generator is hunky dory.

>>> f1 = LetterForm()
>>> f1['letter']
>>> print f1['letter']
<select name="letter" id="id_letter">
<option value="R">R</option>
<option value="N">N</option>
<option value="U">U</option>
</select>
>>> f2 = LetterForm()
>>> print f2['letter']
<select name="letter" id="id_letter">
<option value="O">O</option>
<option value="N">N</option>
<option value="T">T</option>
</select>
>>> 

Now the only thing is… this example is not practical. Using genuinely random choices means that the valid choices on the form submitted by the user will be different to the valid choices on the form used to validate the user input on the next request, and this will likely raise a ValidationError.

Django feature suggestion: allow choices to be any iterable or callable, calling it as appropriate when instantiating the field. If it is callable you could pass a function which returns an iterable at that point, which would save one having to write an __init__ method for the form sub-class.

First steps with South

My first time using a schema evolution tool for a Django project, and I like it. I chose South because it had a clear path for devolving a schema and had a database-independent API (I’ve been testing Postgres with some projects that currently use a MySQL database since we have a Trac installation that uses Postgres anyway).

I have a simple application to store serial numbers and purchasing information for software. It has a License model, but I wanted to add a comment on each license. Following the instructions for converting an existing app on the South wiki, here’s what I did to convert my software license application.

First I added south to INSTALLED_APPS in settings.py and ran manage.py syncdb to install the South application tables for the first time. This was before using South to actually manage any of the evolutions.

% ./manage.py syncdb
Syncing...
Creating table south_migrationhistory

Synced:
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.sessions
 > django.contrib.sites
 > django.contrib.admin
 > myproject.software
 > south

Not synced (use migrations):
 - 
(use ./manage.py migrate to migrate these)

Then I created an initial migration for my application. This gives me a migration that matches the existing database schema.

% ./manage.py startmigration software --initial
Creating migrations directory at '/Users/david/myproject/../myproject/software/migrations'...
Creating __init__.py in '/Users/david/myproject/../myproject/software/migrations'...
Created 0001_initial.py.

The next step was to bring this application under South’s control. You have to pretend to apply the initial migration (because it already exists in the database), which is done using South’s --fake switch. I discovered a problem with my use of _ in my 0001_initial.py migration: the easiest fix was to import it explicitly within the migration (this fix was necessary for the second migration too).

from south.db import db
from django.db import models
from myproject.software.models import *
from django.utils.translation import gettext_lazy as _

Then I was ready to bring my application under South’s control.

% ./manage.py migrate software 0001 --fake
 - Soft matched migration 0001 to 0001_initial.
Running migrations for software:
 - Migrating forwards to 0001_initial.
 > software: 0001_initial
   (faked)

Now I was ready to alter my model definitions and get South to do the tedious work of updating the database schema. I added a new Note model in the application’s models.py and added a new field to the existing License model.

class License(models.Model):
    """A license for a software title."""
    ...
    department = models.CharField(_("Department"), max_length=100, blank=True)
    ...


class Note(models.Model):
    """A note for a license."""
    license = models.ForeignKey(License, editable=False)
    author = models.CharField(_("author"), max_length=100, editable=False)
    created = models.DateTimeField(auto_now_add=True)
    note = models.TextField(_("note text"))

    def __unicode__(self):
        return self.note

    class Meta:
        ordering = ['-created']

With those changes I used the startmigration command to generate a migration for the new model and fields.

% ./manage.py startmigration software notes_department --model Note --add-field License.department
Created 0002_notes_department.py.

(Afterwards I edited the migration to import _ as noted above.)

The final step was to use South to apply this migration (and any others) to the database.

% ./manage.py migrate software 
Running migrations for software:
 - Migrating forwards to 0002_notes_department.
 > software: 0002_notes_department
   = ALTER TABLE "software_license" ADD COLUMN "department" varchar(100) NOT NULL; []
   = CREATE TABLE "software_note" ("id" serial NOT NULL PRIMARY KEY, "license_id" integer NOT NULL, "author" varchar(100) NOT NULL, "created" timestamp with time zone NOT NULL, "note" text NOT NULL); []
   = ALTER TABLE "software_note" ADD CONSTRAINT "license_id_refs_id_61c4291d" FOREIGN KEY ("license_id") REFERENCES "software_license" ("id") DEFERRABLE INITIALLY DEFERRED; []
   = CREATE INDEX "software_note_license_id" ON "software_note" ("license_id"); []
 - Sending post_syncdb signal for software: ['Note']
 - Loading initial data for software.

Sweet!