Monthly Archives: June 2009

Casio versus Canon versus Fuji

I bought me a Casio Exilim EX-Z29 (lipgloss pink) to replace a four-year-old Canon Digital Ixus 50. The Canon in turn replaced a Fuji FinePix F40 or something.

That old Fuji was a fantastic little camera, given me by my brother and sister about eight years ago. Its great strength was getting good shots in all kinds of low-light party conditions, often without a flash, but it eventually limped along and would make a little scrunch noise when the lens cover retracted. Time for a new Canon!

The Canon was a good camera, way over my intended budget (it cost about £220 new) but I reckoned it would be worth it. However I was always disappointed with it because it couldn’t handle low-light situations without the power of a thousand suns from the flash, and I kind of like to take pics at parties without a flash going off to announce I am taking a pic. And often you want a picture with natural light; maybe it is indoors and there’s already a lot of ambient light or something, but the flipping Canon would always insist on using the flash as soon as it determined the sun had passed midday. If I disabled the flash I would usually end up with severely out-of-focus or blurred pics. Not a Fuji!

So eventually the lens on the Canon got scuffed in the bottom left corner, leaving my pics with an irritating blurry halo like I hadn’t quite finished applying the vaseline. Time for a new camera!

The Casio was on offer on Amazon for £105 (it was another £30 if you didn’t want lipgloss pink – I fancied the idea of a pink camera) and it seems good. Took a bunch of pics which have all come out well, including some where I deliberately disabled the flash. I am happier with the Casio for point-and-click duty than I ever was with the Canon.

The difference between a relatively expensive Canon and a cheap-ish Casio is automatic picture orientation. The Canon would tag portrait shots as revolved, the little Casio does not. Minor bummer.

The lipgloss pink is not too blatant, I am pleased I went for it.

Tuples to dicts, toot sweet!

Looking through Trac‘s search internals I came across a chunk where a list of tuples is converted to a list of dictionaries for the convenience of the template engine. Each tuple has five fields: href, title, date, author and excerpt.

for idx, result in enumerate(results):
    results[idx] = {'href': result[0], 'title': result[1],
                    'date': format_datetime(result[2]),
                    'author': result[3], 'excerpt': result[4]}

This allows the template author to use nice names for the fields in a row, like ${result.href} etc. Looking at this reminded me of another approach that uses list comprehension, zip and dict.

keys = ('href', 'title', 'date', 'author', 'excerpt')
results = [dict(zip(keys, row)) for row in results]
for row in results:
    row['date'] = format_datetime(row['date'])

The second line in this snippet is where the list of dictionaries is created, but one still has to go back and format the datetime values (the third and fourth lines). There’s no advantage in speed (the majority of the execution time is spent in format_datetime) but I like it a little better.

Maybe if Trac used the second approach I would like Trac a little better too.

Installers: correlate of hate

The degree of hatefulness for a Mac installer often correlates to the degree of hatefulness for the software it installs. Adobe Creative Suite is this rule’s exemplar. And if you have a Blackberry mobile phone you may have had fun trying to get the PocketMac for Blackberry software working.

The PocketMac installation experience is pretty poor. Not surprisingly it uses Mindvision’s Installer Vise. But the developers have really pushed the boat out and gone to the trouble of using an Apple installer package icon for the install application. Do not be deceived! It is an application, not a package.

The PocketMac installer

The PocketMac installer

And if you succeed in installing the sync application, don’t expect it to work unless you happen to have learnt that you should only connect the Blackberry to the USB cable after you have opened the sync application.

Fiery RIPs, old-fashioned installation hate

Whether you buy from Canon or Xerox (or Epson or Konica Minolta, etc.), if you are buying a printer for heavy-duty use in a Mac design studio then it will probably have a Fiery RIP1 manufactured by Electronics For Imaging.

There are lots of things to hate about Fiery RIPs.

But right now I am going to hate their backward attitude to Mac driver installation. It is the year 2009 and this lot have yet to supply drivers as anything other than Installer Vise applications. So that means no simple command-line roll-outs with Apple Remote Desktop. That means no way of looking at the installation manifest ahead of installation. No way of having any confidence that the installer won’t screw everything up.

To compound their sins, EFI like to give their installer applications a very pretty icon that looks just like a stylized Apple installer package. It is a very pretty icon of a box; a box full of hate.

Purple box of Installer Vise hate

Purple box of Installer Vise hate


  1. RIP stands for Raster Image Processor. Which leads to quite enjoyable conversations about ripping through pages. 

os.walk and UnicodeDecodeError

My Python program was raising a UnicodeDecodeError when using os.walk to traverse a directory containing files and folders with UTF-8 encoded names. What had me baffled was the exact same program was working perfectly on the exact same hardware just minutes earlier.

Turns out the difference was between me starting my program as root from an interactive bash shell, versus the program getting started as part of the boot sequence by init (on a Debian Lenny system). When started interactively, the locale was set to en_GB.UTF-8 and so names on the filesystem were assumed to be UTF-8 encoded. When started by init, the locale was set to ASCII.

The fix, as described in this article Python: how is sys.stdout.encoding chosen?, was to wrap my program in a script that set the LC_CTYPE environment variable.

#!/bin/sh
export LC_CTYPE='en_GB.UTF-8'
/path/to/program.py

Django and time zone-aware date fields

Django makes it inordinately complicated to support time zone-aware dates and times because it has so far simply ignored the problem (so far being Django 1.0.2).

This is understandable given the database-agnostic nature of the Django ORM: although PostgreSQL 8.3 supports a datetime type which is time zone-aware, MySQL 5.1 does not (I have no idea what SQLite does about time zones). By ignoring time zones, Django works with the lowest common denominator.

Given time zone support in Postgres, there is a chunk of work to write a variation of models.DateTimeField which can handle time zone-wise datetimes. Python 2.5 does not help things – Python’s native datetime module is similarly agnostic about time zones, the standard library does not include a module for handling wise datetimes.

(If regular datetime instances are naive then datetime instances that honour time zones are wise.)

Django does make it pretty easy to write a custom field class, which means it shouldn’t be too difficult to write a custom datetime field class that is time zone-wise. As ever it is the Django project’s regard for documentation that transforms that which is possible into that which is practical.

Given your backend database has a time zone-wise datetime type (i.e. PostgreSQL), what input values does one need to handle in a time zone-wise custom field class?

  • value set to None
  • value set to a naive datetime instance
  • value set to a wise datetime instance
  • value set to a naive datetime string
  • value set to a wise datetime string

Now the essence of a custom field in Django is two methods: to_python and get_db_prep_value. If the custom field defines

__metaclass__ = models.SubfieldBase

then the to_python method will be called any time a value is assigned to the field, and we can make sure that a suitable type is returned before the model object is saved. Because Postgres supports time zone-wise datetimes and if we take care to return a wise datetime instance we can ignore get_db_prep_value.

When Django reads a record from the database it strips the time zone information, effectively giving your custom field a naive datetime string that belongs to the same time zone as the database connection object. (At least this seems to be true for Postgres and the psycopg2 adaptor.) And since the database connection sets the time zone to be the same as set by settings.TIME_ZONE your custom class needs to treat any naive datetime strings as belonging to the time zone set with settings.TIME_ZONE.

So this leads to the important behaviour for a time zone-wise DateTimeField sub-class: always convert naive datetimes to the time zone set in settings.TIME_ZONE.

For convenience my custom field class, the TZDateTimeField, returns a sub-class of Python’s datetime which has an extra method that converts the datetime to the zone defined by the project’s time zone. Therefore whether the field value has been set from a naive or wise datetime instance, or a naive or wise date string you will end up with a time zone-wise value and you can get the value converted to the project’s time zone. This extra method is intended for use in a Django template.

What I was hoping was that the backend would store the datetime as a datetime in an arbitrary zone, potentially a different time zone from one record to the next for the same field. That behaviour would allow one to infer that one datetime value was created in this time zone while another datetime value was created in that time zone. Instead all datetime values are effectively normalized to your Django project’s time zone.

So here is an example of a model class that uses my time zone-aware datetime field. It ought to work just like a regular DateTimeField but always stores a time zone-aware datetime instance:

from django.db import models
from timezones.fields import TZDateTimeField
from datetime import datetime


class Article(models.Model):
    pub_date = TZDateTimeField(default=datetime.now)

And below is my custom field definition, which has a dependency on the pytz module to handle all the difficult stuff. You can grab the complete module over here, including tests in doctest format. The tests are intended to be run by Django’s manage.py test management command, and so one needs to add the module to the list of installed apps.

"""A time zone-aware DateTime field.

When saving, naive datetime objects are assumed to belong to the local time
zone and are converted to UTC. When loading from the database the naive datetime
objects are converted to UTC.

These field types require database support. MySQL 5 will not work.
"""
from datetime import datetime, tzinfo, timedelta
from django.conf import settings
from django.core.exceptions import ValidationError
from django.db import models
import pytz
import re


# 2009-06-04 12:00:00+01:00 or 2009-06-04 12:00:00 +0100
TZ_OFFSET = re.compile(r'^(.*?)\s?([-\+])(\d\d):?(\d\d)$')


class TZDatetime(datetime):
    def aslocaltimezone(self):
        """Returns the datetime in the local time zone."""
        tz = pytz.timezone(settings.TIME_ZONE)
        return self.astimezone(tz)


class TZDateTimeField(models.DateTimeField):
    """A DateTimeField that treats naive datetimes as local time zone."""
    __metaclass__ = models.SubfieldBase

    def to_python(self, value):
        """Returns a time zone-aware datetime object.

        A naive datetime is assigned the time zone from settings.TIME_ZONE.
        This should be the same as the database session time zone.
        A wise datetime is left as-is. A string with a time zone offset is
        assigned to UTC.
        """
        try:
            value = super(TZDateTimeField, self).to_python(value)
        except ValidationError:
            match = TZ_OFFSET.search(value)
            if match:
                value, op, hours, minutes = match.groups()
                value = super(TZDateTimeField, self).to_python(value)
                value = value - timedelta(hours=int(op + hours), minutes=int(op + minutes))
                value = value.replace(tzinfo=pytz.utc)
            else:
                raise

        if value is None:
            return value

        # Only force zone if the datetime has no tzinfo
        if (value.tzinfo is None) or (value.tzinfo.utcoffset(value) is None):
            value = force_tz(value, settings.TIME_ZONE)
        return TZDatetime(value.year, value.month, value.day, value.hour,
            value.minute, value.second, value.microsecond, tzinfo=value.tzinfo)


def force_tz(obj, tz):
    """Converts a datetime to the given timezone.

    The tz argument can be an instance of tzinfo or a string such as
    'Europe/London' that will be passed to pytz.timezone. Naive datetimes are
    forced to the timezone. Wise datetimes are converted.
    """
    if not isinstance(tz, tzinfo):
        tz = pytz.timezone(tz)

    if (obj.tzinfo is None) or (obj.tzinfo.utcoffset(obj) is None):
        return tz.localize(obj)
    else:
        return obj.astimezone(tz)