Tag Archives: python

Jinja2 templates and Bottle

Although Bottle’s built-in mini-template language is remarkably useful, I nearly always prefer to use Jinja2 templates because the syntax is very close to Django’s template syntax (which I am more familiar with) and because the Bottle template syntax for filling in blocks from a parent template is a bit limiting (but that’s kind of the point).

Bottle provides a nice jinja2_view decorator that makes it easy to use Jinja2 templates, but it isn’t that obvious how to configure the template environment for auto-escaping and default context, etc.

(The rest of this relates to Bottle version 0.11 and Jinja2 version 2.7.)

Template paths

Bottle’s view decorator takes an optional template_lookup keyword argument. The default is to look in the current working directory and in a ‘views’ directory, i.e. template_lookup=('.', './views/').

You can override the template path like so:

from bottle import jinja2_view, route

@route('/', name='home')
@jinja2_view('home.html', template_lookup=['templates'])
def home():
    return {'title': 'Hello world'}

Which will load templates/home.html.

Most likely you will want to use the same template path for every view, which can be done by wrapping jinja2_view:

import functools
from bottle import jinja2_view, route

view = functools.partial(jinja2_view, template_lookup=['templates'])

@route('/', name='home')
def home():
    return {'title': 'Hello world'}

@route('/foo', name='foo')
def foo():
    return {'title': 'Foo'}

That would have loaded templates/home.html and templates/foo.html.

Another way of setting a global template path for the view decorator is to fiddle with Bottle’s global default template path:

from bottle import TEMPLATE_PATH, jinja2_view, route

TEMPLATE_PATH[:] = ['templates']

@route('/', name='home')
def home():
    return {'title': 'Hello world'}

N.B. I used TEMPLATES_PATH[:] to update the global template path directly rather than re-assigning it with TEMPLATE_PATH = ['templates'].

Template defaults

Bottle has a useful url() function to generate urls in your templates using named routes. But it isn’t in the template context by default. You can modify the default context on the Jinja2 template class provided by Bottle:

from bottle import Jinja2Template, url

Jinja2Template.defaults = {
    'url': url,
    'site_name': 'My blog',

Jinja2 version 2.7 by default does not escape variables. This surprises me, but it is easy to configure a template environment to auto-escape variables.

from bottle import Jinja2Template

Jinja2Template.settings = {
    'autoescape': True,

Any of the Jinja2 environment keyword arguments can go in this settings dictionary.

Using your own template environment

Bottle’s template wrappers make a new instance of a Jinja2 template environment for each template (although if two views use the same template then they will share the compiled template and its environment).

You can avoid this duplication of effort by creating the Jinja2 template environment yourself, however this approach means you also need to write your own view decorator to use the custom template environment. No biggie.

Setting up a global Jinja2 template environment to look for templates in a “templates” directory:

from bottle import url
import jinja2

env = jinja2.Environment(
    'url': url,
    'site_name': 'My blog',

You then need a replacement for Bottle’s view decorator that uses the previously configured template environment:

import functools

# Assuming env has already been defined in the module's scope.
def view(template_name):
    def decorator(view_func):
        def wrapper(*args, **kwargs):
            response = view_func(*args, **kwargs)

            if isinstance(response, dict):
                template = env.get_or_select_template(template_name)
                return template.render(**response)
                return response

        return wrapper

    return decorator

@route('/', name='home')
def home():
    return {'title': 'Hello world'}


It’s easy to customize the template environment for Jinja2 with Bottle and keep compatibility with Bottle’s own view decorator, but at some point you may decide it is more efficient to by-pass things and setup a custom Jinja2 environment.

Bottle is nice like that.

Grouping URLs in Django routing

One of the things I liked (and still like) about Django is that request routing is configured with regular expressions. You can capture positional and named parts of the request path, and the request handler will be invoked with the captured strings as positional and/or keyword arguments.

Quite often I find that the URL patterns repeat a lot of the regular expressions with minor variations for different but related view functions. For example, suppose you want CRUD-style URLs for a particular resource, you would write an urls.py looking something like:

from django.conf.urls import url, patterns

urlpatterns = patterns('myapp.views',
    url(r'^(?P<slug>[-\w]+)/$', 'detail'),
    url(r'^(?P<slug>[-\w]+)/edit/$', 'edit'),
    url(r'^(?P<slug>[-\w]+)/delete/$', 'delete'),

The detail, edit and delete view functions (defined in myapp.views) all take a slug keyword argument, so one has to repeat that part of the regular expression for each URL.

When you have more complex routing configurations, repeating the (?P<slug>[-\w]+)/ portion of each route can be tedious. Wouldn’t it be nice to declare that a bunch of URL patterns all start with the same capturing pattern and avoid the repetition?

It would be nice.

I want to be able to write an URL pattern that defines a common base pattern that the nested URLs extend:

from django.conf.urls import url, patterns, group
from myapp.views import detail, edit, delete

urlpatterns = patterns('',
        url(r'^$', detail),
        url(r'^edit/$', edit),
        url(r'^delete/$', delete),

Of course there is no group function defined in Django’s django.conf.urls module. But if there were, it would function like Django’s include but act on locally declared URLs instead of a separate module’s patterns.

It happens that this is trivial to implement! Here it is:

from django.conf.urls import url, patterns, RegexURLResolver
from myapp.views import detail, edit, delete

def group(regex, *args):
    return RegexURLResolver(regex, args)

urlpatterns = patterns('',
        url(r'^$', detail),
        url(r'^edit/$', edit),
        url(r'^delete/$', delete),

This way the detail, edit and delete view functions still get invoked with a slug keyword argument, but you don’t have to repeat the common part of the regular expression for every route.

There is a problem: it won’t work if you want to use a module prefix string (the first argument to patterns(...)). You either have to give a full module string, or use the view objects directly. So you can’t do this:

urlpatterns = patterns('myapp.views',
    # Doesn't work.
        url(r'^$', 'detail'),

Personally I don’t think this is much of an issue since I prefer to use the view objects, and if you are using class-based views you will likely be using the view objects anyway.

I don’t know if “group” is a good name for this helper function. Other possibilities: “prefix”, “local”, “prepend”, “buxtonize”. You decide.

Testing with Django, Haystack and Whoosh

The problem: you want to test a Django view for results of a search query, but Haystack will be using your real query index, built from your real database, instead of an index built from your test fixtures.

Turns out you can generalise this for any Haystack back-end by replacing the haystack.backend module with the simple back-end.

from myapp.models import MyModel
from django.test import TestCase
import haystack

class SearchViewTests(TestCase):
    fixtures = ['test-data.json']

    def setUp(self):
        self._haystack_backend = haystack.backend
        haystack.backend = haystack.load_backend('simple')

    def tearDown(self):
        haystack.backend = self._haystack_backend

    def test_search(self):
        results = SearchQuerySet().all()
        assert results.count() == MyModel.objects.count()

My first attempt at this made changes to the project settings and did HAYSTACK_WHOOSH_STORAGE = "ram" which works but was complicated because then you have to re-build the index with the fixtures loaded, except the fixtures don’t get loaded in TestCase.setUpClass, so the choice was to load the fixtures myself or to re-index for each test. And it was specific to the Whoosh back-end of course.

(This is for Django 1.4 and Haystack 1.2.7. In my actual project I get to deploy on Python 2.5. Ain’t I lucky? On a fucking PowerMac G5 running OS X Server 10.5 for fuck sacks.)

Optimizing queries in Haystack results

My Adobe software updates app (which uses Haystack + Django to provide a search feature) has a very inefficient search results template, where for each search result the template links back to the update’s related product page.

The meat of the search results template looks something like this:

{% for result in page.object_list %}
<div class="search-result">
    <a href="{{ result.object.get_absolute_url }}">{{ result.object }}</a>
    <a href="{% url "product_page" result.object.product.slug %}">{{ result.object.product }}</a>
{% endfor %}

The reverse URL lookup triggers a separate SQL query to find the related product object’s slug field for each object in the results list, and that slows down the page response significantly.

For a regular queryset you would tell Django to fetch the related objects in one go when populating the template context in order to avoid the extra queries, but in this case page.object_list is generated by Haystack. So how to tell Haystack to use select_related() for the queryset?

It is easy. When you register a model to be indexed with Haystack for searching, you have to define a SearchIndex model, and you can also override the read_queryset() method that is used by Haystack to get a Django queryset:

# myapp.search_indexes.py
from haystack import indexes, site
from myapp.models import MyModel

class MyModelIndex(indexes.SearchIndex):
    # Indexed fields declared here
    def get_model(self):
        return MyModel

    def read_queryset(self):
        return self.model.objects.select_related()

site.register(MyModel, MyModelIndex)

And that solved it for me. Shaves three quarters off the execution time.

PS This all pertains to Django 1.4 and Haystack 1.2.7.

PPS Also pertains to a version of my Adobe software updates page that I haven’t deployed quite yet.

A context manager for files or file-like objects

I usually design my Python programs so that if a program needs to read or write to a file, the functions will take a filename argument that can be either a path string or a file-like object already open for reading / writing.

(I think I picked up this habit from Mark Pilgrim’s Dive Into Python, in particular chapter 10 about scripts and streams.)

This has the great advantage of making tests easier to write. Instead of having to create dummy temporary files on disk I can wrap strings in StringIO() and pass that instead.

But the disadvantage is I then have a bit of boiler-plate at the top of the function:

def read_something(filename):
    # Tedious but not heinous boiler-plate
    if isinstance(filename, basestring):
        filename = open(filename)

    return filename.read()

The other drawback is that code doesn’t close the file it opened. You could have filename.close() before returning but that will also close file-like objects that were passed in, which may not be what the caller wants. I think the decision whether to close the file belongs to the caller when the argument is a file-like object.

You could set a flag when opening the file, and then close the file afterwards if the flag is set, but that is yet more boiler-plate and quite ugly.

So here is a context manager which behaves like open(). If the argument is a string it handles opening and closing the file cleanly. If the argument is anything else then it just reads the contents.

class open_filename(object):
    """Context manager that opens a filename and closes it on exit, but does
    nothing for file-like objects.
    def __init__(self, filename, *args, **kwargs):
        self.closing = kwargs.pop('closing', False)
        if isinstance(filename, basestring):
            self.fh = open(filename, *args, **kwargs)
            self.closing = True
            self.fh = filename

    def __enter__(self):
        return self.fh

    def __exit__(self, exc_type, exc_val, exc_tb):
        if self.closing:

        return False

And then you use it like this:

from io import StringIO

file1 = StringIO(u'The quick brown fox...')
file2 = 'The quick brown fox'

with open_filename(file1) as fh1, open_filename(file2) as fh2:
    foo, bar = fh1.read(), fh2.read()

If you always want the file to be closed on leaving the block you use the closing keyword argument set to True (the default of False means the file will only be closed if it was opened by the context manager).

file1 = StringIO(u'...jumps over the lazy dog.')
assert file1.closed == False

with open_filename(file1, closing=True) as fh:
    foo = fh.read()

assert file1.closed == True

Today is my brother’s birthday. If I had asked him what he wanted for a present I am pretty certain he would have asked for a blog post about closing files in a computer programming language.

Custom template folders with Flask

Someone was asking on Flask‘s IRC channel #pocoo about sharing templates across more than one app but allowing each app to override the templates (pretty much what Django’s TEMPLATE_DIRS setting is for). One way of doing this would be to customise the Jinja2 template loader.

Here’s a trivial Flask app that searches for templates first in the default folder (‘templates’ in the same folder as the app) and then in an extra folder.

import flask
import jinja2

app = flask.Flask(__name__)
my_loader = jinja2.ChoiceLoader([
app.jinja_loader = my_loader

def home():
    return flask.render_template('home.html')

if __name__ == "__main__":

The only thing special here is creating a new template loader and then assigning it to the jinja_loader attribute on the Flask application. ChoiceLoader will search for a named template in the order of the loaders, stopping on the first match. In this example I re-used the loader that is created by default for an app, which is roughly like FileSystemLoader('/path/to/app/templates'). There are all kinds of other exciting template loaders available.

I really like the fact that Flask and Bottle’s APIs are so similar. Next I want Flask to include Bottle’s template wrapping decorator by default (there’s a recipe in the Flask docs) and for both of them to re-name it @template.

Inspecting your routes in Bottle

Marcel Hellkamp recently added a small feature to Bottle that makes it easy to inspect an application’s routes and determine if a particular route is actually for a mounted sub-application.

(Bottle is a small module written in Python for making websites.)

Route objects (items in the app.routes list) now have extra information when the route was created by mounting one app on another, in the form of a new key mountpoint in route.config.

Here’s a trivial app with another app mounted on it:

import bottle

app1 = bottle.Bottle()

def app1_home(): return "Hello World from App1"

app2 = bottle.Bottle()
def app2_home(): return "Hello World from App2"

app1.mount(prefix='/app2/', app=app2)

And a utility function that returns a generator of prefixes and routes:

def inspect_routes(app):
    for route in app.routes:
        if 'mountpoint' in route.config:
            prefix = route.config['mountpoint']['prefix']
            subapp = route.config['mountpoint']['target']

            for prefixes, route in inspect_routes(subapp):
                yield [prefix] + prefixes, route
            yield [], route

Finally, inspecting all the routes (including mounted sub-apps) for the root Bottle object:

for prefixes, route in inspect_routes(app1):
    abs_prefix = '/'.join(part for p in prefixes for part in p.split('/'))
    print abs_prefix, route.rule, route.method, route.callback

This new feature is sure to revolutionise everything.

Widths & Heights with xlwt + Python

This article about using xlwt to generate Excel in Python reminded me I needed to see exactly how to set column widths (the xlwt documentation doesn’t cover it).

Let’s create a new Excel workbook and add a sheet:

>>> import xlwt
>>> book = xlwt.Workbook(encoding='utf-8')
>>> sheet = book.add_sheet('sheeeeeet')

We need to get a column in order to set its width. You do that by call col() on the sheet, passing the column’s index as the only argument (or row() for accessing rows):

>>> sheet.col(0)    # First column
<xlwt.Column.Column object at 0x10b2a6190>
>>> sheet.row(2)    # Third row
<xlwt.Row.Row object at 0x10b2a7050>

The index is zero-based. You can fetch a column even if you have not written to any cell in that column (this applies equally to rows).

Columns have a property for setting the width. The value is an integer specifying the size measured in 1/256 of the width of the character ‘0’ as it appears in the sheet’s default font. xlwt creates columns with a default width of 2962, roughly equivalent to 11 characters wide.

>>> first_col = sheet.col(0)
>>> first_col.width = 256 * 20              # 20 characters wide (-ish)
>>> first_col.width

For rows, the height is determined by the style applied to the row or any cell in the row. (In fact rows also have a property called height but it doesn’t do what you want.) To set the height of the row itself, create a new style with a font height:

>>> tall_style = xlwt.easyxf('font:height 720;') # 36pt
>>> first_row = sheet.row(0)
>>> first_row.set_style(tall_style)

Setting the style on the row does not change the style of the cells in that row.

There is no obvious way to set a default width and height for all columns and rows. An instance of xlwt.Worksheet.Worksheet has properties for col_default_width and row_default_height but changing those does not actually change the defaults.

The problem is that new columns are always created with an explicit width, while rows take their height from the style information.

My first attempt at setting defaults set the width on every column and the height on every row. It works, but creates 65,536 unnecessary empty row objects.

A slightly better approach is to set the width on every column and to set the font height on the default style record in the workbook:

import itertools
import xlwt

book = xlwt.Workbook(encoding='utf-8')
sheet = book.add_sheet('sheeeeeet')

col_width = 256 * 20                        # 20 characters wide

    for i in itertools.count():
        sheet.col(i).width = col_width
except ValueError:

default_book_style = book.default_style
default_book_style.font.height = 20 * 36    # 36pt


Here I used itertools.count() and wrapped the loop in a try block so I can forget exactly how many columns are permitted. When the loop tries to access a bad index it will throw ValueError and the loop will exit.

You mustn’t replace the default style on an instance of xlwt.Workbook.Workbook, you have to update the property of the existing style (to ensure you are changing the first style record). Unfortunately there is no way to set a default column width (as of xlwt version 0.7.2) so the brute force method of setting every column will have to do – it isn’t so bad since there are only 256 columns.

Talking of widths and heights, have you heard “Widths & Heights” by Magic Arm? Is good.

Free software FTW! Updated filetimes.py

Two years ago (flippin’ heck it seems like only yesterday) I wrote about converting between Unix timestamps and Windows timestamps using Python. In that post I linked to my very simple implementation of a module that provides converting back and forth between the formats.

A few weeks ago I received an e-mail from Timothy Williams with changes to the my module so that it preserves the fractions of a second in the conversion. How sweet is that?!?!! Exclamation mark question mark exclamation mark cellida diaresis em-dash full stop king of punctuation.

It is fantastic that not only did someone find my code useful but also that they were generous enough to take the time to improve it and give the changes back to me. I love tasty, delicious free software and the people like Tim who make it tastier and more delicious.

So here is the new version of filetimes.py incorporating Tim’s fixes.

XPath bug in old versions of ElementTree

I figured out why my XML parsing code works fine using the pure-Python ElementTree XML parsing module but fails when using the speedy and memory-optimized cElementTree XML parsing module.

The XPath 1.0 specification says '.' is short-hand for 'self::node()', selecting a node itself.

Parsing an XML document and selecting the context node with ElementTree in Python 2.5:

>>> from xml.etree import ElementTree
>>> ElementTree.VERSION
>>> doc = "<Root><Example>BUG</Example></Root>"
>>> node1 = ElementTree.fromstring(doc).find('./Example')
>>> node1
<Element Example at 10e0ed8c0>
>>> node1.find('.')
<Element Example at 10e0ed8c0>
>>> node1.find('.') == node1

See how the result of node1.find('.') is the node itself? As it should be.

Parsing an XML document and selecting the context node with cElementTree in Python 2.5:

>>> from xml.etree import cElementTree
>>> doc = "<Root><Example>BUG</Example></Root>"
>>> node2 = cElementTree.fromstring(doc).find('./Example')
>>> node2
<Element 'Example' at 0x10e0e3660>
>>> node2.find('.')
>>> node2.find('.') == node2

Balls. The result of node2.find('.') is None.

However! I have a kludgey work-around that works whether you use ElementTree or cElementTree. Use './' instead of '.':

>>> node1.find('./')
<Element Example at 10e0ed8c0>
>>> node1.find('./') == node1
>>> node2.find('./')
<Element 'Example' at 0x10e0e3660>
>>> node2.find('./') == node2

Kludgey because './' is not a valid XPath expression.

So we are back on track. Also works for Python 2.6 which has the same version of ElementTree.

Fortunately Python 2.7 got a new version of ElementTree and the bug is fixed:

>>> from xml.etree import ElementTree
>>> ElementTree.VERSION
>>> doc = "<Root><Example>BUG</Example></Root>"
>>> node3 = ElementTree.fromstring(doc).find('./Example')
>>> node3
<Element 'Example' at 0x107257210>
>>> node3.find('.')
<Element 'Example' at 0x107257210>
>>> node3.find('.') == node3

However! They also fixed my kludgey work-around:

>>> node3.find('./')
>>> node3.find('./') == node3

So I can’t code something that works for all three versions. This is annoying. I was hoping to just replace ElementTree with the C version, makes my code run in one third the time (the XML parts of it run in one tenth the time). And cannot install any compiled modules – the code can only rely on Python 2.5’s standard library.