Category Archives: Blog

Excel scroll bar bug

[Microsoft Excel 2008 for Mac][excel2008] has an irritating bug where only the active document window has scroll bars. If a second document is open, or even if you only have one document but Excel is not the front-most application then the window has no scroll bars and ignores scroll messages from the mouse. This happens in Excel 12.2.0 (and a couple of earlier revisions behave the same).

Excel scroll bars in inactive window

Excel scroll bars in an inactive window

The correct behaviour is for [scroll bars in inactive windows to be drawn in an inactive state][hig] and to allow scrolling even when the window is not front-most.

I rather like Excel. I thought it was the least crashy of the assorted junk Microsoft released as Office 4.2 for Mac back in 1993. Fuck me that was a pile of shit.

[excel2008]: http://www.microsoft.com/mac/products/Excel2008/
[hig]: http://developer.apple.com/mac/library/documentation/UserExperience/Conceptual/AppleHIGuidelines/XHIGWindows/XHIGWindows.html#//apple_ref/doc/uid/20000961-TPXREF26

Working with Active Directory FILETIME values in Python

[How To Convert a UNIX time_t to a Win32 FILETIME or SYSTEMTIME][kb]:

> Under Win32 platforms, file times are maintained primarily in the form of
> a 64-bit FILETIME structure, which represents the number of 100-nanosecond
> intervals since January 1, 1601 UTC (coordinate universal time).

***UPDATED* New version with fixes by Tim Williams for preserving microseconds. See [here for details][updated].**

It just so happens that [Microsoft Active Directory][msad] uses the same 64-bit value to store some time values. For example [the `accountExpires` attribute][expires] is in this format. Linked below is a module for Python with utility functions for converting between [Python’s datetime instances][datetime] and Microsoft’s FILETIME values.

Very handy if you enjoy querying Active Directory for login accounts that are due to expire. And who wouldn’t enjoy that? On a Monday.

[Download filetimes.py module for converting between FILETIME and `datetime` objects.][filetimes] This code is released under a 2-clause BSD license.

Example usage:

>>> from filetimes import filetime_to_dt, dt_to_filetime, utc
>>> filetime_to_dt(116444736000000000)
datetime.datetime(1970, 1, 1, 0, 0)
>>> filetime_to_dt(128930364000000000)
datetime.datetime(2009, 7, 25, 23, 0)
>>> “%.0f” % dt_to_filetime(datetime(2009, 7, 25, 23, 0))
‘128930364000000000’
>>> dt_to_filetime(datetime(1970, 1, 1, 0, 0, tzinfo=utc))
116444736000000000L
>>> dt_to_filetime(datetime(1970, 1, 1, 0, 0))
116444736000000000L

I even remembered to write tests for once!

[kb]: http://support.microsoft.com/kb/167296
[msad]: http://www.microsoft.com/windowsserver2008/en/us/active-directory.aspx
[datetime]: http://docs.python.org/library/datetime.html
[expires]: http://msdn.microsoft.com/en-us/library/ms675098(VS.85).aspx
[filetimes]: /b/wp-content/filetimes.py
[updated]: http://reliablybroken.com/b/2011/09/free-software-ftw-updated-filetimes-py/

Outputting Excel with Django

[`xlwt`][xlwt] is an excellent Python module for generating [Microsoft Excel][msexcel] documents ([`xlrd` is its counterpart][xlrd] for consuming Excel documents). I use it in a [Django][django] Web application so a visitor can export her data as a spreadsheet.

Django’s documentation includes an example of [how to export data in comma-separated values (CSV) format][djangocsv]. CSV has the significant advantage of being a standard Python module as well as being a relatively simple and non-vendor specific format. However there are some disadvantages to using CSV:

1. Values can only be stored as strings or numbers.
2. Unicode text must be explicitly encoded as UTF-8.
3. Users are often unfamiliar with the `.csv` file name extension – “What the hell do I do with this damn you?”

It would be unfriendly of me to expect a user to open a CSV file and then format a column of date strings as proper date values (especially when the user is almost certainly using Excel already). So I choose Excel format over CSV format.

Dates in Excel documents (97/2004 format) are actually stored as numbers. In order to have them appear as dates one must apply a date formatting. You do this by using `xlwt.easyxf` to create a suitable style instance and then pass that when writing the cell data.

A word of advice: do _not_ instantiate style objects more than once! My initial approach created a new style whenever writing a date/time value. Only once I was testing with more than a few dozen rows did I discover that Excel will grow grumpy and complain about too many fonts being open when trying to display the spreadsheet. The correct approach is to have one instance for each different style and then re-use that instance for the appropriate type of value.

Here is an example that writes all objects of one class to a spreadsheet and sends that file to the client’s browser. You could stuff this in a Django view method.

from datetime import datetime, date
from django.http import HttpResponse
from myproject.myapp.models import MyModel
import xlwt

book = xlwt.Workbook(encoding=’utf8′)
sheet = book.add_sheet(‘untitled’)

default_style = xlwt.Style.default_style
datetime_style = xlwt.easyxf(num_format_str=’dd/mm/yyyy hh:mm’)
date_style = xlwt.easyxf(num_format_str=’dd/mm/yyyy’)

values_list = MyModel.objects.all().values_list()

for row, rowdata in enumerate(values_list):
for col, val in enumerate(rowdata):
if isinstance(val, datetime):
style = datetime_style
elif isinstance(val, date):
style = date_style
else:
style = default_style

sheet.write(row, col, val, style=style)

response = HttpResponse(mimetype=’application/vnd.ms-excel’)
response[‘Content-Disposition’] = ‘attachment; filename=example.xls’
book.save(response)
return response

That code works a peach with a 30,000 row / 25 column database, taking about a minute to generate a 13 megabyte file on [my lowly iMac G5][g5].

You want to buy me [a new Intel iMac][imac], don’t you? Yes, you do.

[xlwt]: http://pypi.python.org/pypi/xlwt
[xlrd]: http://pypi.python.org/pypi/xlrd
[msexcel]: http://office.microsoft.com/excel
[django]: http://www.djangoproject.com/
[djangocsv]: http://docs.djangoproject.com/en/dev/howto/outputting-csv/
[g5]: http://support.apple.com/kb/SP45
[imac]: http://www.apple.com/imac/

Glow Javascript library

Only just come across the [Glow Javascript library][glow] that the [BBC][bbc] released under the [Apache license][apache] a couple of months ago. It looks good, and has [excellent API documentation][glowdocs] as well as discussions of general usage for the majority of the tools.

In particular it has widgets for auto-complete text inputs, a simple class for validating form input and supports CSS-style selectors for referencing DOM objects.

Being that the BBC is mightily concerned with scheduled transmissions, Glow has an implementation of a timetable for displaying concurrent “tracks” of time data – I think this would complement the [Simile timeline library][simile] in various situations.

Seeing as I can’t do Javascript without [jQuery][jquery] or getting depressed, Glow is very interesting.

[apache]: http://www.apache.org/licenses/LICENSE-2.0.html
[glow]: http://www.bbc.co.uk/glow/
[bbc]: http://www.bbc.co.uk/
[glowdocs]: http://www.bbc.co.uk/glow/docs/
[simile]: http://www.simile-widgets.org/timeline/
[jquery]: http://jquery.com/

BBC iCalendar schedules

[Jon Udell][jonudell] recently [wrote about accessing the BBC programming schedules][post] but was put-off by the lack of time zone information in the iCalendar feeds, which prompted me to fix the quick-and-dirty script I have that generates [iCalendar files for the BBC][guide]. (I wrote the first, time zone-blind version of my script in England’s Winter and it worked just perfick back then!)

So [I fix it][fixed]. The updated iCalendar files have events with time zone information.

Everyone’s happy.

Jon Udell’s use of Python to explore data manipulation on the Web was one of the reasons I thought I really ought to get stuck into [Python][python].

[jonudell]: http://blog.jonudell.net/
[post]: http://blog.jonudell.net/2009/08/05/curation-meta-curation-and-live-net-radio/
[guide]: http://reliablybroken.com/guide/
[python]: http://www.python.org/
[fixed]: http://reliablybroken.com/guide/bbcguidetz.py

Microsoft Update versus Internet Explorer 8

Fun implications of designing [your software update service][msupdate] as a pseudo-application that is actually an ActiveX plugin (that needs to guarantee the owning web page cannot be closed) include refusing to allow _any_ web page to come forward when the owning web page is one of many tabs in Internet Explorer 8.

It is a little satisfying seeing Microsoft’s chickens come home to roost, but I would much rather they hadn’t made such obvious, Web-hostile choices in the first place.

[msupdate]: http://update.microsoft.com/microsoftupdate/v6/default.aspx

Django and time zone-aware date fields (redux)

Previously on 24…

I posted [a module for handling time zone-aware datetime objects][oldpost], but I left out all the hassle of dealing with form input. Here is a more complete python package for [Django][django] that includes a form field sub-class that can handle a small set of datetime string formats that include a time zone offset.

[Timezones 0.1][timezones]

This code is released under Django’s BSD license.

[oldpost]: http://reliablybroken.com/b/2009/06/django-and-time-zone-aware-date-fields/
[django]: http://www.djangoproject.com/
[timezones]: http://reliablybroken.com/b/wp-content/uploads/2009/08/timezones-01.tar.gz

Me and Adobe hates you

It’s not true, I don’t hate you, but the [Adobe CS4][cs4] installer does.

I was installing Adobe Creative Suite 4 today on a Macintosh. I had the amazing Adobe CS4 Master Collection media to install from, but all I needed was the Illustrator, InDesign, Photoshop and Bridge. So in the installation process I choose a custom install and de-select most everything except those applications. I un-ticked After Effects because I didn’t want to install After Effects. I un-ticked Soundbooth, I un-ticked Dreamweaver, I un-ticked all them other applications.

The installation process begins. Finally it finishes. Afterwards I had Illustrator CS4, InDesign CS4, Photoshop CS4 and Bridge CS4 installed on my Mac’s hard drive. They all worked a treat.

I also had a dog’s dinner of all the other flipping applications installed, all the applications I did not want to install, I had little vomit stains of Adobe installations that didn’t quite actually install, but installed.

So for example I had a new folder in the Applications folder called “Adobe After Effects CS4” and inside there was an application named “Adobe After Effects CS4.app”. Except it isn’t an actual flipping application. It has a great big no entry icon on it because it isn’t an actual executable application.

CS4 installation mistake

In one sense the installer did what I expected: it did not install a usable After Effects CS4. In another sense it effing ignored what I expected by creating an installation folder for After Effects CS4 and then created an incomplete stub of the After Effects CS4 application.

I wish the Adobe CS4 installer had just done what I asked for. [The Adobe installation and licensing blog is here][oobe].

[oobe]: http://blogs.adobe.com/OOBE/
[cs4]: http://www.adobe.com/products/creativesuite/

Grep template tag for Django

Nice and easy couple of [Django template tags][tagsandfilters] that filter lines of text using a regular expression. I had a block of text where I wanted to remove some of the lines but not others.

You can use `grep` to remove any lines that do not match your pattern:

>>> s = ‘The quick brown fox’
>>> grep(s, ‘quick’)
u’The quick brown fox’

And its converse `grepv` to remove any lines that do match your pattern:

>>> s = ‘The quick brown fox’
>>> grepv(s, ‘quick’)

>>> s2 = s + ‘\nJumps over the lazy dog’
>>> grepv(s2, ‘quick’)
u’Jumps over the lazy dog’

Stick it in a module in your Django application ([documentation][customtags]), then load it up at the top of a template.

from django import template
from django.template.defaultfilters import stringfilter
import re

register = template.Library()

@register.filter
@stringfilter
def grep(value, arg):
“””Lines that do not match the regular expression are removed.”””
pattern = re.compile(arg)
lines = [line for line in re.split(r'[\r\n]’, value) if pattern.search(line)]
return ‘\n’.join(lines)

@register.filter
@stringfilter
def grepv(value, arg):
“””Lines that match the regular expression are removed.”””
pattern = re.compile(arg)
lines = [line for line in re.split(r'[\r\n]’, value) if not pattern.search(line)]
return ‘\n’.join(lines)

[tagsandfilters]: http://docs.djangoproject.com/en/dev/ref/templates/builtins/
[customtags]: http://docs.djangoproject.com/en/dev/howto/custom-template-tags/

The nature and purpose of AppleScript

An amusing sentence from the first chapter of *[AppleScript The Definitive Guide][oreilly]* by Matt Neuberg:

> To find reasons to use AppleScript, just leave your mental annoyance meter turned on.

But don’t forget to switch off your brain once you do start using AppleScript; the frustration of getting anything done with the miserable Macintosh applications that expose their workings via AppleScript never fails to kick my mental annoyance meter into the red.

The following paragraph employs sarcasm. I’ve just being having a wonderful time figuring out that [Adobe InDesign CS3][adobe] keeps the list of items on a page sorted by layer, but sorted in reverse layer order and then sorted forward within each layer. This makes it super sweet when you want to determine if box A in layer 1 overlaps box B in layer 2. Thanks Adobe! Another genius implementation from you guys! You guys fucking rock!

And it isn’t just the fault of the applications you want to script, the language itself is intentionally imprecise in trying to allow programs to be written in natural language. “Whether this English-likeness is a good thing is debateable,” writes Matt Neuberg, but he is being polite because if there is any debate on this question it is a fucking short debate: AppleScript’s English-likeness only serves to have you make mistakes with more confidence.

There is an interesting [history of AppleScript][history] which tantalizes the present-day scripter with the revelation that the language authors wrote a syntax “which resembled Java” but never released it, preferring to support scripts written in fucking French and fucking Japanese!

[oreilly]: http://oreilly.com/catalog/9780596102111/
[history]: http://www.cs.utexas.edu/~wcook/Drafts/2006/ashopl.pdf
[adobe]: http://www.adobe.com/products/indesign/