Monthly Archives: September 2010

Running Django on Mac

These are semi-detailed steps for installing all the bits to host a [Django][django] application on [Mac OS X][macosx]. Tested on 10.5, should work perfectly on 10.6.

Use [MacPorts][macports]: relatively easy to install and the best thing is everything is contained in a directory that you can be confident won’t eff up Apple’s stuff and won’t be effed up by Apple’s stuff.

Install Xcode
————-

You need the compiler and bits that are installed with Xcode. If you can’t find your Mac install discs (Xcode is included with every new Mac but not installed) you can [download it from Apple’s developer website][xcode]. Registration is required but is free.

The current version of Xcode is easy to find, while older versions are available in the downloads section under “Developer Tools”. Xcode version 3.1.4 is the last version that will work for Mac OS X 10.5 systems.

Install MacPorts
—————-

MacPorts have a nice pkg installer. You can also build it from source.

curl -O http://distfiles.macports.org/MacPorts/MacPorts-1.9.1-10.5-Leopard.dmg
hdiutil attach MacPorts-1.9.1-10.5-Leopard.dmg
sudo installer -pkg /Volumes/MacPorts-1.9.1/MacPorts-1.9.1.pkg -target /
hdiutil detach /Volumes/MacPorts-1.9.1

If for some reason MacPorts cannot fetch updates you may need to [pull updates by hand][manualports].

Check your $PATH after installing ports to make sure `/opt/local/bin` is in there. If it isn’t your can do `export PATH=/opt/local/bin:/opt/local/sbin:${PATH}` to fix things, and even add taht line to `~/.profile` so that bash picks it up every time (assuming you haven’t switched your shell).

Install software
—————-

The `port` command is used to manipulate the MacPorts installation. Use it to build and install the various bits we need. This takes a while, especially on old PowerPC machines. Make it more exciting by adding the `–verbose` flag. Exciting!

sudo port install python26
sudo port install apache2
sudo port install mysql5-server
sudo port install mod_python26
sudo port install py26-mysql
sudo port install py26-django
sudo port install py26-south

And if you want to hook Django into a network directory then you almost certainly want to use LDAP.

sudo port install py26-ldap

Cool kids these days say use [mod_wsgi][modwsgi] instead of [mod_python][modpython] for hosting Python applications with Apache, but I am not cool (and on 20 September 2010 I couldn’t persuade mod_wsgi to build from MacPorts on a clean installation).

Configuring and starting MySQL
——————————

*UPDATED: [commenter matea][matea] pointed to [Jason Rowland’s MySQL on Mac][jason] posting that includes steps to secure a default installation, so I’ve updated this section with the appropriate steps.*

I always seem to be the only person who cares about non-English visitors… anyway, so I want to have [MySQL][mysql] use UTF8 for everything. Edit the configuration so it does. As root, create a configuration at `/opt/local/var/db/mysql5/my.cnf` with these lines:

[mysqld]
init-connect = ‘SET NAMES utf8’
character-set-server = utf8
collation-server = utf8_general_ci
skip-networking

[mysql]
default-character-set = utf8

One thing about the line `skip-networking` in the configuration file is that it means MySQL will not listen to __any__ network clients, including connections to `127.0.0.1`. Instead clients should connect to `localhost` or they should specify the path to the socket that MySQL uses for communication. If your MySQL “client” is a Django instance running on the same host then that should not be a problem.

Now initialize the database and start the server. (The use of `-w` in the second line tells launchctl to have the database daemon start at boot. If you don’t want to have MySQL running at boot use `-F` to __force__ start just this one time instead of every time.)

sudo -u mysql /opt/local/bin/mysql_install_db5
sudo launchctl load -w /Library/LaunchDaemons/org.macports.mysql5.plist

And let’s check that the server is up and configured right.

/opt/local/bin/mysql5 -e “SHOW variables LIKE ‘%char%'”

You should see a table showing that the character set for the client and server is set to utf8.

Now run the secure installation script for MySQL. This will ask you to set a password for MySQL’s root account (the administrator) and ask whether to remove the test database and anonymous user access (you should do both):

/opt/local/bin/mysql_secure_installation5

Thaz better.

Configuring Postgresql instead of MySQL
—————————————

If you want to use [Postgres][postgres] instead of MySQL then you need a couple different packages out of ports.

sudo port install postgresql84-server
sudo port install py26-psycopg2

Did you know Apple’s management tools use Postgres? Is true.

Configuring Apache to serve a Django project
——————————————–

Let’s suppose your Django project lives under `/Library/WebServer/example.com/myproj`, and the project’s settings file is `/Library/WebServer/example.com/myproj/settings.py`. Here’s how to configure Apache with mod_python to serve your project.

Create a separate site configuration for Apache in `/Library/WebServer/example.com/site.conf`.

SetHandler python-program
PythonHandler django.core.handlers.modpython
SetEnv DJANGO_SETTINGS_MODULE myproj.settings
PythonOption django.root /
PythonDebug On
PythonPath “[‘/Library/WebServer/example.com’] + sys.path”

Order deny,allow
Allow from all

Of course once everything is hunky dory you will go back and edit the site configuration so that `PythonDebug Off`.

And finally tell Apache to use mod_python and read the site configuration. Edit `/opt/local/apache2/conf/httpd.conf` and add a line at the end of the modules like:

LoadModule python_module modules/mod_python.so

And then a line like:

Include /Library/WebServer/example.com/site.conf

Now fire up Apache:

sudo launchctl load -w /Library/LaunchDaemons/org.macports.apache2.plist

MacPorts has a convenient shortcut for this:

sudo port load apache2

You also want to save Apache a little grief by pre-compiling the Python source files for the project:

/opt/local/bin/python2.6 -m compileall /Library/WebServer/example.com

Hope this helps.

[manualports]: http://reliablybroken.com/b/2010/03/using-macports-behind-a-firewall/
[macosx]: http://www.apple.com/macosx/
[django]: http://www.djangoproject.com
[macports]: http://www.macports.org/
[xcode]: http://developer.apple.com/technology/xcode.html
[mysql]: http://www.mysql.com
[postgres]: http://www.postgresql.org
[modwsgi]: http://docs.djangoproject.com/en/dev/howto/deployment/modwsgi/
[modpython]: http://docs.djangoproject.com/en/dev/howto/deployment/modpython/
[matea]: http://reliablybroken.com/b/2010/09/running-django-on-mac/comment-page-1/#comment-1458
[jason]: http://www.jasonrowland.com/2009/10/install-mysql5-on-snow-leopard-using-macports/

Notes on using the BBC’s JSON schedule data

3 Replies

The [BBC used to publish their schedules][tarball] in a great big XML tarball. They did this daily and it contained the broadcasts for the next seven days. I wrote a script that created [iCalendar][ical] files out of this for each channel.

But a couple weeks ago they [stopped][old] publishing the schedules in this format. In its place they provide [schedules for each channel or by genre or by series][new], and they also provide other formats in addition to XML.

One great advantage of the new data is that it is also available in iCalendar format, which _almost_ makes my script redundant were it not for the fact there is no weekly schedule for each channel.

[I re-wrote my script][guide] to use the newer schedule data and I got opinions!

1. Only daily schedules available for channels.

You can ask for today’s schedule, for tomorrow’s schedule, for a specific date’s schedule. You cannot ask for a week’s schedule (which is pretty much all I want) or for anything other than a day’s schedule.

However for a particular programme (but not for a channel) you can get the upcoming schedule.

2. A programme title consisting only of numbers is encoded as an integer.

I came across [a broadcast where the programme title was “606”][606] but the title appeared in the JSON data as `”title”: 606`. The value of the title should always be a string, as should other textual values.

3. Post-midnight broadcasts appear in the schedule for two days.

This is a side-effect of the fact I need to look at several consecutive days’ schedules in order to combine them into a week’s schedule. If you look at today’s schedule, programmes that are broadcast after midnight are included. If you look at tomorrow’s schedule, programmes that start immediately after midnight are included, even though they also appeared on today’s schedule.

This behaviour makes perfect sense when the schedules are consumed by a real person. If I look up today’s schedule I _want_ to see those transmissions that happen early the next morning, even though they are strictly not part of today’s schedule.

But for consuming the data programmatically, having the overlap means you have to deal with duplicate events. In my case I dealt with this by converting transmission events to tuples which are then easy to compare and you can weed out the duplicates by throwing them all [into a Python set][set].

There is a conflict between the notion of a calendar day and a transmission day. For most BBC channels the day begins and ends sometime between 1 o’clock and 5 o’clock in the morning. For some channels it isn’t obvious that one day’s programming has stopped and another has begun because there may be no dead air where nothing is transmitted, but think back to the good old days where all you got in the middle of the night was the white dot to let you know the telly was still working.

My re-written script pulls the schedule using the JSON data feeds, but I wonder if things would be easier if it just concatenated the events that are published in the daily iCalendar files? I guess we’ll never ever know.

[tarball]: http://backstage.bbc.co.uk/data/7DayListingData?v=16wk
[ical]: http://en.wikipedia.org/wiki/ICalendar
[new]: http://www.bbc.co.uk/programmes/developers
[old]: http://www0.rdthdo.bbc.co.uk/services/api/
[606]: http://www.bbc.co.uk/5live/programmes/schedules/2010/10/02
[guide]: http://reliablybroken.com/guide/
[set]: http://docs.python.org/library/stdtypes.html#set-types-set-frozenset

Chrome’s annotated scrollbar

How to manage more than one Mac

1 Reply

This is an overview of what tools there are for managing a bunch of Macintosh computers on a network. It is intended as a starting point for someone wondering what on earth to do with all those pretty-looking computers the hippie designers like so much.

Basic environment
—————–

You must remove admin access for users, otherwise you cannot be certain where the user data is, and if you cannot isolate the user’s data then you cannot trust that updating the system won’t delete their stuff.

You really need to have the Macs bound to a directory and have account info stored in the directory. This means you don’t need to create and maintain local accounts on each Mac. It is also makes life much simpler when you need to migrate data from one Mac to another because the account uid/gid will be persistent.

Mac OS X Server
—————

The server version of Mac OS X includes a [software update server][sus]. By hosting your own local copy of the Apple software update server you conserve network bandwidth and you get to withhold what updates are presented to the Mac clients should you want to.

You also get to use Apple’s directory service and to control policy with MCX.

You don’t necessarily need OS X Server, but it is very useful.

[sus]: http://support.apple.com/kb/HT4069

Reporting and management suites
——————————-

From there you have a bunch of approaches. Big environments (educational) use things like [Radmind][1] and [Puppet][2] to enforce software load-sets across hundreds, thousands of Macs.

For small network sizes people use [Apple Remote Desktop][3] (ARD) and/or [Casper][4]. ARD is extremely useful whatever you decide to use because the administration software includes remote desktop stuff so the help desk monkey can take over the user’s screen, fix the problem and put the telephone back down.

Although these tools handle reporting, remote installations, running scripts, batch configuration, etc. you still need to spend a lot of time building installation packages. You need a base system image that can be quickly deployed to a Mac, and you need additional images or installation packages containing the various software that goes on top.

[1]: http://rsug.itd.umich.edu/software/radmind/
[2]: http://www.puppetlabs.com/
[3]: http://www.apple.com/remotedesktop/
[4]: http://www.jamfsoftware.com/products/casper-suite

Deploying system software and applications
——————————————

To build system restore images use [InstaDMG][5]. Much better than everything else.

What you want for deploying software is some form of un-attended network push. This means the software installation has to work with access via SSH, whether the user is logged in or not and needs to succeed when installed as root.

With luck your software vendor already uses well-behaved package format installers (i.e. Microsoft most of the time). Or the software is a self-contained app so it can be installed and updated by copying it to /Applications (maybe you go to the trouble of making a pkg for it first).

But flipping heck Adobe hates Mac administrators. The Adobe Creative Suite installers (including those on volume license media) are absolutely useless. The updaters are even worse. [Adobe provides a volume deployment kit][6] which is bafflingly obtuse and which doesn’t actually work for the admin’s most important installation scenario.

Casper and [LANRev][7] include tools for building pkg installers from the Adobe media. Adobe’s enterprise kit supposedly does the same thing. The other thing you do is build your own pkg from what gets installed. That isn’t too painful, but having to build your own update pkgs for the numerous Adobe updates is painful.

(The [Apple Installer-Dev list][8] is full of developers trying to work around bugs in PackageMaker that crop up as soon as you want to do anything other than copy files to disk. Many admins prefer [Iceberg][9] or [Packages][packages] and other software for building installer packages.)

[5]: http://code.google.com/p/instadmg/
[6]: http://www.adobe.com/devnet/creativesuite/enterprisedeployment.html
[7]: http://www.lanrev.com/
[8]: http://lists.apple.com/mailman/listinfo/installer-dev
[9]: http://s.sudre.free.fr/Packaging.html
[packages]: http://s.sudre.free.fr/Software/Packages/about.html

System configuration and user preferences
—————————————-

To some degree you configure the system as part of building the base image. Things like have the Mac automatically configure the network, bind to a domain, set power-saving mode, disable auto-login, etc. should all be part of a first-run script that happens when a new Mac is switched on.

In general you should avoid changing the default settings for everything else, either per-Mac or per-user. Saves you much hassle.

Most software these days uses plist format and reads the settings from the correct places (even recent versions of Office are good at this). Adobe of course doesn’t, but I have infinite patience. Assuming all the prefs you need to manage are in plist format, you either just run scripts to set things on an ad-hoc basis or you enforce policy with login hooks. If you use [Open Directory to manage Macs][10] you can use the Mac OS X Server tools to push settings to all users. Since 10.5 you can [use each Mac’s local directory in much the same way as Open Directory to manage settings too][11] (although you need to then handle updating each Mac yourself).

You can use [login/logout hooks][12] which run for each user. You usually find if you run a system-wide script to adjust per-user settings that at some point you mess up the permissions, no matter how careful your script is. The other approach is to use [per-user launchd scripts][13], although they were effectively broken in 10.4 and not much better in 10.5. The [launchd man pages][launchd] are required reading even so.

[10]: http://images.apple.com/server/macosx/docs/User_Management_v10.6.pdf
[11]: http://www.afp548.com/article.php?story=using-mcx-in-the-dslocal-domain
[12]: http://support.apple.com/kb/HT2420
[13]: http://developer.apple.com/mac/library/documentation/MacOSX/Conceptual/BPSystemStartup/Articles/LaunchOnDemandDaemons.html
[launchd]: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man5/launchd.plist.5.html

Interesting bits
—————-

– Mac OS X Server docs: [http://www.apple.com/server/macosx/resources/documentation.html](http://www.apple.com/server/macosx/resources/documentation.html)
– Macos-x-server mailing list: [http://lists.apple.com/mailman/listinfo/macos-x-server](http://lists.apple.com/mailman/listinfo/macos-x-server)
– Munki, for deploying software: [http://code.google.com/p/munki/](http://code.google.com/p/munki/)
– DeployStudio, like Casper: [http://www.deploystudio.com/Home.html](http://www.deploystudio.com/Home.html)
– AFP548, fantastic tips: [http://www.afp548.com/](http://www.afp548.com/)
– Mike Bombich, admin: [http://www.bombich.com/](http://www.bombich.com/)
– Greg Neagle, admin: [http://managingosx.wordpress.com/](http://managingosx.wordpress.com/)
– Preston Holmes, admin: [http://ptone.com/dablog/](http://ptone.com/dablog/)
– MacEnterprise mailing list: [http://www.macenterprise.org/mailing-list](http://www.macenterprise.org/mailing-list)
– MacManagers mailing list: [http://www.mac-mgrs.org/](http://www.mac-mgrs.org/)
– Adobe installer blog: [http://blogs.adobe.com/oobe/](http://blogs.adobe.com/oobe/)

Reliably Broken

It's a blog: let's do funch!

Monthly Archives: September 2010

Running Django on Mac

Notes on using the BBC’s JSON schedule data

Chrome’s annotated scrollbar

How to manage more than one Mac