Django current 'active' page highlighting for navigation

It is pretty common to want to highlight the current page in your navigation bar. Unfortunately it is surprisingly hard to do with django. After reading a few blogs I decided on the following, which involves creating a custom template tag and enabling the request context processor. First create a custom template tag (don't forget the __init__.py in your templatetags directory):

#!/usr/bin/env python
#project/site/templatetags/active.py

from django import template
register = template.Library()

@register.simple_tag
def active(request, pattern):
    import re
    if re.search(pattern, request.path):
        return 'current_page_item'
    return ''

Add names to your views in urls.py so you can reference the regex in the template. The prototype is '(regular expression, Python callback function [, optional dictionary [, optional name]])':

(r'^$', 'home view', {}, 'home')

Then in your template:

{% load active %}
{% url home as home %}

<li class="{% active request home %}">Home</li>

In settings.py define the 'request' context processor (you also have to define the other defaults so you don't lose them):

TEMPLATE_CONTEXT_PROCESSORS = ('django.core.context_processors.auth',
                                'django.core.context_processors.debug',
                                'django.core.context_processors.i18n',
                                'django.core.context_processors.media',
                                'django.core.context_processors.request')

Now you just need a line in your CSS that does somehthing different for the 'current_page_item' class.

Re-implementing the python 'inspect' module (whoops)

I am working with python introspection to have plugins for a framework registered (mostly) automatically. Until I read this blog post I was writing most of it myself, based on how the unittesting framework registers tests. Now I know there is a python inspect module that does all the heavy lifting, and the unittest module should be using it :) I learnt some interesting things about python introspection along the way I thought I should record.

Here is my implmentation of

inspect.getmembers(testclasses,inspect.ismodule)

to get a list of modules (.py files) from a package (directory with __init__.py):

from types import ModuleType
list = dir(testclasses)
modules = filter(lambda x: getattr(testclasses,x).__class__ == ModuleType, list)

Similarly, I was going to get the testmethods using a filter and the following method:

def isTestMethod(attrname, theclass=theclass, prefix=settings.TEST_METHOD_PREFIX):
    """It is a test method if it is callable and starts with the prefix"""
    return attrname.startswith(prefix) and hasattr(getattr(theclass, attrname), '__call__')

which can be replaced by

inspect.getmembers(theclass,inspect.ismethod)

Using django components outside of django

Using django components (such as the model for DB abstraction) outside of django has a few gotchas, since you need to set up the environment correctly first. Most of the time you can just get away with setting DJANGO_SETTINGS_MODULE:

export DJANGO_SETTINGS_MODULE=yoursite.settings

But this blog has a great summary of the different options. Another approach worth noting is the one manage.py uses:

from django.core.management import setup_environ
from mysite import settings

setup_environ(settings)

Testing JSON django fixtures

My JSON testing fixture for django was bad, and django gave a really unhelpful error:

ValueError: No JSON object could be decoded

A line number would have been nice, jeez. To find out what the actual problem was I used the json module in ipython:

import json
a=open("testing.json",'r').read()
json.loads(a)

Which will either work, or give you a line number for the problem :)

Accessing Australian public transport data and Google's General Transit Feed Specification (GTFS)

I've used some pretty ordinary public transport websites. To do something about the problem, I recently embarked on a mission to create a google maps overlay. The idea was to show a particular public transport provider how their information delivery could be improved. In the process I found out lots of interesting stuff about public transport information, so I thought I'd record that first.

The San Francisco Bay Area Rapid Transit (BART) website is probably the best example of delivering useful information to commuters. They have a public, open, real-time API for developers that has spawned some great tools such as iPhone apps that display real-time arrival times for your favourite stations, provide route-planning and much more.

BART also publishes a General Transit Feed Specification (GTFS) zip file, which is a format developed by Google that allows you to provide amazing public transport functionality through google maps. Whenever you plan a route with google maps you can click on 'by public transit' and it will give you a selection of routes, times, and turn-by-turn directions for any walking required. See this random example trip I made in San Francisco.

All this is well and good for the US, but what about Australia? I started looking around, and I was amazed by the TransPerth website.


View Larger Map

Not only have they published a GTFS zip file so you can view stops and do fantastic trip planning via google maps, they also have a mobile website with real-time arrival times for bus stops and train stations, live timetables, and a 'stops near you' search on their website. The trip planner on the website is also the best I have ever used, and rivals that of google maps (they may just use a maps api...). Not surprisingly there is a Perth iPhone app, which seems to be much more feature-complete than the other capital cities, due to the data provided by TransPerth, which is infinitely preferable to screen-scraping. Of all the cities currently providing GTFS files, sadly Perth is the only Australian participating.

So, step 1 in giving commuters access to better public transport information is to publish a GTFS file. This at least means you have a database of all your stop coordinates, route names, timetable info etc. Updating the GTFS file is going to be easiest if you just use the GTFS format as your schema, it looks fairly sensible. I imagine there are lots of operators with giant excel spreadsheets holding route information, so it is probably already a step up. Next, take transperth as an example - they are kicking some goals.

PS. I was thinking - a cheap way to do GPS positioning for buses would be to stick an android phone in each one and hook it up to google latitude.

Using ssh forced-command with rsync for backups

I want to use ssh forced command to limit a backup user to just running rsync. The idea is to allow backups to be deposited without granting full shell access. The trickiest part of the problem is figuring out what command rsync will run on the server. The rsync man page gives a clue with this cryptic statement:

--server and --sender are used internally by rsync, and should never be typed by a user under normal circumstances. Some awareness of these options may be needed in certain scenarios, such as when setting up a login that can only run an rsync command. For instance, the support directory of the rsync distribution has an example script named rrsync (for restricted rsync) that can be used with a restricted ssh login.

As an aside, rrsync is a perl script that parses SSH_ORIGINAL_COMMAND and provides a way to limit rsync to certain directories. This is not a bad idea, but I always want to run the same command, so it is over-kill.

I found an insight into the --server option, which solved the mystery of what command rsync runs. Just run your regular command with '-v -v -n', and rsync will tell you. Neat!

rsync -rtz -v -v -n /home/mw/src backup@host:/home/backup/
opening connection using: ssh -l backup host rsync --server -vvntrze.iLs . /home/backup/

The actual command I will run uses one less 'v' and ditches the dry-run 'n'. So now my SSH forced command in ~/.ssh/authorized_keys looks like this:

command="rsync --server -vtrze.iLs . /home/backup/",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding ssh-dss AAAA....

Chuck it in a cron, and we are done.

HOWTO set a proxy for apt updates with a local mirror and problems with http_proxy overriding apt settings

This has annoyed me for some time now. The http_proxy environment variable in Ubuntu overrides the APT proxy settings, despite this apparently being fixed in Debian in Jan 2009. The apt configuration is more specific, and should win out over the environment variable. I'll explain why this is a problem.

Here is how it is supposed to work.

The simplest case is to set a proxy for apt by editing "/etc/apt/apt.conf" and adding this line:

Acquire::http::Proxy "http://proxy.mydom:3128";

The problems start if you have a local mirror - I do this to save bandwidth due to a large number of ubuntu installs on the network. For this config, remove any proxy lines from /etc/apt/apt.conf and create /etc/apt/apt.conf.d/30proxy:

Acquire
{
  http {
    Proxy "http://proxy.example.com:8080/";
    Proxy::ubuntu.mydom.com "DIRECT";
  }
}

With the http_proxy environment variable unset this works fine, until you go to install something like flashplugin-nonfree, which downloads a tarball from adobe. Apt completely ignores your proxy configuration and tries to download it directly:

Connecting to archive.canonical.com|91.189.88.33|:80

Which obviously doesn't work. You can set the http_proxy environment variable, but then apt won't work because it sends everything through the proxy, and the local mirror settings (ubuntu.mydom.com) you have in /etc/apt/sources.list can't go through the proxy (and shouldn't). That's what the "DIRECT" above is supposed to do.

The only way to actually make this work is described by Troy. You need to set the no_proxy environment variable:

export no_proxy="ubuntu.mydom.com"

Then make sure it actually gets kept by sudo. First get the list of var's sudo is currently preserving (look at those under "Environment variables to preserve"):

sudo sudo -V

Change /etc/sudoers with 'sudo visudo' and add:

Defaults env_keep="no_proxy http_proxy https_proxy ftp_proxy XAUTHORIZATION XAUTHORITY TZ PS2
PS1 PATH MAIL LS_COLORS KRB5CCNAME HOSTNAME HOME DISPLAY COLORS"

Check that it got kept:

sudo printenv | grep no_proxy

Chuck no_proxy and http_proxy in ~/.bashrc and you are good to go. Simple, right?