Technical notes, my online memory: January 2011

Monday, January 24, 2011

HOWTO add a directory to your python path

You can add to your python path like this:

import sys
sys.path.append("/home/me/mypy")

The (python) way of the future: pip and virtualenv

I recently discovered the magic of pip and virtualenv. Pip is shaping up as a serious contender for replacement of distutils and easy_install. I haven't used it much yet, and will have to reserve judgement until I package a decent-sized project with pip.

Using pip is easy, although there is a bootstrapping problem that, ironically, I used easy_install to solve. If your distro packages a recent version of pip, you can use it to skip this bit.

sudo apt-get install python-setuptools
s easy_install pip
s pip install virtualenv

This gives you a virtualenv install. Virtualenv is brilliant, and has significantly influenced the way I write python. Virtualenv allows you to build a complete python environment in a directory that you 'activate' by running an in-built shell script that sets your python path environment variables. The are many advantages to this approach, but here are a few I have encountered:

Simplified testing - test your code against multiple versions of libraries and utilities
Keep separate dev and production environments on your dev machine, allowing you to test in a production environment without pushing the code to your actual production server.
The isolation provided by Virtualenv ensures you aren't depending on libraries on your dev box that aren't installed in production, and upgrading libraries won't break other applications

You can even create virtual environments with different versions of the python interpreter using:

virtualenv -p python2.6 2.6ENV

So, once you have a virtualenv, activate it and install yolk to give you a picture of what is inside the environment.

source .testenv/bin/activate
~> pip install yolk
~> yolk -l
Python          - 2.6.4        - active development (/usr/lib/python2.6/lib-dynload)
distribute      - 0.6.14       - active
pip             - 0.8.1        - active
wsgiref         - 0.1.2        - active development (/usr/lib/python2.6)
yolk            - 0.4.1        - active

Then go ahead and install the other packages you need:

pip install django

You can also build a requirements file to install exactly the same versions somewhere else:

pip freeze -E .testenv > requirements.txt

The holy trinity is apparently pip, virtualenv and fabric. Fabric is something I'll have to play with soon. There is a good tutorial on pip and virtualenv here, and using them with django here.

The only limitation/problem I have run into is that you can't move the virtual environment once it is created. There is a "--relocatable" directive to address this, but the webpage says that it is somewhat experimental at the moment.

Syntax highlighting code in Blogger

Posting code in Blogger by default results in something that is inexcusably ugly. How can you make code posted in blogger look pretty? For me, syntax highlighting and scrollable text boxes are a must.

The solution is: google-code-prettify, which does the highlighting for code.google.com (you could also consider SyntaxHighlighter). Now the next problem is, how do you distribute the javascript and CSS for prettify? Unfortunately Google doesn't make it available via their CDN like they do for other libraries like jQuery. This would obviously be the best option for users since:

it would negate the need for additional hosting; and
minimise load time since browsers could cache the code across many sites, and if needed download it from the closest CDN node.

So the remaining options are to slap it into your blogger template, host it yourself somewhere (like Google pages), use the subversion browse code link (ick), or figure out what code.google.com does, since it uses the same code.

I opted for the last option, which works a treat. Put this in your blogger template, right after the <head> tag:

<link type="text/css" rel="stylesheet" href="http://www.gstatic.com/codesite/ph/3799605220899551948/css/ph_core.css">
 
<link type="text/css" rel="stylesheet" href="http://www.gstatic.com/codesite/ph/3799605220899551948/css/ph_detail.css" >
 
<link type="text/css" rel="stylesheet" href="http://www.gstatic.com/codesite/ph/3799605220899551948/css/d_sb_20080522.css" >

Then add an onload event to the body tag to call the PrettyPrint function:

<body onload="prettyPrint()" >

and, since the prettyprint CSS doesn't give you a scrolling box by default (!?), add this to the CSS in your template to avoid your code being chopped off:

/* Override prettyprint CSS, which doesn't allow scrolling box */
pre.prettyprint {
  overflow: auto;
}

Once you have done all that, you just need to wrap the code in your posts in:

<pre class="prettyprint">
</pre>

In summary, much harder than it needs to be. Blogger team, please build this functionality in!

Wednesday, January 19, 2011

List of DNS Providers

Just came across a list of DNS providers I made a while ago, and thought I'd post it here. They all seem to still be in business. The top three I have heard good reports about:

GPG HOWTO

Here is a good, quick intro to using GPG, including generating and publishing keys, encrypting and signing.

Tuesday, January 18, 2011

The Google Apps administrator control panel

The Google Apps administrator control panel for "your_domain.com" can be found at the fairly un-intuitive address:

https://www.google.com/a/your_domain.com

Wednesday, January 12, 2011

Linux boot process digram

Finally got around to converting my RHCE notes on the linux boot process into a diagram.

Wednesday, January 5, 2011

Creating a HTML DIV that adjusts its size to automatically fit the content

To create a div that automatically adjusts its size to fit the content, use this in your CSS:

margin: 0 auto;

Using 'margin' for this purpose doesn't seem particularly intuitive to me, and isn't mentioned in CSS doco I read, but it works.

Rendering a django queryset as a html table (like form.as_table) using a custom template filter

Django has some nice helpers for forms (forms.as_p and forms.as_table) that you can use in your template to minimise boilerplate HTML. Unfortunately, despite some discussion/confusion, and at least one project, there doesn't appear to be a similar elegant solution for querysets.

My approach to the problem was to implement a custom template filter you can use in a template like this (where object_list is your queryset):

{{ object_list|result_as_table }}

i.e. you hand it a queryset, and it gives you back a table.

Here is the code for the filter. I'm using this for a number of subclasses of a class called resultdata, so that they can share an identical template and just provide different querysets in urls.py.

def result_as_table(queryset, fieldnames=None):
    """Take a resultdata queryset and return it as a HTML table.
    Columns will be returned in the order they are declared in the model.
    
    Optionally, provide a list of fieldnames, which will be used to limit the output."""


    dictlist = queryset.values()
    output = "<table>\n"
    output_keys = []

    if fieldnames:
        names = fieldnames
    else:
        names = dictlist.field_names

    for name in names:

        if not name.endswith("_id"):
            output_keys.append(name)
            output = "".join( (output, "<th>%s</th>\n" % escape(name.replace("_"," ").capitalize())) )

    for rddict in dictlist:

        output = "".join( (output, "<tr>\n") )

        for key in output_keys:
            val = rddict[key]
            if not key.endswith("_id"):

                display_func = get_display_method(rddict, queryset, key)

                if display_func:
                    output = "".join( (output, "<td>%s</td>\n" % escape(display_func())) )
                else:
                    output = "".join( (output, "<td>%s</td>\n" % escape(val)) )

        output = "".join( (output, "</tr>\n") )

    return mark_safe("".join( (output, "</table>\n") ))

result_as_table.is_safe = True

This filter introspects the column headers, strips out any id fields (since users don't care about them), converts underscores to spaces in column headers, and finds the corresponding display methods for any 'choices' fields so you can still get the nice human-readable output. It outputs HTML, so it uses django's own 'django.utils.html.escape' method and marks the output as safe so django doesn't escape all the HTML.

Finding the display methods was a little harder than I expected, since the python inspect code was dying when given a django model object. This code is unfortunately tied to the resultdata object due to the naming of the keys in the values() dictionary, making it less generic than I would like.

def get_display_method(rddict, queryset, key):
    """Re-implementation of inspect.getmembers(rddict,inspect.ismethod) and 
    a test to see if this is a get_field_status method.  Had to reimplement inspect
    because it was bombing on the object - expecting some property that is not present."""

    rdobj = queryset[0].__class__.objects.filter(resultdata_id = rddict["resultdata_id"]).get()
    targetname = "get_%s_display" % key
    display_func = False

    for name in dir(rdobj):
        if name == targetname:
            try:
                display_func = getattr(rdobj, name)
                break
            except Exception,e:
                pass

    return display_func

There was a major gotcha with using the field_names property of the queryset.values() dictionary. If you passed a queryset with annotations like this is your urls.py:

list_rd = {
    'extra_context': {'fieldnames': ['status','first','total']},
    'queryset': ResultData.objects.annotate(total = Count('link__clickback'), 
                                            first = Min('link__clickback__timestamp')),
}

urlpatterns += patterns('project.app.views',
    url(r'^result/detail/$', 'detailed_results', kwargs=list_rd, name='detailed_results'),
)

The annotations would not get added into the field_names property. This was solved by providing the optional fieldnames parameter.