Where there is awk, there is sed

I couldn't do a post on awk without also quickly covering sed. Bruce Barnett has written a great tutorial for sed that is worth reading.

Sed is your friend for applying regex's to files (well, streams really, since it is the 'stream editor'). The regex syntax is the same as for vim, and since that is my primary editor I only tend to use sed where the files are large and will take ages to load into vim.

Some quick examples to illustrate what I'm talking about:

sed s'/user root/user nothingtoseehere/g' < /var/log/auth.log

sed s'!session closed for user \([^ ]*\)!\1 closed a session!g' < \
/var/log/auth.log

sed s'!session closed for user \([^ ]*\)!&, allegedly!g' < \
/var/log/auth.log

Escaped parenthesis

\(\)

capture a value, and

&

refers to the whole match. You can also use sed like grep. By default it prints every line, but that can be disabled with "-n" and you can cause matching lines to be printed by appending a "p" option:

sed -n 's/pattern/&/p' < file.txt

OR, if we aren't making a substitution

sed -n '/pattern/p' < file.txt

OR, you can just use grep -e, which is less confusing:

grep -e "pattern" file.txt

Rewriting a file in-place, making a backup with the .bak extension:

sudo sed -i.bak s'!XKBOPTIONS=""!XKBOPTIONS="ctrl:nocaps"!' /etc/default/keyboard

AWK - selecting columns for output

AWK is a super-handy old-skool UNIX tool. There are plenty of good tutorials out there for it, but I'm jotting down some basic uses here for my own benefit.

I have used AWK mainly to select and print certain columns from input, like this, which will print the 1st and 7th columns:

awk '{print $1,$7}' < /var/log/syslog

Columns break-up is determined by the value of the FS (input field separator) variable, which is space by default (in POSIX mode this actually means space and tab but not newline). You can change this with:

awk 'BEGIN {FS=";"}{print $1,$7}' < /var/log/syslog

OR

awk -F: '{print $1,$7}' < /var/log/syslog

The output from awk is separated by the OFS (output field separator) variable, also a space by default. To write out CSV you might use:

cat /var/log/syslog | awk -F: 'BEGIN{OFS=","}{print $1,$3}'

There is plenty more you can do with awk, including simple programming tasks such as counting, summing etc. cut is a simple alternative if all you want to do is cut fields from an input stream. It is doesn't take much to hit its limitations however. Consider the output of last, the first two columns of which look like this:

user     pts/0
user     pts/1
reboot   system boot

This awk command will print the first two columns correctly:

last | awk '{print $1,$2}'

Whereas this cut command:

last | cut -d" " -f1-5

Won't produce the first two columns cleanly and we need to specify 5 columns to try and skip the empty fields. The problem is there are variable numbers of spaces between the username and the tty line.

Tips for hardening apache on ubuntu for django deployment

There is good doco for deploying django on apache with mod_python or wsgi. Here are a couple of extra tips for Ubuntu. First, edit

/etc/apache2/conf.d/security

and enable:

ServerTokens Prod
ServerSignature Off
TraceEnable Off

And in the apache config in your "Location /" directive with the other django stuff:

Options -Indexes -Includes -Multiviews SymLinksIfOwnerMatch

Take a look at Apache's security tips and it is also worth understanding how the Apache configuration directives (Directory, Location, etc.) work.

SSH client config

For Internet-connected hosts, running SSH on a different port is a really good idea since it cuts down the noise of authentication attempts from bots looking for weak passwords. Running on a different port is not a substitute for a secure configuration (ie. no root login, key-only auth) - it is purely useful in cutting down log noise.

Unfortunately you have to remember which port you chose :) To minimise the hassle you should add entries in your client /etc/ssh/ssh_config:

Host nickname
    Port 43210
    HostName mysshserver
    User myuser

Now you can use "ssh nickname" and ssh will translate that to:

ssh -p 43210 mysshserver

Installing Windows 7 onto a netbook using USB

I wanted to install Windows 7 onto a netbook to replace an aging desktop as my only windows-on-metal box. This is a breeze with modern linux distros, but is of course far harder than it needs to be for windows.

I had a crack at using unetbootin on Linux with the Windows 7 ISO, despite a suspicious lack of mention of support for windows, and sure enough it didn't boot (unetbootin didn't give me any errors, it just didn't boot).

More googling turned up the Windows 7 USB/DVD Download Tool, which converts a Windows install ISO into a bootable USB installer - exactly what I wanted. After half an hour of downloading and installing dependencies (due to the lack of windows package management and a bizarre need to do the Genuine Windows check), I had the tool installed and it happily created a bootable USB for me.

This one worked perfectly, and dropped Windows 7 starter onto the netbook.

HOWTO call a python superclass method

Use the python 'super' method to call a superclass __init__:

class Foo(Bar):
    def __init__(self):
        super(Foo, self).__init__()

Working with Amazon S3 storage

I was using duplicity on Amazon S3 storage for backup, but gave it up because it was waaaay too slow (I believe the slowness was mainly duplicity, rather than network traffic or S3). So, time to delete the data from S3. I logged onto the Amazon S3 web interface, but found it pretty useless: I had hundreds of files to delete and there was no way to 'select all', or even delete a whole bucket at once. In fact, I couldn't even get the web interface to delete a single file for me. Seems like it is in Amazon's interest to make deleting data hard...

So I installed the 's3cmd' package on Ubuntu, which worked a treat. Setup with:

s3cmd --configure

Then to delete all the data in a bucket:

s3cmd del s3://data.bucket.name/*
s3cmd rb s3://data.bucket.name