Technical notes, my online memory

Really basic wireless survey with built-in linux tools

2012-01-16T22:04:00.000-08:00

What channel should I use for my access point? It seems like I'm always asking this question as people start new access points in my neighbourhood. Some wireless access points can pick a channel automatically, but my experience with this has been pretty poor. I suspect the algorithm is too simplistic to be of any use. A naive implementation might pick the first channel that has no other APs on it, which could be a terrible choice if the strongest interference signal is on the adjacent channel.

Ideally you want to pick the channel that gives you the best signal-to-noise ratio, but where there are heaps of APs at varying strengths in different parts of the house the choice is non-obvious. Sadly there doesn't seem to be a FOSS wireless survey tool like Fluke AirMagnet, and while you can use kismet to collect signal strength data you will have to do some analysis of the data yourself (my wireless card also didn't play well with kismet).

I decided to optimise for one location - where I use the laptop the most. So I set my AP to channel 1 and:

sudo iwlist wlan0 scanning > ch1.txt

Rinse and repeat for all the other channels (setting the channel on my AP is quick, but this might be impractical if yours is slow, requires re-auth etc.). Then compare the signal strengths:

grep --before-context=4 "myessid" ch*.txt

Pick the channel with the highest quality number (best signal strength).

Root a TMobile MyTouch 4G (HTC Glacier) to install Cyanogen

2012-01-16T21:42:00.000-08:00

This was my first time rooting an android phone. There are most probably better ways to do this, but here are my notes for better or worse. I followed these instructions, which were pretty good and I won't reproduce them, so consider this a diff. First install the jdk from the 'default-jdk' apt package.

Install the Android SDK, including the platform-tools component to get adb. If you try to use it without root you'll get a permissions error like this:

$ android-sdk-linux/platform-tools/adb devices
List of devices attached 
???????????? no permissions

Add this UDEV rule to /etc/udev/rules.d/51-android.rules. Note this gives all users write permissions:

SUBSYSTEM=="usb", SYSFS{idVendor}=="0bb4", MODE="0666"

Plug it back in:

$ android-sdk-linux/platform-tools/adb devices
List of devices attached 
XXXXXXXXXXXX device

Change the mode on the phone to 'charge-only', and unmount the SD card if it is mounted. This allows adb and subsequent rooting tools to access /sdcard. If you have it in disk drive mode you'll get errors like these from adb:

failed to copy 'Superuser.apk' to '/sdcard//Superuser.apk': Read-only file system

To do the cyangogen install you can use the 'ROM Manager' app by ClockworkMod, available in the android market.

Ubuntu system monitor in the taskbar post-unity (Natty and Oneiric)

2012-01-10T20:35:00.000-08:00

With the unity changes from Natty onward, the traditional Ubuntu gnome taskbar disappeared to be replaced by a set of application panel indicators. I was used to seeing my machine vitals (cpu, network, load etc.) displayed using system monitor in the taskbar. I was pleased to see that this can still be achieved by installing the 'indicator-multiload' package.

Depending on the version you have, the autostart might not work. If it doesn't, just:

mkdir ~/.config/autostart

and toggle the 'autostart' checkbox under preferences.

Broken BCM4313 "ping: sendmsg: No buffer space available"

2012-01-09T21:23:00.000-08:00

My Broadcom wireless card was broken on Ubuntu 11.10 Oneiric Ocelot. The card connected to my AP fine, but after a short time would barf. A ping to the AP would return a few successful replies followed by:

ping: sendmsg: No buffer space available

This is a known issue, and seems to affect quite a few people.

First I tried installing the proprietary broadcom driver from the 'broadcom-sta-common' package, but that was worse - it wouldn't even recognise the interface. I tried a whole bunch of later kernels and finally found one that worked: 3.1.8.

The ubuntu mainline kernel repo makes trying new kernels easy. Just download and install these debs:

linux-headers-3.1.8-030108_3.1.8-030108.201201061759_all.deb
linux-headers-3.1.8-030108-generic-pae_3.1.8-030108.201201061759_i386.deb
linux-image-3.1.8-030108-generic_3.1.8-030108.201201061759_i386.deb

Update: spoke too soon. This is still broken on 3.1.8 it just takes longer to manifest. I'm now also seeing a bunch of these messages:

kernel: [25244.951243] ieee80211 phy0: START: tid 1 is not agg'able

Update 2012-01-30: Now I'm also seeing lots of this:

ieee80211 phy0: brcms_c_prec_enq_head: No where to go, prec == 4

I'm going to up net.core.rmem and net.core.wmem as suggested in a comment, but that's really only treating the symptoms.

Firewire and DMA attacks on OS X

2012-01-06T16:18:00.000-08:00

I recently spent some time testing Firewire memory access on OS X to determine the current state of vulnerability to DMA attacks. There is a lot of prior art (Uwe Hermann has a good summary), and I was particularly interested in the claims of deficiencies in the DMA defences provided by Lion (Update: it looks like this was CVE-2011-3215 fixed in 10.7.2, which agrees with my findings).

To check for firewire and thunderbolt ports programmatically you can parse the output of system_profiler:

$ system_profiler SPFireWireDataType SPThunderboltDataType
FireWire:

    FireWire Bus:

      Maximum Speed: Up to 800 Mb/sec
       ...

Thunderbolt:

    MacBook Pro:

      Vendor Name: Apple, Inc.
      Device Name: MacBook Pro
      UID: 0x0001000A19999900
       ...

I used two MacBookPros, one running 10.6.8 and one running 10.7.2 fully patched as at 2011-12-29. Hardware specs:

$ system_profiler SPHardwareDataType

Model Name: MacBook Pro
Model Identifier: MacBookPro8,2
Processor Name: Intel Core i7
Processor Speed: 2 GHz
  ...

I used the pyfw-20041111 module to do the DMA using dumpmem which reads 256MB from 0x10000000:

python demo_dumpmem.py
hexdump -C cfffffffffffffff-10000000-20000000.memdump

and I also tested using ramdump.py and the libforensic1394 libraries, with the same results.

python ./ramdump.py 1024 f
SBP-2 not used, forcing 2048 byte pages.
[<forensic1394.device.Device object at 0xfffffffff>]
Attempting to access device 0: Apple Computer, Inc. Macintosh
Wrote file ramdump.bin.

Target 10.7.2, attacked from 10.6.8.

When targeting the 10.7.2 machine I could only access memory when the screen was unlocked, in all other scenarios I just received zeros. Note I tested with and without FileVault2 enabled, and the results were the same.

Status	DMA Capture Result
Logged in, not locked	Captured
Logged in, locked	0s
Login, log out, at login screen	0s
System booted, not logged in	0s
Switch user from fast user switching menu	0s

The cleartext password for the logged in user can be trivially retrieved with:

strings memdump | grep --after-context=4 longname | grep --after-context=1 password

Target 10.6.8, attacked from 10.7.2.

10.6.8 is the complete opposite, capture is possible in all cases (note that there is a NVRAM setting that can be used to defend against this attack without requiring a firmware password).

Status	DMA Capture Result
Logged in, not locked	Captured
Logged in, locked	Captured
Login, log out, at login screen	Captured
System booted, not logged in	Captured
Switch user from fast user switching menu	Captured

Get cleartext password with:

strings memdump | grep --after-context=3 managedUser | grep --after-context=1 password

OS X Open Firmware settings: use nvram security-mode to disable firewire DMA without a firmware password

2012-01-06T16:11:00.000-08:00

First some background. OS X uses Open Firmware, which is similar to a PC's BIOS. The nvram utility can be used to manipulate Open Firmware settings stored in NVRAM. Print NVRAM settings with:

nvram -p

For firewire DMA the important setting is security-mode. Here is an explanation of the different nvram security modes:

The "none" mode effectively disables security. The "command" mode just restricts the commands that may be executed to "go" and "boot". Additionally, under the "command" mode, the "boot" command may not have any arguments--that is, it will only boot the device specified in the boot device variable; no other command may be entered or any settings changed unless the password is supplied. Moreover, this password protection feature also applies to booting up with the option key held down (which allows you to choose from available bootable volumes through a built-in graphical user interface). Finally, in "full" mode, the machine is completely prohibited from booting until the password is entered.

To set a firmware password (puppet recipe):

nvram security-password=mypass
nvram security-mode=command

Note that anyone with root can read the hex encoded password with nvram security-password. To remove the password ('none' is the default security mode):

nvram -d security-password
nvram security-mode=none

When a firmware password is set, Firewire DMA is disabled. This can be abused to disable firewire DMA without setting a password - just by setting security-mode to something other than 'none'. This works for example (and will take effect after a reboot):

nvram security-mode=NONE

This causes the firewire controller to put itself into secure mode:

OSString * securityModeProperty = OSDynamicCast( OSString, options->getProperty("security-mode") );
if( securityModeProperty != NULL && strncmp( "none", securityModeProperty->getCStringNoCopy(), 5 ) != 0 ) {
  // set security mode to secure/permanent
  mode = kIOFWSecurityModeSecurePermanent;
}

Gather OS X hardware and software information with system_profiler

2012-01-06T10:53:00.000-08:00

system_profiler is a good way to gather lots of recon information about OS X hardware and software. Some examples

     system_profiler -detailLevel full
       Gives you absolutely everything

     system_profiler -listDataTypes
       Shows a list of the available data types.

     system_profiler SPSoftwareDataType SPNetworkDataType
       Generates a text report containing only software and network data.

     system_profiler SPThunderboltDataType SPFireWireDataType
       Just firewire and thunderbolt interfaces

OS X hidden folders, extended attributes, flags, and file ACLs

2011-12-29T12:37:00.000-08:00

On lion the ~/Library directory is hidden by default in Finder. This doesn't bother me normally, since I rarely use Finder and it is visible on the commandline, but I recently got stuck needing to submit logs from ~/Library using a web-based form.

There are some one-off tricks you can use to access it, but, as far as I can see, no equivalent of the windows 'show hidden files' setting in Finder.

To see extended attributes (indicated by trailing @), file ACLs (indicated by trailing +), and flags, use this:

$ ls -l@eOd Library/
drwx------@ 38 user  group  hidden 1292 Dec 28 16:02 Library/
 com.apple.FinderInfo   32 
 0: group:everyone deny delete

You can manipulate the 'hidden' flag (which seems to actually be implemented as a FinderInfo attribute) and others with chflags:

chflags nohidden Library

Which gives:

$ ls -l@eOd Library/
drwx------+ 38 user  group  - 1292 Dec 28 16:02 Library/
 0: group:everyone deny delete

Simple cocoa dialogs for commandline tools in OS X

2011-12-20T15:22:00.000-08:00

Got a commandline tool that needs to interact with a user? cocoaDialog is an app that can provide simple, good looking dialogs and can be invoked just by passing commandline options.

NSA response to USB-borne malware

2011-12-18T20:35:00.000-08:00

The Washington Post has an interesting article on the NSA's response to a USB-based malware infection on the US secret network in October 2008.

Ruby %x, system(), `backtick`, and puppet Facter::Util::Resolution.exec differences

2011-12-14T15:40:00.000-08:00

Below are three different ways of running a system command from ruby:

irb(main):014:0> %x('echo "sdfsd"')
sh: echo "sdfsd": command not found
=> ""
irb(main):015:0> system('echo "sdfsd"')
sdfsd
=> true
irb(main):016:0> `echo "sdfsd"`
=> "sdfsd\n"

%x doesn't give you a shell environment, so builtin commands like echo don't exist. system() gives you a shell, but doesn't save the output. Backticks give you a shell environment and return the stdout output.

In puppet if you're using Resolution.exec:

Facter::Util::Resolution.exec('echo "sdfsd"')

Things are not so obvious. Depending on the puppet version, you might get a shell environment and puppet is also doing some checking with 'which' that might result in a 'nil' return if your command is not a simple binary.

Convert group and user GeneratedUIDs to human-readable form on OS X

2011-12-09T16:31:00.000-08:00

When working with users and groups on OS X you frequently find that you need to lookup GUIDs to get their human-readable names. Here are a few examples of the fairly painful syntax (good candidates for bash aliases).

Get a group/user for a known GeneratedUID from local directory services:

dscl . -search /Users GeneratedUID 00052AE8-5000-6000-9007-666F6B666A66
dscl . -search /Groups GeneratedUID 00052AE8-5000-6000-9007-666F6B666A66

and from LDAP:

dscl /LDAPv3/ldap.company.com/ -search /Groups GeneratedUID 00052AE8-5000-6000-9007-666F6B666A66
dscl /LDAPv3/ldap.company.com/ -search /Users GeneratedUID 00052AE8-5000-6000-9007-666F6B666A66
ldapsearch -LLLx -b ou=group,dc=company,dc=com "apple-generateduid=00052AE8-5000-6000-9007-666F6B666A66" cn
ldapsearch -LLLx -b ou=people,dc=company,dc=com "apple-generateduid=00052AE8-5000-6000-9007-666F6B666A66" uid

Get a GeneratedUID for a known user/group:

dscl localhost -read /Search/Users/auser | grep -m1 GeneratedUID | cut -c15-
dscl localhost -read /Search/Groups/agroup | grep -m1 GeneratedUID | cut -c15-
ldapsearch -LLLx -b ou=group,dc=company,dc=com "cn=agroup" apple-generateduid
ldapsearch -LLLx -b ou=people,dc=company,dc=com "uid=auser" apple-generateduid

Ruby Time and DateTime: parse a string, compute the difference, simple right?

2011-12-09T12:52:00.000-08:00

Time in ruby is a confusing maze of suckiness.

The Ruby Time class doesn't include a method to convert a string into a Time object (like the C standard 'strptime'), despite having a Time.strftime function. What the?

It turns out that Time.parse and Time.strptime do exist, but confusingly they aren't in the core class, but in a separate library you have to require that overrides the class. WTF?

irb(main):001:0> Time.parse
NoMethodError: undefined method `parse' for Time:Class
 from (irb):1
irb(main):002:0> require 'time'
=> true
irb(main):003:0> Time.parse
ArgumentError: wrong number of arguments (0 for 1)
 from (irb):3:in `parse'
 from (irb):3

So it seems that the best way to read a time from a file and compare it to the current time is the following, which gives you an answer in seconds:

irb(main):010:0> a=Time.parse('2011-01-10 12:34:00 UTC')
=> Mon Jan 10 12:34:00 UTC 2011
irb(main):012:0> b=Time.now.utc
=> Fri Dec 09 20:47:32 UTC 2011
irb(main):013:0> b-a
=> 28800812.788154

The only downside of this is that the library never fails, it tries really hard to give you a Time object back, to the point where it will just return the current time if the string is rubbish:

irb(main):027:0> Time.parse('asdfasdfas')
=> Fri Dec 09 14:02:40 -0800 2011

whereas DateTime is actually more sensible:

irb(main):028:0> DateTime.parse('asdfasdfas')
ArgumentError: invalid date
 from /usr/lib/ruby/1.8/date.rb:1576:in `new_by_frags'
 from /usr/lib/ruby/1.8/date.rb:1621:in `parse'
 from (irb):28
 from  :0

So my final solution was:

  begin
    DateTime.parse(time_string)
  rescue ArgumentError
    return 'UNKNOWN'
  end
  earlier = Time.parse(time_string)

  now = Time.now.utc
  hours = (now-earlier)/3600
  return hours

Read on for an attempt to do the same with DateTime, which is a terrible, terrible API.

DateTime has a simple way to ingest strings (and also has a strptime):

irb(main):015:0> require 'date'
=> false
irb(main):023:0> a=DateTime.parse('2011-12-09 00:00:00 UTC')
=> #
irb(main):024:0> puts a
2011-12-09T00:00:00+00:00
=> nil

But you can't take a difference between Time and DateTime:

irb(main):027:0> a=DateTime.parse('2011-12-01 00:00:00 UTC')
=> #
irb(main):028:0> b=Time.now
=> Fri Dec 09 12:10:40 -0800 2011
irb(main):029:0> b-a
TypeError: can't convert DateTime into Float
 from (irb):29:in `-'
 from (irb):29
 from  :0

Subtracting two DateTimes works but gives you an answer that is a rational number of days (?!):

irb(main):031:0> b-a
=> Rational(76394798789, 8640000000)
irb(main):032:0> c=b-a
=> Rational(76394798789, 8640000000)
irb(main):033:0> puts c
76394798789/8640000000
=> nil
irb(main):034:0> puts c.to_f
8.8419906005787
=> nil

Since a fractional number of days is pretty useless you can then convert this to an array of [hours,minutes,seconds,frac]:

irb(main):036:0> DateTime.day_fraction_to_time(c)
=> [212, 12, 27, Rational(98789, 8640000000)]

Why the subtraction doesn't return another type of time object is beyond me.

auditd on OS X

2011-12-07T08:51:00.000-08:00

Auditd gets a limited description in the Snow Leopard Security Config doc. I'm posting a quick summary here, and will update it as I learn more.

auditd rules are kept in /etc/security. The audit_control rules apply to all users and audit_user allows for per-user rules.

Audit logs are stored in binary format in /var/audit/logstarttime.logfinishtime and can be read with:

praudit /var/audit/20111018000205.20111018000916

Create an array of identical strings in python

2011-11-17T16:15:00.001-08:00

To create an array of identical strings in python (sometimes useful for testing):

In [12]: list("sdfsd" for x in range(1,4))
Out[12]: ['sdfsd', 'sdfsd', 'sdfsd']

Update: a commenter points out a better way. I knew you could do

'a'*5

but didn't think to try this. Thanks!

A shorter version:

In [16]: ["sdfsd"]*5
Out[16]: ['sdfsd', 'sdfsd', 'sdfsd', 'sdfsd', 'sdfsd']

And it's faster too....

$ python -mtimeit -n10000 "list('sdfsd' for i in xrange(1000))"
10000 loops, best of 3: 69.7 usec per loop
$ python -mtimeit -n10000 "['sdfsd']*1000"
10000 loops, best of 3: 5.66 usec per loop

OS X groups, subgroups and GeneratedUIDs

2011-11-16T10:41:00.000-08:00

OS X groups are more confusing than they need to be, due to the use of UUIDs. First, lets create a group to test with and include a single user (test1):

$ sudo dseditgroup -o create -t group test00
$ sudo dseditgroup -o edit -a test1 -t user test00
$ dseditgroup -o read test00
dsAttrTypeStandard:GeneratedUID -
  C110609F-0C0A-42E7-B663-D93BA03ECDAC
dsAttrTypeStandard:RecordName -
  test00
dsAttrTypeStandard:AppleMetaNodeLocation -
  /Local/Default
dsAttrTypeStandard:GroupMembers -
  6D456134-85E6-4912-88C7-7C6139057B9B
dsAttrTypeStandard:RecordType -
  dsRecTypeStandard:Groups
dsAttrTypeStandard:NestedGroups -
dsAttrTypeStandard:PrimaryGroupID -
  505
dsAttrTypeStandard:GroupMembership -
  test1

Note that we have GroupMembership that includes test1 and the GroupMembers attribute lists the GUID for the test1 user. Neither of these attributes tell us about groups that are members of this group. To see how they are stored, we will add a subgroup:

$ sudo dseditgroup -o edit -a testsubgroup -t group test00
$ dseditgroup -o read test00
dsAttrTypeStandard:GeneratedUID -
  C110609F-0C0A-42E7-B663-D93BA03ECDAC
dsAttrTypeStandard:RecordName -
  test00
dsAttrTypeStandard:AppleMetaNodeLocation -
  /Local/Default
dsAttrTypeStandard:GroupMembers -
  6D456134-85E6-4912-88C7-7C6139057B9B
dsAttrTypeStandard:RecordType -
  dsRecTypeStandard:Groups
dsAttrTypeStandard:NestedGroups -
  6B3D7A4A-A795-4569-8640-2105F73EEA6B
dsAttrTypeStandard:PrimaryGroupID -
  505
dsAttrTypeStandard:GroupMembership -
  test1

Note that the subgroup UUID is now listed under NestedGroups, but the human-readable name is not, presumably because apple wanted to make this as painful as possible. There isn't a nice way to map UUID to a recordname, but searching in the local node does work (although it seems to ignore the output formatting options):

$ dscl -plist . -search /Groups GeneratedUID 6B3D7A4A-A795-4569-8640-2105F73EEA6B
testsubgroup  GeneratedUID = (
    "6B3D7A4A-A795-4569-8640-2105F73EEA6B"
)

So the outcome of this is that to code to answer a simple question: 'who are the members of this group?' is actually really complicated.

There is however a way to check if a user is a member of a group, and it will follow nested group relationships:

$ sudo dseditgroup -o edit -a test2 -t user testsubgroup
$ dseditgroup -o checkmember -m test2 test00
yes test2 is a member of test00

Using netgroups to control login and admin privileges on OS X

2011-11-15T16:11:00.000-08:00

I wanted to use netgroups defined in /etc/netgroup to control:

Login access via access_loginwindow
OS X admin privs via the admin local group
sudo access via /etc/sudoers

Unfortunately I couldn't get it to work (on 10.6), but I'll document here for my own sake and the rest of the internet :) I suspect this was because I wanted to just use the flat file, without any NIS server, and OS X was expecting to do lookups to refresh the information in the file.

Run Directory Utility

open /System/Library/CoreServices/Directory\ Utility.app/

and tick 'BSD Flat File and NIS' and inside there tick 'Use User and Group records in BSD Local node'. Interestingly this is gone on Lion - it is now just NIS and there is no option for BSD local node...

Assuming you have some entries in /etc/netgroup:

smalltest (-,auser,)

You should be able to see them with dscl:

$ dscl localhost -read /BSD/local/NetGroups/smalltest
dsAttrTypeNative:triplet: -,auser,
AppleMetaNodeLocation: /BSD/local
RecordName: smalltest

And should theoretically be able to use them. I tried this for sudo, and although it passed syntax checking it just didn't work.

+smalltest ALL=(ALL) ALL

And I couldn't get dseditgroup to recognise the netgroup for use in access_loginwindow:

$ sudo dseditgroup -o edit -a smalltest -t group com.apple.access_loginwindow
Group not found.

OS X Service ACLs com.apple.access_*

2011-11-15T15:47:00.000-08:00

To place ACLs on OS X services, you can create groups like this:

sudo dseditgroup -o create -t group com.apple.access_loginwindow
sudo dseditgroup -o edit -a auser -t user com.apple.access_loginwindow

which in this case will limit who can login at the console (i.e. loginwindow) to just the user 'auser'. You can check the membership with:

$ dscl . -read /Groups/com.apple.access_loginwindow
AppleMetaNodeLocation: /Local/Default
GroupMembership: auser
PrimaryGroupID: 504
RecordName: com.apple.access_loginwindow
RecordType: dsRecTypeStandard:Groups

Similar groups will control access to other services such as com.apple.access_ssh, com.apple.access_screensharing.

Compiling C libraries for use in python modules on OS X: specify a target architecture

2011-11-14T15:40:00.000-08:00

My pycurl installation was failing with these errors:

$pip install pycurl

[snip]

assembler (/usr/bin/../libexec/gcc/darwin/ppc/as or /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not installed

Installed assemblers are:

/usr/bin/../libexec/gcc/darwin/x86_64/as for architecture x86_64

/usr/bin/../libexec/gcc/darwin/i386/as for architecture i386

The solution was to specify an architecture like this:

env ARCHFLAGS="-arch x86_64" pip install pycurl

OS X logging: asl.conf, syslog.conf, syslog master filters and log rotation

2011-11-10T14:30:00.000-08:00

The apple syslog daemon (/System/Library/LaunchDaemons/com.apple.syslogd.plist) takes configuration from 3 different sources *groan*.

asl.conf

asl.conf (the apple syslog configuration file) controls what data is stored in the apple syslog binary databases under /private/var/log/asl. It is powerful and allows everything from filtering:

# save everything from emergency to notice
? [<= Level notice] store

to access control. This restricts read access to uid 0 (root) and gid 80 (admin):

# authpriv messages are root/admin readable
? [= Facility authpriv] access 0 80

syslog.conf

In addition, and independent of, asl.conf, syslogd also reads syslog.conf which is the familiar unix/linux/bsd style syslog configuration file. This allows you to write the same log lines to plaintext log files in /var/log (or wherever) that also get stored by asl in binary form as above.

Syslog master filters

As if that wasn't enough, the syslog daemon itself also has a global master filter rule, which you can inspect with:

$ syslog -c 0
Master filter mask: Debug

AND you can set per-process filters:

$ syslog -c syslogd
ASL Data Store filter mask: Emergency - Debug

Reading logs

To read asl logs, just use the syslog command. It actually comes with some nifty filtering of its own. For example this command shows log lines sent by login for the past 2 hours:

syslog -k Sender login -k Time ge -2h

To help you formulate queries like the above you can see the raw key value pairs using:

syslog -F raw

There is a gotcha in the default apple configuration which has lines like:

# redirect com.apple.message.domain to /var/log/DiagnosticMessages
? [T com.apple.message.domain] store_dir /var/log/DiagnosticMessages

# redirect com.apple.performance* messages to /var/log/performance
? [A= Facility com.apple.performance] store_dir /var/log/performance

# redirect com.apple.eventmonitor* messages to /var/log/eventmonitor
? [A= Facility com.apple.eventmonitor] store_dir /var/log/eventmonitor

Which means just using the syslog command as above doesn't give you all the logs, including stuff like apple updates:

$ syslog  | grep 'downloading "Thunderbolt Software Update, 1.0"'

To look at the DiagnosticMessages log use:

$ syslog -d /var/log/DiagnosticMessages/ | grep 'downloading "Thunderbolt Software Update, 1.0"'
Software Update[1865] : SWU: downloading "Thunderbolt Software Update, 1.0"

To read the BSD logs just go look at the files in /var/log.

Log Rotation

Log rotation configuration is in

/etc/newsyslog.conf

and has configuration lines that look like this (ie. completely different to linux /etc/logrotate.d):

# logfilename          [owner:group]    mode count size when  flags [/pid_file] [sig_num]
/var/log/appfirewall.log  640  5     1000 *     J

See the newsyslog.conf man page for the details.

OS X Terminal screwed up command history in iPython

2011-11-08T19:11:00.000-08:00

I was getting some weird behaviour in ipython (0.11) in a OS X (10.6.8) Terminal window. When I used the up arrow to access the commandline history, all long lines (and even some short lines) were screwed up - the text wrapping was wrong. Others have noticed the same thing and it turns out the solution is to re-install readline (which requires Xcode to compile it):

sudo pip install readline

Equivalent of 'watch' command for OS X

2011-11-07T14:27:00.000-08:00

'watch' doesn't exist inbuilt for OS X like it does for linux. Someone has actually implemented it and it is available from fink, but here is a bash function that uses a simple while loop to do basically the same thing. Drop this into bash_aliases and you now have a simple version of watch.

watch () {
  while [ 1 ]; do "${@}" ; sleep 2 ; done
}

HOWTO write an OS X (seatbelt) sandbox profile

2011-11-07T14:17:00.000-08:00

To create a simple seatbelt sandbox profile, start with a trace by saving this as trace.sb:

(version 1)
(trace "/tmp/traceout.sb")

Then

sandbox-exec -f trace.sb binary_to_be_sandboxed
sandbox-simplify /tmp/traceout.sb > ./tracesimple.sb

Simplify crunches down the verbose log into a more compact profile. Edit it (you especially want to remove any spurious dtrace lines that are artifacts of the capture process) and run the binary in its sandbox:

sandbox-exec -f tracesimple.sb binary_to_be_sandboxed

This blog has a simple shell script to automate the above process, but you probably want to manually inspect and edit your profile before using it for real.

You can see a bunch of built-in sandboxes in

/usr/share/sandbox

and you can view which running processes are sandboxed by adding the sandbox column to the activity monitor gui.

OS X kernel panic: "You need to restart your computer"

2011-11-07T13:24:00.000-08:00

This is the OS X kernel panic screen:

Apple has some information on how you can configure OS X to send kernel core dumps over TCP and some information on how to debug panics. You can also find a panic log at

/Library/Logs/DiagnosticReports/Kernel_time_host.panic

NFS shares

2011-10-28T13:18:00.000-07:00

Show NFS shares exported from a host (can also point to localhost):

showmount -e 192.168.1.1

/etc/exports line on linux (share /blah with 192.168.1.2)

/blah               192.168.1.2(ro)

On the mac the options format is different. As with linux, root_squash to nobody is the default.

/blah -ro 192.168.1.2

Installing pip and virtualenv on OS X

2011-10-25T12:56:00.000-07:00

Download and install the appropriate easy_install egg.

sudo sh setuptools-0.6c11-py2.X.egg

and from then on it is the same as for linux

sudo easy_install pip
sudo pip install virtualenv

Create a virtualenv instance:

virtualenv ENV

and you'll get something like:

Could not call install_name_tool -- you must have Apple's development tools installed

So you need to go off and install XCode, which will probably take a while (4.2 is 1.7 G).

Once you have Xcode, things should be good:

$ virtualenv ENV
New python executable in ENV/bin/python
Installing setuptools............done.
Installing pip...............done.
$ source ENV/bin/activate

OSX launchd and inetd services

2011-10-20T15:20:00.000-07:00

launchd is roughly equivalent to cron, init.d, and xinetd all rolled into one.

Use it by dropping a plist into one of these directories:

/Library/LaunchDaemons runs as root on system startup
/Library/LaunchAgents runs as the user on login
~/Library/LaunchAgents runs as the user for the specific user login

There is also /System/Library/Launch{Agents,Daemons} which is for the jobs installed by the OS. Use the directories above for any custom jobs.

List launchd jobs with

launchctl list

Start:

sudo launchctl load /Library/LaunchDaemons/com.myorg.somejob.plist

Stop:

sudo launchctl unload /Library/LaunchDaemons/com.myorg.somejob.plist

There are many different options for jobs, see the launchd.plist man page for a complete list. A simple inetd service might look something like this:

<dict>
 <key>Disabled</key>
 <false/>
 <key>Label</key>
 <string>com.myorg.somejob</string>
 <key>Program</key>
 <string>/usr/local/sbin/somejob</string>
 <key>Sockets</key>
 <dict>
  <key>Listeners</key>
  <dict>
   <key>SockServiceName</key>
   <string>9999</string>
  </dict>
 </dict>
 <key>inetdCompatibility</key>
 <dict>
  <key>Wait</key>
  <false/>
 </dict>
</dict>

/usr/local/sbin/somejob will be called whenever there is a connection on port 9999, and you can interact with the socket by reading and writing STDIN/STDOUT as normal for inetd.

Force MCX refresh - apply policy without a logout/reboot

2011-10-19T16:57:00.000-07:00

If you are managing settings with MCX (e.g. like this), you want your MCX settings to apply as soon as they are pushed to the machine. Unfortunately this doesn't happen automatically. They get stored in the local directory services node, which you can see here:

dscl . -mcxread /Computers/local_computer

But they don't get propagated to the relevant plists under managed preferences until a logout or reboot happens. You can check the managed preferences plists like this:

defaults read /Library/Managed\ Preferences/com.apple.loginwindow

Fortunately on 10.6 and later, you can run mcxrefresh to force this propagation without a reboot:

sudo mcxrefresh -n myusername

Puppet expansion of facts inside command strings requires double quoted string

2011-10-19T16:50:00.000-07:00

Small gotcha for puppet facts. If you want a puppet fact to be expanded inside a quoted string you need to use double quotes (i.e. a ruby-style format string).

  exec { 'something':
    # This has to be a double quote for the fact to get expanded
    command     => "/usr/bin/something ${my_custom_fact}",
    refreshonly => 'true',
  }

Chrome Extensions vs. Apps vs. Themes and what it looks like in your extensions directory

2011-10-12T08:25:00.000-07:00

If you look in your chrome extensions directory on disk, things are a little confusing.

There are extensions, which you can see listed at about:extensions.
There are apps, which can be either packaged or hosted and will contain an app launch stanza like this in the manifest.json
```
   "app": {
      "launch": {
         "web_url": "https://mail.google.com/mail/mu/?mui=ca"
      },
      "urls": [ "https://mail.google.com/mail/mu/" ]
   },
```
These don't seem to appear anywhere in the UI apart from when you open a new tab (there is no about:apps, see about:about for a full list of what is available).

Themes, which are packaged in a similar way, but don't contain javascript or HTML. The manifest.json will have entries like this:

   "theme": {
      "colors": {
         "bookmark_text": [ 105, 96, 87 ],
         "frame": [ 92, 82, 73 ],
         "ntp_background": [ 235, 234, 211 ],
         "ntp_link": [ 105, 96, 87 ],
         "ntp_section": [ 158, 163, 139, 1 ],
         "ntp_section_link": [ 105, 96, 87 ],
         "ntp_section_text": [ 105, 96, 87 ],
         "ntp_text": [ 105, 96, 87 ],
         "tab_background_text": [ 255, 255, 255 ],
         "tab_text": [ 255, 255, 255 ],
         "toolbar": [ 158, 163, 139 ]
      },

Also, just figuring out the name of extensions can be confusing. In the manifest.json you will often see:

"name": "__MSG_some_name__",

Which means you need to go look in the relevant locale file like _locales/en/messages.json to find the actual name, it will look like this (note MSG and underscores stripped):

{
   "some_name": {
      "message": "My Rad Extension"
   }
}

The exception is if it is a theme, the messages.json file will look like:

{"themeName": {"message": "Earthy", "description": "theme name"}}

Except it also has a UTF-8 Byte Order Mark prepended to the start of the file, which is the three bytes 'ef bb bf'. If you're trying to parse it with an external JSON parser, those bytes might cause it to fail.

Using OS X directory services (dscl, dseditgroup) in single user mode

2011-10-07T12:36:00.000-07:00

To access directory services you first need a writable root:

mount -uw /

Then you need the daemon loaded. On snow leopard:

launchctl load /System/Library/LaunchDaemons/com.apple.DirectoryServices.plist

And on Lion:

launchctl load /System/Library/LaunchDaemons/com.apple.opendirectoryd.plist

On lion you might see an error about DirectoryServices.plist not existing when you connect with dscl, it can be ignored.

Transparent screen lock for monitoring displays

2011-10-06T16:36:00.000-07:00

There are situations where a transparent screen lock is useful: i.e. monitoring displays, build status etc. On ubuntu I have used xtrlock, and I'm looking for something similar for OS X.

OS X and DNS

2011-10-04T15:34:00.000-07:00

If you look at /etc/resolv.conf on a mac and expect to see the definitive DNS config, you are being misled. While some tools might use this config, most applications will use the mDNSResponder (the name is confusing, it is also used for unicast DNS) config maintained by configd. You can see this with:

scutil --dns

You can check out the mDNSReponder config with:

defaults read /System/Library/LaunchDaemons/com.apple.mDNSResponder

Which, incidentally, is the place to add -NoMulticastAdvertisements to disable bonjour.

You can make a query through mDNSResponder with:

$ dscacheutil -q host -a name slashdot.org
name: slashdot.org
ip_address: 216.34.181.45

$ dscacheutil -q host -a ip_address 216.34.181.45
name: slashdot.org
alias: 45.181.34.216.in-addr.arpa 
ip_address: 216.34.181.45

To see the DNS cache, dump the state of mDNSResponder into system.log with:

sudo killall -INFO mDNSResponder

You can also turn on (very verbose) DNS logging into system.log:

sudo killall -USR1 mDNSResponder

Or even turn on packet capture with:

sudo killall -USR2 mDNSResponder

You can see the other (non-DNS) caching in the local directory service with:

dscacheutil -statistics
dscacheutil -cachedump -entries

Automount and directory service search paths on OS X

2011-10-03T23:02:00.000-07:00

The OS X directory service uses the search path to retrieve owner, group, and automount information. A config looks like this:

$ dscl localhost -read /Search
CSPSearchPath: /Local/Default /BSD/local /LDAPv3/main.ldap.example.com
DHCPLDAPDefault: off
LSPSearchPath: /Local/Default /BSD/local
NSPSearchPath: /Local/Default /BSD/local
ReadOnlyNode: ReadOnly
SearchPath: /Local/Default /BSD/local /LDAPv3/main.ldap.example.com
SearchPolicy: dsAttrTypeStandard:CSPSearchPath

The SearchPolicy tells you which one will be used - in this case the Custom Search Path or CSPSearchPath (LSPSearchPath and NSPSearchPath are read-only), which will look in /Local/Default then /BSD/local then in the LDAP server main.ldap.example.com. This is roughly the equivalent of nsswitch.conf on linux. You can also see the same information in Directory Utility:

"/System/Library/CoreServices/Directory Utility.app/Contents/MacOS/Directory Utility"

This app is used to configure the services that the mac will talk to for authentication (LDAP, NIS, Local, AD etc.), you can access that config on the command line with:

sudo defaults read /Library/Preferences/DirectoryService/DirectoryService

Automounts are kept in /etc/auto_master and you can get it to look in LDAP for the mounting info by adding '+auto_master' to the config. It will look for an automount map (see /etc/autofs.conf for more configuration options):

dn: ou=auto.master,ou=automount,ou=admin,dc=example,dc=com
ou: auto.master
objectClass: top
objectClass: automountMap

And shares identified by cn:

dn: cn=/home,ou=auto.master,ou=automount,ou=admin,dc=example,dc=com
cn: /home
objectClass: top
objectClass: automount
automountInformation: -nosuid home.nfs:/home

You can dump a full list of shares with a LDAP search like this:

ldapsearch -LLLx -b ou=auto.master,ou=automount,ou=admin,dc=example,dc=com

or via dscl like this:

dscl localhost -readall /LDAPv3/main.ldap.example.com/Automount

LDAP search queries

2011-09-16T14:32:00.000-07:00

Some quick examples of common LDAP search queries.

Search for a particular user:

ldapsearch -LLLx "uid=myuser"

If your LDAP database is giant, you might want to limit that search to just the people tree:

ldapsearch -LLLx -b ou=people,dc=myorg,dc=com "uid=myuser"

Find a netgroup:

ldapsearch -LLLx -b ou=netgroup,dc=myorg,dc=com "cn=mymachine.myorg.com"

Wildcards also work:

ldapsearch -LLLx -b ou=netgroup,dc=myorg,dc=com "cn=*fred*.myorg.com"

Regular group:

ldapsearch -LLLx -b ou=group,dc=myorg,dc=com "cn=sysadmins"

Reading MCX local_computer policies on the command line

2011-09-16T12:05:00.000-07:00

Mactech has an article on using local mcx policies. To read them:

dscl . -mcxread /Computers/local_computer

and you can also see the full contents of the local_computer record like this:

defaults read /var/db/dslocal/nodes/Default/computers/local_computer

Reading and modifying OS X plist files on the command line

2011-09-14T18:32:00.000-07:00

OS X uses plist files to store configuration information - you can think of them as OS X's version of the windows registry.

To read a plist you can use:

defaults read /Library/Preferences/com.apple.CrashReporter

and similarly to write a value to a plist

sudo defaults write /Library/Preferences/com.apple.CrashReporter SomeKey -bool TRUE
sudo defaults write /Library/Preferences/com.apple.CrashReporter SomeKey -string "somevalue"

and delete

sudo defaults delete /Library/Preferences/com.apple.CrashReporter SomeKey

You can also use plutil to dump an XML version to stdout:

plutil -convert xml1 -o - filename.plist

as well as convert between binary and XML formats. Xcode also ships with a plist editor.

Managing OS X users, groups, and access control from the commandline

2011-09-14T17:50:00.000-07:00

User info

dscl . read /Users/testing

Group info

dscl . read /Groups/admin

Create a user

sudo dscl . -create /Users/testing

Add a user to the admin group

sudo dseditgroup -o edit -a testing -t user admin

Remove a user from the admin group

sudo dseditgroup -o edit -d testing -t user admin

Delete a user

sudo dscl . -delete /Users/testing

Redirection of stdout and stderr

2011-09-09T08:31:00.000-07:00

Some notes on redirection, this article has a good summary. I hold these patterns in my head, but had long since forgotten what the syntax actually meant. Send stdout and stderr to the same file:

./generatesomeoutput &> some_file

Send stderr to stdout:

./generatesomeoutput 2>&1

which is filedescriptor '2' (stderr) to file descriptor '1' (stdout), and the '&' indicates that '1' is a filedescriptor not a filename.

Mounting DMG images, the hard way

2011-09-07T12:53:00.000-07:00

Opening a DMG in the finder is a fast way to get it mounted. The manual steps are broken down as follows:

Attach the disk image (if you omit the 'nomount' flag it will do the mounting automatically, I'm stopping that for demonstration purposes)

root# hdiutil attach -nomount testing.dmg
/dev/disk1           Apple_partition_scheme          
/dev/disk1s1         Apple_partition_map             
/dev/disk1s2         Apple_Driver_ATAPI              
/dev/disk1s3         Apple_HFS

The disk now exists:

dhcp-172-26-92-208:images root# diskutil list
[...snip...]
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     Apple_partition_scheme                        *4.2 GB     disk1
   1:        Apple_partition_map                         30.7 KB    disk1s1
   2:         Apple_Driver_ATAPI                         2.0 KB     disk1s2
   3:                  Apple_HFS Mac OS X                4.2 GB     disk1s3

And you can mount all partitions like this:

diskutil mountDisk /dev/disk1

or just one with:

diskutil mount /dev/disk1s3

Since we didn't specify targets, they will appear under /Volumes.

HOWTO Convert an Apple disk image (DMG) to ISO

2011-09-07T11:51:00.001-07:00

hdiutil convert /path/to/filename.dmg -format UDTO -o /path/to/savefile.iso

HOWTO: dd a disk and watch progress

2011-09-04T17:36:00.000-07:00

On linux, dd a disk and watch progress with this command. $! is the PID of the most recent background command:

dd if=/dev/zero of=/dev/null bs=512k&

while [ $! ]; do kill -USR1 $! && sleep 10 ; done

OS X uses a different signal:

dd if=/dev/zero of=/dev/null bs=512k&

while [ $! ]; do kill -SIGINFO $! && sleep 10 ; done

python datetime, strptime, strftime: working with datetime strings

2011-09-01T15:22:00.000-07:00

I frequently find myself wanting to convert between a datetime object and a string representation. Here is a quick code example of swapping between the two:

t=datetime.datetime.now()

astring = t.strftime("%Y-%m-%d %H:%M:%S")

new_t = datetime.datetime.strptime(astring, "%Y-%m-%d %H:%M:%S")

Is this Mac OS X app compiled with ASLR?

2011-08-11T09:58:00.000-07:00

To determine if an OS X Mach-o binary is compiled with ASLR, look for the Position Independent Executable (PIE) flag in the Mach header:

$ otool -hv MobileSafari 



MobileSafari:

Mach header

      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags

   MH_MAGIC     ARM         V7  0x00     EXECUTE    40       4560   NOUNDEFS DYLDLINK TWOLEVEL PIE

Python: add an entry to an dict of arrays or create one if it doesn't exist

2011-08-09T12:52:00.000-07:00

I frequently am working with dictionaries of arrays and want to add an element to an existing array, or create one if it doesn't exist. There is a neat, if not particularly easy to understand, way to do this with setdefault:

result = {}

for thisthing in alist:

   result.setdefault(thisthing.somekey, []).append(thisthing.somevalue)

So you will end up with something like:

{'key1':[value1, value2], 'key2': [value3]}

HOWTO install OS X from a system image

2011-07-29T13:46:00.000-07:00

To install OS X from a system image, first partition the disk using diskutil:

diskutil partitionDisk disk0 2 GPT JHFS+ os 90% JHFS+ installer 10%

Download the install image onto the os partition:

curl -o /Volumes/os/osximage.dmg -L http://mydomain.com/osximage.dmg

and restore it to the installer partition:

asr restore --erase --noprompt --noverify --target /dev/disk0s3 --source /Volumes/os/osximage.dmg

Reboot!

64-bit errors in real programs

2011-07-26T22:24:00.000-07:00

There is a great article on Intel's website about real programming errors in C/C++ that surfaced as bugs when the code was compiled and run as a 64-bit program. A fair proportion involve some sort of type casting that makes assumptions about pointer size, and many are subtle bugs that will only be triggered when memory allocation is > 4GB.

As 64-bit becomes the new norm I think we'll see a lot of buffer overflows and memory corruption bugs as a consequence of errors like these.

Mac boot options

2011-07-21T09:58:00.000-07:00

To enter single-user, hold down ⌘-S at boot.

To get a menu of boot options (netboot, USB etc.) hold down option during boot.

See the full list of boot options.

Remount a hfs partition read-write on a mac

2011-07-21T09:54:00.000-07:00

To remount a partition read-write on linux you would use:

mount -o remount,rw /dev/disk /mnt/point

On a mac the equivalent is:

mount -u -w /dev/disk /mnt/point

In single user mode to get a writable root:

mount -uw /

Ruby learnings

2011-07-21T09:51:00.000-07:00

I had a brief foray into Ruby land recently, so thought I would record some lessons learnt here.

Interactive ruby is like the really poor cousin to ipython. Commandline completion is not on by default, you want to enable it using:

alias irb='irb -r irb/completion'

and apparently the only way to get syntax highlighting is to install the wirble gem.

There are a bunch of ways to run shell commands. The one I used was %x:

cmd = "dscl . -read #{file} #{key} 2> /dev/null"
  output = %x{#{cmd}}

I was looking for a equivalent of the python string function 'startswith'. The ruby equivalent actually has a ? in the function name (WTF?):

if output.start_with?("#{key}: ")

Useful Mac keyboard shortcuts

2011-07-12T17:32:00.000-07:00

General
Fn-arrowkeys	PgUp and PgDown
Control-Shift-⏏	Lock screen (sleep)
⌘-Space	Open Spotlight
Terminal
ESC-.	Paste last word of the last command under the cursor (equivalent of Alt-. in linux)
⌘-Shift-arrowkeys	Left right arrows switch between tabs
Chrome (full list of shortcuts)
⌘-Option-arrowkeys	Let right arrows to switch between tabs.

HOWTO enable remote desktop for windows 7 users

2011-07-06T14:30:00.000-07:00

The technet article has all the details on how to configure remote desktop, but the tl;dr to enable it for a user is:

Control Panel -> System -> Remote Settings -> Select Users

HOWTO list local administrators on the Windows commandline

2011-07-06T14:27:00.000-07:00

To get a list of local admins:

net localgroup administrators

Installing the shitty Amazon MP3 downloader on Ubuntu Lucid

2011-06-13T18:36:00.000-07:00

While it sucks to have to use a MP3 downloader app at all, it's good Amazon actually has a linux version (it wasn't always the case). But it is a shame they did such a bad job of it. They wrote the downloader for Ubuntu Jaunty (now a number of releases old) and haven't updated it since.

The correct way to distribute this app would have been to provide a PPA you could put in your apt sources, which would be updated for each new version of Ubuntu, and allow the full package management capability. By using .debs any dependencies need to be manually satisfied, and you can't push updates or bug-fixes, although it looks like Amazon has written that functionality into the program itself. The deb also depends on very specific versions of libboost, rather than just specifying a minimum version number, which would allow it to work on later releases.

So installing it on lucid, or any other version is a bit of a pain. Here is one solution, that grabs the original dependencies from launchpadlibrarian and installs them:

wget https://launchpadlibrarian.net/26959932/libboost-signals1.34.1_1.34.1-16ubuntu1_i386.deb \
https://launchpadlibrarian.net/26959936/libboost-thread1.34.1_1.34.1-16ubuntu1_i386.deb \
https://launchpadlibrarian.net/26959922/libboost-iostreams1.34.1_1.34.1-16ubuntu1_i386.deb \
https://launchpadlibrarian.net/26959918/libboost-filesystem1.34.1_1.34.1-16ubuntu1_i386.deb \
https://launchpadlibrarian.net/26959916/libboost-date-time1.34.1_1.34.1-16ubuntu1_i386.deb \
https://launchpadlibrarian.net/26959928/libboost-regex1.34.1_1.34.1-16ubuntu1_i386.deb \
https://launchpadlibrarian.net/34165098/libicu40_4.0.1-2ubuntu2_i386.deb
sudo dpkg -i *.deb
sudo dpkg -i amazonmp3.deb

Contacting file owners: using find, awk, and xargs to creating lists of files and operate on them

2011-05-24T14:21:00.000-07:00

Say you want to contact a bunch of file owners to do some cleanup or housekeeping. This will get you a list of files sorted by owner.

find . -type f -name "*blah*" "%u %M %g %y %p\n" | sort

But we can do better. Here is a neat little awk command that will drop filenames into txt files named after the user.

find . -type f -name "*blah*" "%u %M %g %y %p\n" | sort | awk 'BEGIN {FS=" "} { print $5 >>"/home/me/temp/"$1".txt" }'

So the end result is that /home/me/temp/mike.txt contains a list of files that mike owns. You can then tell mike to do something with those files (in this example, delete) using xargs like this:

xargs -a mike.txt rm -f

How do you tell mike? well you can get a list of addresses to drop into your mail client:

find . -type f -name "*blah*" "%u@company.com\n" | sort | uniq

then drag and drop the USERNAME.txt files into the mail. With a bit more effort you could automate the sending mail step too.

Adding other arguments when using *args and **kwargs in python

2011-05-20T18:27:00.000-07:00

Sometimes you need to use *args and **kwargs to pass your positional and keyword args to another class. But what if your class takes an additional argument you don't want to pass to the super class? Here are a couple of options.

Add a new positional argument

  def __init__(self, a_list, *args, **kwargs):
    super(MyClass, self).__init__(*args, **kwargs)
    self._mylist = a_list

Add a new keyword argument

  def __init__(self, *args, **kwargs):
    self._mylist = kwargs.pop('a_list', ['some', 'default', 'list'])
    super(MyClass, self).__init__(*args, **kwargs)

HOWTO: delete/rename files with special characters (! ~ %) on the linux commandline

2011-05-16T17:13:00.000-07:00

Sometimes you (or your code) screws up and you end up with a filename that is hard to address on the filename because it has special characters in its name. In some cases you can get away with prepending a ./ or quoting, or escaping with backslash. When those don't work, this can get you out of trouble.

First, grab the file or directory's inode:

ls -lhi

Then use the find command with the inode of the troublemaker:

find . -type f -inum 25167125 -exec mv {} binary_safe \;

HOWTO tunnel through multiple boxes with SSH

2011-05-05T22:36:00.000-07:00

Making ssh tunnels, even through multiple machines, is easy. These two commands will effectively link localhost:9999 with box3:80, via box2.

user@box1:~$ ssh -L 9999:localhost:9999 box2
user@box2:~$ ssh -L 9999:localhost:80 box3

Loving the Cyber Bomb? Why don't we love the Cyber Security Bounties instead?

2011-05-04T23:50:00.000-07:00

A paper Loving the Cyber Bomb? The Dangers of Threat Inflation in Cybersecurity Policy by Brito and Watkins of the Mercatus Institute at Virginia's George Mason University makes the argument that the cyber threat is being overblown by government agencies and defence contractors in the chase for dollars. It draws some analogies with the 'evidence' of WMDs that led to the invasion of Iraq.

I don't really buy the analogy that the cyber threat is as vaporous as Iraq's WMDs. Brito and Watkins seem to be implying that the argument that anything less than the ability of an attacker to "derail trains, release chlorine gas, or bring down the power grid" doesn't matter that much. We have plenty of evidence that companies of all sizes are regularly compromised, and I'm sure they'd tell you the loss of business, and/or costs incurred, mattered. I think the Estonian government and businesses would also argue with Brito and Watkins' downplaying of the 2007 DDOS. A cited lack of previous power outages provably due to electronic attack to-date doesn't mean it isn't possible, and it might be more likely to happen at the consumer level as the industry moves towards smart metering.

However, Brito and Watkins do highlight the dangers of poorly targeted government spending being poured into defence contractor's pockets for little gain in security. It was interesting to read about the courtship of the newly-established US Cyber Command ('Cyber Pork' p. 26) by various US towns and states, in a bid to attract its billions of dollars of government investment. Maryland eventually won that battle.

How could these dollars be spent to get the best security for your dollar? I would suggest bounties for specific, measurable, and product-agnostic security improvements. Something like the items from this list produced by the Australian government. Government could offer to cover the costs of implementation (up to a fixed amount) of the top 5 security controls. The offer could be restricted to select government agencies, as well as companies running important infrastructure such as power and water. If you spent $500k on an application whitelisting rollout for a power company it would seem cheap when the next Conficker rolled around.

HOWTO mount a remote luks encrypted volume on demand

2011-05-02T22:31:00.000-07:00

I wanted to create a cron'd backup to a luks volume on a remote machine. My preference was to not have the volume mounted automatically so if my friend rebooted the box it wouldn't block the boot process waiting for the password. It would also be nice if it just mounted itself when necessary and was locked for the rest of the time.

First authorise my user to mount and unlock the volume using specific sudo commands (in /etc/sudoers):

Cmnd_Alias CRYPTOPEN=/sbin/cryptsetup luksOpen /dev/disk/by-uuid/41885992-3f80-4aaa-bc60-9c5854017ca9 crypt-backup --key-file /tmp/keyfile
Cmnd_Alias MOUNT=/bin/mount /dev/mapper/crypt-backup /mnt/backup
Cmnd_Alias UMOUNT=/bin/umount /mnt/backup
Cmnd_Alias CRYPTCLOSE=/sbin/cryptsetup luksClose crypt-backup

myuser ALL=(root) NOPASSWD: CRYPTOPEN,MOUNT,UMOUNT,CRYPTCLOSE

Then, a script on my side:

#!/bin/sh

scp /data/backup/scripts/backup/hdd_keyfile.luks home:/tmp/keyfile && \
ssh home "chmod 600 /tmp/keyfile && sudo /sbin/cryptsetup luksOpen /dev/disk/by-uuid/41885992-3f80-4aaa-bc60-9c5854017ca9 crypt-backup --key-file /tmp/keyfile"
if [ $? -ne 0 ]; then
    echo "cryptsetup failed."
    ssh home "shred -u /tmp/keyfile"
    exit 1
fi

ssh home "sudo /bin/mount /dev/mapper/crypt-backup /mnt/backup"
if [ $? -ne 0 ]; then
    echo "mount failed."
    exit 1
fi

rsync -rtv --compress-level=4 /data/ home:/mnt/backup/data/
rsync -rtv --compress-level=4 /mp3/ home:/mnt/backup/mp3/

ssh home "sudo /bin/umount /mnt/backup && sudo /sbin/cryptsetup luksClose crypt-backup"
if [ $? -ne 0 ]; then
    echo "umount failed."
    exit 1
fi

Chuck it in a cron. Done.

Dropbox authentication issues

2011-04-25T09:53:00.000-07:00

There has been some discussion of whether the dropbox authentication system is a vulnerability, or just as insecure as everything else. At the very least: if changing your password doesn't change the underlying key stored on your machine, that an attacker could have copied, it makes denying access to an attacker post-compromise fairly difficult.

#ifdef DEBUG print statements in C

2011-04-13T19:28:00.000-07:00

The #ifdef DEBUG design pattern for debug output is extremely common in C code. Here is a nice DEBUG macro I have used in the past:

#ifdef DEBUG
#define DEBUG_PRINT(fmt, args...)  \
       fprintf(stderr, "%s:%d:%s(): " fmt, __FILE__, __LINE__, __FUNCTION__, ##args)
#else
#define DEBUG_PRINT(fmt, args...)    /* Don't do anything in non-debug builds */
#endif

As suggested in this post, letting the compiler see your debug code is important to ensure it isn't broken by people changing code without making debug builds. This macro achieves that goal by letting the preprocessor insert a statement that will be pulled out by the optimiser for non-debug builds. This is the best solution I have come across.

#ifdef DEBUG
#define DEBUG_TEST 1
#else
#define DEBUG_TEST 0
#endif

#define DEBUG_PRINT(fmt, args...) \
        do { if (DEBUG_TEST) fprintf(stderr, "%s:%d:%s(): " fmt, __FILE__, \
                                __LINE__, __FUNCTION__, ##args); } while (0)

Mingw example makefile for console app and DLL

2011-04-13T19:26:00.000-07:00

Here is a basic Makefile to use with mingw to compile some common types of windows binaries - a windows console app, a non-console 'windows' subsystem app, and a DLL.

# Example mingw makefile

TARGETNAME=myapp

CC = i586-mingw32msvc-gcc

# A basic windows app (won't bring up a console window)
CFLAGS = -Wall -O -Wl,--subsystem,windows

# for debugging make this a console app (so we can write to stdout/stderr) easily;
# turn off optimisations and include symbols to help with running in a debugger
CFLAGS_DEBUG = -Wall -O0 -g -Wl,--subsystem,console -DDEBUG

# Create a dll instead of an exe
CFLAGS_DLL = -Wall -O0 -g -shared -Wl,--subsystem,windows -DDLL

# Any libraries you need - in this case wininet for communicating to the internet
LIBS = -lwininet

# Strip the binary for our prod build
STRIP = i586-mingw32msvc-strip

# UPX pack to minimise size for our prod build
UPX=upx -9

# Leave symbols, turn off optimisation, and send output to console
debug: $(TARGETNAME).c
        $(CC) -o $(TARGETNAME)_dbg $(CFLAGS_DEBUG) $^ $(LIBS)

# No debug symbols, debug statements removed, subsystem windows means we won't pop a window
$(TARGETNAME): $(TARGETNAME).c
        $(CC) -o $@ $(CFLAGS) $^ $(LIBS)

$(TARGETNAME).dll: $(TARGETNAME).c
        $(CC) -o $@ $(CFLAGS_DLL) $^ $(LIBS)

release: $(TARGETNAME)
        $(STRIP) $<
        $(UPX) $<

Using a word macro to run an executable on Windows 7, Office 2007

2011-04-13T19:25:00.000-07:00

To write a macro in Office 2007, you'll want the developer tab in the ribbon. Turn it on by clicking the Office icon in the top left corner of Word and then 'Word Options'. Find the 'Show Developer tab in the Ribbon' and check it.

Use this macro code to run blah.exe when the document is opened (using AutoOpen):

Private Declare Function WinExec Lib "Kernel32.dll" (ByVal lpCmdLine As String, ByVal nShowCmd As Long) As Long

Sub AutoOpen()

    Call WinExec("C:\blah.exe", SW_NORMAL)

End Sub

Streaming internet radio with MythTV

2011-04-03T23:11:00.000-07:00

Streaming internet radio with mythtv is, sadly, still confined to the realm of hacks. One simple solution is to just drop ".pls" files into your videos directory. Assuming you use mplayer to play your videos, that should just work. Unfortunately I like to use the internal player so I can use the time stretch feature on videos as well as recordings. The internal player doesn't play ".pls" files, so that option is out.

The other approach is to hack in a new menu item and get it to run mplayer. There are a few descriptions of how to do this, so here is my 2c.

Edit /usr/share/mythtv/themes/defaultmenu/library.xml to add:

<button>
    <type>MUSIC_RADIO</type>
    <text>Listen to Internet Radio</text>
    <description>Listen to Internet Radio</description>
    <action>MENU inetradio.xml</action>
    <depends>mythmusic</depends>
</button>

And create inetradio.xml in the same directory. The Australian Broadcasting Corporation (ABC) links are a little hard to find - you can get them here.

<mythmenu name="INETRADIO">

<button>
<type>MUSIC_PLAY</type>
<text>NPR</text>
<action>EXEC /usr/bin/mythradio http://npr.ic.llnwd.net/stream/npr_live24</action>
</button>

<button>
<type>MUSIC_PLAY</type>
<text>TripleJ</text>
<action>EXEC /usr/bin/mythradio http://www.abc.net.au/res/streaming/audio/aac/triplej.pls</action>
</button>

<button>
<type>MUSIC_PLAY</type>
<text>RawFM</text>
<action>EXEC /usr/bin/mythradio http://www.rawfm.com.au/stream/playlists/160kbitMP3.pls</action>
</button>

<button>
<type>MUSIC_PLAY</type>
<text>666 ABC Canberra</text>
<action>EXEC /usr/bin/mythradio http://www.abc.net.au/res/streaming/audio/aac/local_canberra.pls</action>
</button>

<button>
<type>MUSIC_PLAY</type>
<text>ABC Classic FM</text>
<action>EXEC /usr/bin/mythradio http://www.abc.net.au/res/streaming/audio/aac/classic_fm.pls</action>
</button>

<button>
<type>MUSIC_PLAY</type>
<text>Stop Playing</text>
<action>EXEC pkill -f mplayer</action>
</button>

</mythmenu>

And create /usr/bin/mythradio (this wouldn't be necessary if you could give multiple commands to "EXEC" - unfortunately it seems to break the syntax and mythtv says it is an invalid XML file). The include file is a one line mplayer config that tells mplayer to allow the screensaver to run, so you can avoid burn-in while listening to the radio.

#!/bin/bash
pkill -f mplayer
/usr/bin/mplayer -include /etc/mythtv/mplayer_noscr.conf $1 &

and here is /etc/mythtv/mplayer_noscr.conf:

stop-xscreensaver=no

HOWTO Troubleshoot windows sound problems (for parents using google video chat)

2011-03-12T23:08:00.000-08:00

The good people at google, have a nice parent-friendly intro to video chat. Unfortunately, getting to the point of having a gmail account with the plugin installed, is, in my experience, the easy part.

The bigger challenge is troubleshooting audio problems, and the google troubleshooting page doesn't offer much help, apart from a weird endorsement of Logitech products. For the people I chat with, the problem is usually a muted microphone, and/or muted sound. Instructions for modifying sound setup on Windows 7 are below.

Look at the bottom right corner of the screen for the speaker icon

Left click the speaker, and make sure the volume is turned up

Check if that solves the problem. If not, right click the speaker and choose playback devices

Click Properties

and check the slider is turned up on the Levels tab.

Click OK, and get out to the desktop.

Right click on the speaker again to check your microphone. Click recording devices.

The green bar should move up and down as it registers sound. Try talking.

If that all looks OK, google provides an interface for you to check your settings from within gmail.

Then click on the chat tab

and on the + sign next to "Verify your settings".

If you answer 'Yes' to all three questions in that box, and you still don't have working audio or video, you can tell the person on the other side that it is a problem on their end, and point them to these instructions :)

HOWTO Create multiple chrome and firefox profiles on linux

2011-03-12T12:15:00.000-08:00

I like using multiple browser profiles for security reasons. I keep my valuable cookies, such as those for email etc. in one profile, and do regular web-browsing in another. This means if I hit a XSS through regular browsing, the scope for cookie stealing is limited. When using firefox with the noscript plugin it also makes sense to have different whitelists for these two usage scenarios.

Creating firefox profiles is simple, just use the profile manager, which you can bring up with:

firefox --no-remote -P

Once you have created the profile, I copy the shortcut icon and change the target to point at the new profile like this:

firefox %u --no-remote -P browse

Chrome is almost as easy, but the doco can be a little harder to find. You just need to specify a new user data directory like this:

/opt/google/chrome/google-chrome --enable-udd-profiles --user-data-dir=.config/google-chrome/Browse

and change your shortcut icon to run the same command.

On Ubuntu natty and oneiric this got harder with the new launcher. There are a lot of ways to get it done, but this seemed the simplest to me:

sudo apt-get install alacarte gnome-panel

Start "Main Menu" by searching for it in the dash.
Add an item for your application and close the editor.
Logout to refresh the dash (there must be a way to do this without a logout, but not sure what it is).
Start the application by searching for it in the dash.
Right-click on the icon and "Keep in launcher"

I also created a couple of (ugly) custom icons to keep my 'Browse' and 'Mail' profiles clearly marked.

HOWTO add a directory to your python path

2011-01-24T23:51:00.000-08:00

You can add to your python path like this:

import sys
sys.path.append("/home/me/mypy")

The (python) way of the future: pip and virtualenv

2011-01-24T23:50:00.000-08:00

I recently discovered the magic of pip and virtualenv. Pip is shaping up as a serious contender for replacement of distutils and easy_install. I haven't used it much yet, and will have to reserve judgement until I package a decent-sized project with pip.

Using pip is easy, although there is a bootstrapping problem that, ironically, I used easy_install to solve. If your distro packages a recent version of pip, you can use it to skip this bit.

sudo apt-get install python-setuptools
s easy_install pip
s pip install virtualenv

This gives you a virtualenv install. Virtualenv is brilliant, and has significantly influenced the way I write python. Virtualenv allows you to build a complete python environment in a directory that you 'activate' by running an in-built shell script that sets your python path environment variables. The are many advantages to this approach, but here are a few I have encountered:

Simplified testing - test your code against multiple versions of libraries and utilities
Keep separate dev and production environments on your dev machine, allowing you to test in a production environment without pushing the code to your actual production server.
The isolation provided by Virtualenv ensures you aren't depending on libraries on your dev box that aren't installed in production, and upgrading libraries won't break other applications

You can even create virtual environments with different versions of the python interpreter using:

virtualenv -p python2.6 2.6ENV

So, once you have a virtualenv, activate it and install yolk to give you a picture of what is inside the environment.

source .testenv/bin/activate
~> pip install yolk
~> yolk -l
Python          - 2.6.4        - active development (/usr/lib/python2.6/lib-dynload)
distribute      - 0.6.14       - active
pip             - 0.8.1        - active
wsgiref         - 0.1.2        - active development (/usr/lib/python2.6)
yolk            - 0.4.1        - active

Then go ahead and install the other packages you need:

pip install django

You can also build a requirements file to install exactly the same versions somewhere else:

pip freeze -E .testenv > requirements.txt

The holy trinity is apparently pip, virtualenv and fabric. Fabric is something I'll have to play with soon. There is a good tutorial on pip and virtualenv here, and using them with django here.

The only limitation/problem I have run into is that you can't move the virtual environment once it is created. There is a "--relocatable" directive to address this, but the webpage says that it is somewhat experimental at the moment.

Syntax highlighting code in Blogger

2011-01-24T21:37:00.001-08:00

Posting code in Blogger by default results in something that is inexcusably ugly. How can you make code posted in blogger look pretty? For me, syntax highlighting and scrollable text boxes are a must.

The solution is: google-code-prettify, which does the highlighting for code.google.com (you could also consider SyntaxHighlighter). Now the next problem is, how do you distribute the javascript and CSS for prettify? Unfortunately Google doesn't make it available via their CDN like they do for other libraries like jQuery. This would obviously be the best option for users since:

it would negate the need for additional hosting; and
minimise load time since browsers could cache the code across many sites, and if needed download it from the closest CDN node.

So the remaining options are to slap it into your blogger template, host it yourself somewhere (like Google pages), use the subversion browse code link (ick), or figure out what code.google.com does, since it uses the same code.

I opted for the last option, which works a treat. Put this in your blogger template, right after the <head> tag:

<link type="text/css" rel="stylesheet" href="http://www.gstatic.com/codesite/ph/3799605220899551948/css/ph_core.css">
 
<link type="text/css" rel="stylesheet" href="http://www.gstatic.com/codesite/ph/3799605220899551948/css/ph_detail.css" >
 
<link type="text/css" rel="stylesheet" href="http://www.gstatic.com/codesite/ph/3799605220899551948/css/d_sb_20080522.css" >

Then add an onload event to the body tag to call the PrettyPrint function:

<body onload="prettyPrint()" >

and, since the prettyprint CSS doesn't give you a scrolling box by default (!?), add this to the CSS in your template to avoid your code being chopped off:

/* Override prettyprint CSS, which doesn't allow scrolling box */
pre.prettyprint {
  overflow: auto;
}

Once you have done all that, you just need to wrap the code in your posts in:

<pre class="prettyprint">
</pre>

In summary, much harder than it needs to be. Blogger team, please build this functionality in!

List of DNS Providers

2011-01-19T18:05:00.000-08:00

Just came across a list of DNS providers I made a while ago, and thought I'd post it here. They all seem to still be in business. The top three I have heard good reports about:

GPG HOWTO

2011-01-19T17:42:00.000-08:00

Here is a good, quick intro to using GPG, including generating and publishing keys, encrypting and signing.

The Google Apps administrator control panel

2011-01-18T22:24:00.000-08:00

The Google Apps administrator control panel for "your_domain.com" can be found at the fairly un-intuitive address:

https://www.google.com/a/your_domain.com

Linux boot process digram

2011-01-12T03:33:00.000-08:00

Finally got around to converting my RHCE notes on the linux boot process into a diagram.

Creating a HTML DIV that adjusts its size to automatically fit the content

2011-01-05T21:21:00.000-08:00

To create a div that automatically adjusts its size to fit the content, use this in your CSS:

margin: 0 auto;

Using 'margin' for this purpose doesn't seem particularly intuitive to me, and isn't mentioned in CSS doco I read, but it works.

Rendering a django queryset as a html table (like form.as_table) using a custom template filter

2011-01-05T14:49:00.001-08:00

Django has some nice helpers for forms (forms.as_p and forms.as_table) that you can use in your template to minimise boilerplate HTML. Unfortunately, despite some discussion/confusion, and at least one project, there doesn't appear to be a similar elegant solution for querysets.

My approach to the problem was to implement a custom template filter you can use in a template like this (where object_list is your queryset):

{{ object_list|result_as_table }}

i.e. you hand it a queryset, and it gives you back a table.

Here is the code for the filter. I'm using this for a number of subclasses of a class called resultdata, so that they can share an identical template and just provide different querysets in urls.py.

def result_as_table(queryset, fieldnames=None):
    """Take a resultdata queryset and return it as a HTML table.
    Columns will be returned in the order they are declared in the model.
    
    Optionally, provide a list of fieldnames, which will be used to limit the output."""


    dictlist = queryset.values()
    output = "<table>\n"
    output_keys = []

    if fieldnames:
        names = fieldnames
    else:
        names = dictlist.field_names

    for name in names:

        if not name.endswith("_id"):
            output_keys.append(name)
            output = "".join( (output, "<th>%s</th>\n" % escape(name.replace("_"," ").capitalize())) )

    for rddict in dictlist:

        output = "".join( (output, "<tr>\n") )

        for key in output_keys:
            val = rddict[key]
            if not key.endswith("_id"):

                display_func = get_display_method(rddict, queryset, key)

                if display_func:
                    output = "".join( (output, "<td>%s</td>\n" % escape(display_func())) )
                else:
                    output = "".join( (output, "<td>%s</td>\n" % escape(val)) )

        output = "".join( (output, "</tr>\n") )

    return mark_safe("".join( (output, "</table>\n") ))

result_as_table.is_safe = True

This filter introspects the column headers, strips out any id fields (since users don't care about them), converts underscores to spaces in column headers, and finds the corresponding display methods for any 'choices' fields so you can still get the nice human-readable output. It outputs HTML, so it uses django's own 'django.utils.html.escape' method and marks the output as safe so django doesn't escape all the HTML.

Finding the display methods was a little harder than I expected, since the python inspect code was dying when given a django model object. This code is unfortunately tied to the resultdata object due to the naming of the keys in the values() dictionary, making it less generic than I would like.

def get_display_method(rddict, queryset, key):
    """Re-implementation of inspect.getmembers(rddict,inspect.ismethod) and 
    a test to see if this is a get_field_status method.  Had to reimplement inspect
    because it was bombing on the object - expecting some property that is not present."""

    rdobj = queryset[0].__class__.objects.filter(resultdata_id = rddict["resultdata_id"]).get()
    targetname = "get_%s_display" % key
    display_func = False

    for name in dir(rdobj):
        if name == targetname:
            try:
                display_func = getattr(rdobj, name)
                break
            except Exception,e:
                pass

    return display_func

There was a major gotcha with using the field_names property of the queryset.values() dictionary. If you passed a queryset with annotations like this is your urls.py:

list_rd = {
    'extra_context': {'fieldnames': ['status','first','total']},
    'queryset': ResultData.objects.annotate(total = Count('link__clickback'), 
                                            first = Min('link__clickback__timestamp')),
}

urlpatterns += patterns('project.app.views',
    url(r'^result/detail/$', 'detailed_results', kwargs=list_rd, name='detailed_results'),
)

The annotations would not get added into the field_names property. This was solved by providing the optional fieldnames parameter.

HOWTO add headers to the Django test client

2010-12-22T16:29:00.000-08:00

To mess with the HTTP headers used by the Django web test client, make your request with your custom headers in a dict. From inside a django test method it looks like this:

self.client.get('/some/path/', **{'HTTP_USER_AGENT':'silly-human', 'REMOTE_ADDR':'127.0.0.1'})

Import multiple vcf files into gmail

2010-12-17T17:43:00.003-08:00

You can import contacts into gmail - it currently accepts csv files or vcf files. Unfortunately you can only upload one file at a time. If you are trying to import a full phone backup of hundreds of vcf files, that isn't useful. It turns out vcf's have their own internal structure and you can just combine them all into one giant file (using cat on linux):

cat *.vcf > ../allinone.vcf

HOWTO: Convert a darcs repository to mercurial

2010-12-12T20:53:00.000-08:00

It seems the "easiest" way to convert a darcs repo to a mercurial repo is via git =( This uses darcs-fast-export, git-fast-import and hg convert. Other solutions such as tailor and "darcs2hg.py" exist but seem hard(er) to setup or have errors.

Get darcs fast-export from bzr-fast-import:
```
git clone git://vmiklos.hu/bzr-fastimport
```
The only thing you care about in the entire bzr-fast-import repo is "darcs-fast-export" (a python script) in bzr-fastimport/exporters/darcs.

Checkout the darcs project

mkdir project; cd project; darcs get codebox:/code/project project.darcs; cd ..

Convert darcs to git (in Ubuntu karmic git-fast-import isn't in PATH)

mkdir project/project.git; cd project/project.git; git init
../../darcs-fast-export.py ../project.darcs | /usr/lib/git-core/git-fast-import; cd ..

Convert git to hg

hg convert project.git project.hg; cd project.hg; hg update

Generate SSL certificates for openvpn with easy-rsa

2010-12-08T21:21:00.000-08:00

Easy-rsa is distributed with openvpn (on Ubuntu anyway), and makes generating SSL certs a lot easier.

Here is typical usage:

cd /usr/share/doc/openvpn/examples/easy-rsa/2.0
[edit vars with your site-specific info]
source ./vars
./clean-all
./build-dh     -> takes a long time, consider backgrounding
./pkitool --initca
./pkitool --server myserver
./pkitool client1

Keys and certs are written to the "keys" directory.

HOWTO rotate large numbers of images in Ubuntu

2010-12-07T01:34:00.000-08:00

I needed a way for my wife to rotate a whole bunch of JPEG images. It had to be simple to use (i.e. GUI not command line).

I stumbled across a blog that suggested the nautilus-image-converter plugin, which worked perfectly. It has a simple right-click interface that allows you to rotate images in place or use a renaming scheme. It can also do resizing.

Brilliant.

Apache and client side SSL certificate verification

2010-12-02T21:27:00.000-08:00

To require client SSL certificate verification, add this to your apache config:

SSLVerifyClient require
SSLVerifyDepth 1
SSLCipherSuite HIGH:MEDIUM
SSLCACertificateFile /etc/ssl/ca_that_signed_client_certs.pem

And to log what is going on with the SSL client cert verification, use something like this:

ErrorLog /var/log/apache2/error.log
LogLevel info

CustomLog /var/log/apache2/access.log combined

CustomLog /var/log/apache2/ssl.log "%t %h %{SSL_PROTOCOL}x verify:%{SSL_CLIENT_VERIFY}x %{SSL_CLIENT_S_DN}x \"%r\" %b"

Commercial SSL certificate untrusted - what did I pay for?

2010-11-30T21:13:00.000-08:00

I recently bought a commercial SSL certificate, and was slightly mystified as to why the browser was calling it untrusted. How could they possibly be selling certs that Firefox doesn't trust? After some head scratching I realised the answer was that I needed to install the intermediate certificates (provided by the CA) on the server side, to complete the chain of trust.

During the SSL certificate exchange the web server (in this case Apache) can provide the client with additional certificates to enable it to establish a chain of trust. Use the SSLCertificateChainFile directive in your site config, something like:

    SSLCertificateChainFile /etc/apache2/ssl/ExternalCARoot1.crt
    SSLCertificateChainFile /etc/apache2/ssl/CACompanySecureServerCA.crt

According to the apache help, you can cat these two together and just specify one file. Say the browser trusts RootCA1, it can check that RootCA1 signed ExternalCARoot1.crt, which signed CACompanySecureServerCA.crt, which signed my certificate. Without those intermediate certificates, the browser cannot establish trust.

Making blogger look prettyish: removing the attribution footer and increasing the post width

2010-11-27T03:29:00.000-08:00

The new templates provided by blogger go a long way to making it look prettier. There are IMHO a few fundamental problems. The first is the attribution footer gadget - that is nice for the original designer, but I don't need to advertise for them on my blog. A lot of people seem to be trying to change this behaviour.

To remove the attribution footer, search in your css for 'attribution' and use html comments to comment out those sections. Check with preview to see if they are gone. When you click 'save template' blogger will ask if you want to delete the attribution gadget. You can delete it, and it will stay gone.

Next, making the post wider. Blogger is stuck being optimised for small screen sizes no-one uses any more. To increase the post width, change the 'value' attribute of this tag (search for 'content.width'):


<b:variable default='930px' name='content.width' type='length' value='1000px'/>

And to change the width of your gadget panel, change value of:


<b:variable default='360px' name='main.column.right.width' type='length' value='370px'/>

Set file modification time of a JPEG to the EXIF time

2010-11-24T21:20:00.001-08:00

After editing a photo, it is nice to be able to set the file modification time back to its original so filesystem date sorting is still sensible. This can be achieved by reading the "Exif.Photo.DateTimeOriginal" or "Exif.Image.DateTime" out of the JPEG header. exiv2 will do this for you:

exiv2 -T rename *.JPG

To do every file recursively under a directory, cd into the directory and use this:

find . -type f -iname "*.jpg" -print0 | xargs -0 exiv2 -T rename

HOWTO do a DNS zone transfer

2010-10-04T23:06:00.000-07:00

Use dig to get a list of nameservers and then perform a DNS zone transfer:

dig -t NS transfer.me.com
dig -t AXFR transfer.me.com @ns1.transfer.me.com

World meeting time planner

2010-09-27T06:26:00.000-07:00

Planning a meeting or phonecall across multiple timezones can be tough.

I've used a few online tools, but my current favourite is a gnome app called slashtime by Andrew Cowie. I saw Andrew speak at Linux Conf Au in 2009 on GUI design, using slashtime as an example, and it was a great talk.

Highly recommend. Would be great to see a debian package so it is apt-gettable.

HOWTO list windows shares from a linux box using smbclient

2010-09-19T20:11:00.003-07:00

To get a list of all SMB shares exposed by a windows box, use smbclient:

smbclient -L my.windows.host --user=myfirst.mylast

jQuery: host locally or use googleapis.com CDN?

2010-09-17T02:44:00.000-07:00

There are two main ways to host your blob of jQuery minimised javascript: on your own webserver or, on Google's. Which is better? A large amount of time has been spent debating that very topic.

What are the pros and cons of using Google?

Pros

Fast, geographically distributed, reliable CDN
Free, saves using your bandwidth
If many people use the Google version of jQuery (and they do), it is highly likely the user will have it in their browser cache, given the long expiry times Google sets when they serve the file (although there are some caveats ). This means the user probably won't have to request the file at all. Even if users regularly clear their browser cache it is likely to be cached by a proxy.

Cons

The CDN might be down, or slow, which will impact your site.
If there is no Internet connection it won't work (definitely not the best choice for internal webservers)
If you already have other javascript bundled into a minimised file (common practice), it is an extra web request that needs to be made, when you could just include it in the bundle.
You are giving Google information about your customers (i.e. forcing them to make a request) to Google. Given the large amount of caching, this will not be comprehensive, and there is a reasonable chance you are running Google analytics anyway.

On balance, I think using Google's version is the better option.

Using the django-admin time and date widgets in your own django app: too hard, use jQuery

2010-09-17T02:22:00.001-07:00

The django doco encourages you to use the django admin widgets (such as the time and date picker) in your own apps. Unfortunately actually doing this turns out to be more work than using external widgets like the excellent jQuery-ui.

jQuery comes with good tutorials and doco. jQuery-ui, which builds on jQuery, has a great array of helpful widgets. If, like me, you just want a date picker widget, it is super easy.

You'll want to add something like the following to your template (in your base template if it will be used on every page):

<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js" type="text/javascript"></script>
<script src="http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.5/jquery-ui.min.js" type="text/javascript"></script>
<script type="text/javascript">
  jQuery(function()
    {
      jQuery('.vDateField').datepicker({ 
          dateFormat: 'yy-mm-dd',
          constrainInput: 'true', 
          maxDate: ' 1m',
          minDate: '0d' })
    });
</script>

To make it look pretty jQuery has you covered, with ThemeRoller. Pick a style you like, customise it if you need to, and download your new CSS. Drop that in your template too.

Taking a disk image and creating a hash of the data with one read of the source disk

2010-09-14T00:18:00.000-07:00

I have previously blogged about how to take a disk image over the network. The more common case is you want to make a forensic copy of a locally-connected disk. Usually this is a disk you connect using a write blocker, such as one from wiebetech, to prevent any changes being made to the source disk.

This command takes a forensic image and a hash of the original disk at the same time, requiring only one read of the source disk:

mkfifo /tmp/disk.dat; sha1sum /tmp/disk.dat & dd bs=256k if=/dev/sdc | tee /tmp/disk.dat > /mnt/destination/disk.dd

Chrome (chromium) browser makes random 10 character HEAD requests on startup

2010-09-13T18:31:00.000-07:00

I recently saw this in a proxy log:

http://yyvssjupua/ 192.168.20.1/- - HEAD - from squid.
http://mskwuzkkpu/ 192.168.20.1/- - HEAD - from squid.
http://dfoigxiyyl/ 192.168.20.1/- - HEAD - from squid.

What the? After talking to the user, who told me he was running chromium, I found out this was legit chromium behaviour. Apparently some ISPs will send you to a page with their advertising if you visit a url that has a DNS lookup failure. Bastards. To combat this, on startup chrome makes three requests to random domains that are guaranteed to generate lookup failures. If they get a HTML page back, chrome knows to disable the 'did you mean' functionality that asks if you meant to visit the host or perform a search query for the host so it doesn't keep pointing you to the ISP's ad page. Smart!

Cisco 'show everything' and password cracking

2010-09-11T19:04:00.001-07:00

To do a 'show everything' on a cisco device, use 'show tech-support'. This includes show run, process listings, interface info, and basically every bit of information you can get through running other commands. Note that user type 7 passwords (see below) are automatically sanitised from the output.

Cisco still uses a terrible password encryption scheme for user passwords that can be trivially cracked. The following user password uses the weak encryption (you can tell by the number 7 preceeding the hash):

username jdoe password 7 07362E590E1B1C041B1E124C0A2F2E206832752E1A01134D

While user passwords are encrypted using this weak scheme, enable passwords are MD5 hashes that look like this (note the 5):

enable secret 5 $1$iUjJ$cDZ03KKGh7mHfX2RSbDqP.

Cisco is stuck using the reversible encryption scheme for the near future due to the need to support certain authentication protocols (notably CHAP).

Enable (MD5) passwords can be cracked using standard tools such as John the Ripper or rainbow tables.

Type 7 passwords can be cracked with the following simple perl script.

#!/usr/bin/perl -w
# $Id: ios7decrypt.pl,v 1.1 1998/01/11 21:31:12 mesrik Exp $
#
# Credits for orginal code and description hobbit@avian.org,
# SPHiXe, .mudge et al. and for John Bashinski 
# for Cisco IOS password encryption facts.
#
# Use for any malice or illegal purposes strictly prohibited!
#

@xlat = ( 0x64, 0x73, 0x66, 0x64, 0x3b, 0x6b, 0x66, 0x6f, 0x41,
0x2c, 0x2e, 0x69, 0x79, 0x65, 0x77, 0x72, 0x6b, 0x6c,
0x64, 0x4a, 0x4b, 0x44, 0x48, 0x53, 0x55, 0x42 );
while (<>) {
  if (/(password|md5)\s+7\s+([\da-f]+)/io) {
    if (!(length($2) & 1)) {
      $ep = $2; $dp = "";
      ($s, $e) = ($2 =~ /^(..)(.+)/o);
      for ($i = 0; $i < length($e); $i+=2) {
        $dp .= sprintf "%c",hex(substr($e,$i,2))^$xlat[$s++];
      }
      s/7\s+$ep/$dp/;
    }
  }
  print;
}

Booting and/or mounting a raw disk image under windows

2010-09-11T18:31:00.000-07:00

CERT has developed a cool tool called LiveView that allows you to boot a raw disk image (such as one produced by 'dd') using VMWare. LiveView preserves disk integrity by writing all disk changes to a separate file. The tool works under windows and linux, and boots a range of Windows versions.

Alternatively, if you just want to mount a disk image in windows (something that is trivial in linux using the loopback device), there is a tool called imdiskinst that can help you out.

HOWTO dump out all email attachments from a Microsoft PST archive

2010-09-09T23:21:00.001-07:00

On ubuntu install 'readpst' and 'uudeview', then:

readpst -o mbox -j4 mpst.pst

Which will use 4 processes to give you a bunch of mbox files in the 'mbox' directory. Then, extract all the attachments:

cd mbox
uudeview -p ../pst_attachments *

Where there is awk, there is sed

2010-08-25T05:20:00.001-07:00

I couldn't do a post on awk without also quickly covering sed. Bruce Barnett has written a great tutorial for sed that is worth reading.

Sed is your friend for applying regex's to files (well, streams really, since it is the 'stream editor'). The regex syntax is the same as for vim, and since that is my primary editor I only tend to use sed where the files are large and will take ages to load into vim.

Some quick examples to illustrate what I'm talking about:

sed s'/user root/user nothingtoseehere/g' < /var/log/auth.log

sed s'!session closed for user \([^ ]*\)!\1 closed a session!g' < \
/var/log/auth.log

sed s'!session closed for user \([^ ]*\)!&, allegedly!g' < \
/var/log/auth.log

Escaped parenthesis

\(\)

capture a value, and

refers to the whole match.You can also use sed like grep. By default it prints every line, but that can be disabled with "-n" and you can cause matching lines to be printed by appending a "p" option:

sed -n 's/pattern/&/p' < file.txt

OR, if we aren't making a substitution

sed -n '/pattern/p' < file.txt

OR, you can just use grep -e, which is less confusing:

grep -e "pattern" file.txt

AWK - selecting columns for output

2010-08-25T03:48:00.001-07:00

AWK is a super-handy old-skool UNIX tool. There are plenty of good tutorials out there for it, but I'm jotting down some basic uses here for my own benefit.

I have used AWK mainly to select and print certain columns from input, like this, which will print the 1st and 7th columns:

awk '{print $1,$7}' < /var/log/syslog

Columns break-up is determined by the value of the FS (input field separator) variable, which is space by default (in POSIX mode this actually means space and tab but not newline). You can change this with:

awk 'BEGIN {FS=";"}{print $1,$7}' < /var/log/syslog

OR

awk -F: '{print $1,$7}' < /var/log/syslog

The output from awk is separated by the OFS (output field separator) variable, also a space by default. To write out CSV you might use:

cat /var/log/syslog | awk -F: 'BEGIN{OFS=","}{print $1,$3}'

There is plenty more you can do with awk, including simple programming tasks such as counting, summing etc.cut is a simple alternative if all you want to do is cut fields from an input stream. It is doesn't take much to hit its limitations however. Consider the output of last, the first two columns of which look like this:

user     pts/0
user     pts/1
reboot   system boot

This awk command will print the first two columns correctly:

last | awk '{print $1,$2}'

Whereas this cut command:

last | cut -d" " -f1-5

Won't produce the first two columns cleanly and we need to specify 5 columns to try and skip the empty fields. The problem is there are variable numbers of spaces between the username and the tty line.

Tips for hardening apache on ubuntu for django deployment

2010-08-23T06:08:00.000-07:00

There is good doco for deploying django on apache with mod_python or wsgi. Here are a couple of extra tips for Ubuntu. First, edit

/etc/apache2/conf.d/security

and enable:

ServerTokens Prod
ServerSignature Off
TraceEnable Off

And in the apache config in your "Location /" directive with the other django stuff:

Options -Indexes -Includes -Multiviews SymLinksIfOwnerMatch

Take a look at Apache's security tips and it is also worth understanding how the Apache configuration directives (Directory, Location, etc.) work.

SSH client config

2010-08-20T20:31:00.000-07:00

For Internet-connected hosts, running SSH on a different port is a really good idea since it cuts down the noise of authentication attempts from bots looking for weak passwords. Running on a different port is not a substitute for a secure configuration (ie. no root login, key-only auth) - it is purely useful in cutting down log noise.

Unfortunately you have to remember which port you chose :) To minimise the hassle you should add entries in your client /etc/ssh/ssh_config:

Host nickname
    Port 43210
    HostName mysshserver
    User myuser

Now you can use "ssh nickname" and ssh will translate that to:

ssh -p 43210 mysshserver

Installing Windows 7 onto a netbook using USB

2010-08-16T06:41:00.000-07:00

I wanted to install Windows 7 onto a netbook to replace an aging desktop as my only windows-on-metal box. This is a breeze with modern linux distros, but is of course far harder than it needs to be for windows.

I had a crack at using unetbootin on Linux with the Windows 7 ISO, despite a suspicious lack of mention of support for windows, and sure enough it didn't boot (unetbootin didn't give me any errors, it just didn't boot).

More googling turned up the Windows 7 USB/DVD Download Tool, which converts a Windows install ISO into a bootable USB installer - exactly what I wanted. After half an hour of downloading and installing dependencies (due to the lack of windows package management and a bizarre need to do the Genuine Windows check), I had the tool installed and it happily created a bootable USB for me.

This one worked perfectly, and dropped Windows 7 starter onto the netbook.