Contacting file owners: using find, awk, and xargs to creating lists of files and operate on them

Say you want to contact a bunch of file owners to do some cleanup or housekeeping. This will get you a list of files sorted by owner.

find . -type f -name "*blah*" "%u %M %g %y %p\n" | sort

But we can do better. Here is a neat little awk command that will drop filenames into txt files named after the user.

find . -type f -name "*blah*" "%u %M %g %y %p\n" | sort | awk 'BEGIN {FS=" "} { print $5 >>"/home/me/temp/"$1".txt" }'

So the end result is that /home/me/temp/mike.txt contains a list of files that mike owns. You can then tell mike to do something with those files (in this example, delete) using xargs like this:

xargs -a mike.txt rm -f

How do you tell mike? well you can get a list of addresses to drop into your mail client:

find . -type f -name "*blah*" "%u@company.com\n" | sort | uniq

then drag and drop the USERNAME.txt files into the mail. With a bit more effort you could automate the sending mail step too.

Adding other arguments when using *args and **kwargs in python

Sometimes you need to use *args and **kwargs to pass your positional and keyword args to another class. But what if your class takes an additional argument you don't want to pass to the super class? Here are a couple of options.

Add a new positional argument

  def __init__(self, a_list, *args, **kwargs):
    super(MyClass, self).__init__(*args, **kwargs)
    self._mylist = a_list

Add a new keyword argument

  def __init__(self, *args, **kwargs):
    self._mylist = kwargs.pop('a_list', ['some', 'default', 'list'])
    super(MyClass, self).__init__(*args, **kwargs)

HOWTO: delete/rename files with special characters (! ~ %) on the linux commandline

Sometimes you (or your code) screws up and you end up with a filename that is hard to address on the filename because it has special characters in its name. In some cases you can get away with prepending a ./ or quoting, or escaping with backslash. When those don't work, this can get you out of trouble.

First, grab the file or directory's inode:

ls -lhi

Then use the find command with the inode of the troublemaker:

find . -type f -inum 25167125 -exec mv {} binary_safe \;

HOWTO tunnel through multiple boxes with SSH

Making ssh tunnels, even through multiple machines, is easy. These two commands will effectively link localhost:9999 with box3:80, via box2.

user@box1:~$ ssh -L 9999:localhost:9999 box2
user@box2:~$ ssh -L 9999:localhost:80 box3

If you just want to ssh through one machine to another, as of OpenSSH 5.4 there is a better way (I haven't tested this yet):

 $ cat ~/.ssh/config
 Host internalhost.myhouse.com
   ProxyCommand ssh -W %h:%p sshgateway.myhouse.com

So when you ssh to internalhost.myhouse.com you will connect through sshgateway.myhouse.com. You can chain these together, so if there was a third host only accessible from internalhost that would look like:

 $ cat ~/.ssh/config
 Host internalhost.myhouse.com
   ProxyCommand ssh -W %h:%p sshgateway.myhouse.com

 Host ilovelayers.myhouse.com
   ProxyCommand ssh -W %h:%p internalhost.myhouse.com

Loving the Cyber Bomb? Why don't we love the Cyber Security Bounties instead?

A paper Loving the Cyber Bomb? The Dangers of Threat Inflation in Cybersecurity Policy by Brito and Watkins of the Mercatus Institute at Virginia's George Mason University makes the argument that the cyber threat is being overblown by government agencies and defence contractors in the chase for dollars. It draws some analogies with the 'evidence' of WMDs that led to the invasion of Iraq.

I don't really buy the analogy that the cyber threat is as vaporous as Iraq's WMDs. Brito and Watkins seem to be implying that the argument that anything less than the ability of an attacker to "derail trains, release chlorine gas, or bring down the power grid" doesn't matter that much. We have plenty of evidence that companies of all sizes are regularly compromised, and I'm sure they'd tell you the loss of business, and/or costs incurred, mattered. I think the Estonian government and businesses would also argue with Brito and Watkins' downplaying of the 2007 DDOS. A cited lack of previous power outages provably due to electronic attack to-date doesn't mean it isn't possible, and it might be more likely to happen at the consumer level as the industry moves towards smart metering.

However, Brito and Watkins do highlight the dangers of poorly targeted government spending being poured into defence contractor's pockets for little gain in security. It was interesting to read about the courtship of the newly-established US Cyber Command ('Cyber Pork' p. 26) by various US towns and states, in a bid to attract its billions of dollars of government investment. Maryland eventually won that battle.

How could these dollars be spent to get the best security for your dollar? I would suggest bounties for specific, measurable, and product-agnostic security improvements. Something like the items from this list produced by the Australian government. Government could offer to cover the costs of implementation (up to a fixed amount) of the top 5 security controls. The offer could be restricted to select government agencies, as well as companies running important infrastructure such as power and water. If you spent $500k on an application whitelisting rollout for a power company it would seem cheap when the next Conficker rolled around.

HOWTO mount a remote luks encrypted volume on demand

I wanted to create a cron'd backup to a luks volume on a remote machine. My preference was to not have the volume mounted automatically so if my friend rebooted the box it wouldn't block the boot process waiting for the password. It would also be nice if it just mounted itself when necessary and was locked for the rest of the time.

First authorise my user to mount and unlock the volume using specific sudo commands (in /etc/sudoers):

Cmnd_Alias CRYPTOPEN=/sbin/cryptsetup luksOpen /dev/disk/by-uuid/41885992-3f80-4aaa-bc60-9c5854017ca9 crypt-backup --key-file /tmp/keyfile
Cmnd_Alias MOUNT=/bin/mount /dev/mapper/crypt-backup /mnt/backup
Cmnd_Alias UMOUNT=/bin/umount /mnt/backup
Cmnd_Alias CRYPTCLOSE=/sbin/cryptsetup luksClose crypt-backup

myuser ALL=(root) NOPASSWD: CRYPTOPEN,MOUNT,UMOUNT,CRYPTCLOSE

Then, a script on my side:

#!/bin/sh

scp /data/backup/scripts/backup/hdd_keyfile.luks home:/tmp/keyfile && \
ssh home "chmod 600 /tmp/keyfile && sudo /sbin/cryptsetup luksOpen /dev/disk/by-uuid/41885992-3f80-4aaa-bc60-9c5854017ca9 crypt-backup --key-file /tmp/keyfile"
if [ $? -ne 0 ]; then
    echo "cryptsetup failed."
    ssh home "shred -u /tmp/keyfile"
    exit 1
fi

ssh home "sudo /bin/mount /dev/mapper/crypt-backup /mnt/backup"
if [ $? -ne 0 ]; then
    echo "mount failed."
    exit 1
fi

rsync -rtv --compress-level=4 /data/ home:/mnt/backup/data/
rsync -rtv --compress-level=4 /mp3/ home:/mnt/backup/mp3/

ssh home "sudo /bin/umount /mnt/backup && sudo /sbin/cryptsetup luksClose crypt-backup"
if [ $? -ne 0 ]; then
    echo "umount failed."
    exit 1
fi

Chuck it in a cron. Done.