Unpack a debian .deb package

When you are building .deb's it's handy to be able to unpack them to check the contents, especially postinst and similar scripts. This command gives you all the package contents:

dpkg-deb -R google-chrome-stable_current_amd64.deb .

The postinst and other package-related scripts will be in the DEBIAN directory:

$ ls DEBIAN/
control  postinst  postrm  prerm

Creating a debian package that can run with System V, Upstart, or Systemd

We now have a gaggle of ways daemons can be run on linux, and ubuntu in particular. I want my .deb to be installable on a wide range of ubuntu and debian systems, some of them quite old, so here's my solution.

The general idea is to provide files for all three systems, and pick the right one to use at post-install time as described here, but with the added complication that we need systemd as well (for Ubuntu after 15.10, which uses systemd by default).

My postinstall file looks like this:

case "$1" in
  configure)
    ${DAEMON} ${DAEMON_ARGS} "--install"

    if [ -x /sbin/initctl ] && /sbin/initctl version | /bin/grep -q upstart; then
      # Early versions of upstart didn't support restarting a service that
      # wasn't already running:
      # https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/430883
      /usr/sbin/service myservice stop 2>/dev/null || true
      /usr/sbin/service myservice start 2>/dev/null
    elif [ -x /bin/systemctl ]; then
      # Systemd
      /bin/systemctl enable myservice
      /bin/systemctl restart myservice
    elif [ -x "/etc/init.d/myservice" ]; then
      update-rc.d myservice defaults >/dev/null
      invoke-rc.d myservice start || exit $?
    fi

  ;;

  abort-upgrade|abort-remove|abort-deconfigure)
  ;;

  *)
    echo "postinst called with unknown argument \`$1'" >&2
    exit 1
  ;;
esac

If you're using debhelper you need to make sure you're using at least version 9.20130504, when systemd support was added. Then, just like you do for Upstart and System V you need to put your systemd unit file in:

debian/mypackage.service

and it will be copied into

lib/systemd/system/package.service

in the package build directory as described here.

Storing and using GPG keys on the Yubikey

I wanted to move to using GPG keys for encryption and signing stored on a Yubikey 4. There's a bunch of HOWTOs out there, I'll put a pile of links at the end.

I started out making a bootable Ubuntu USB drive with the intention of generating the master key on there while offline, putting the subkeys on the Yubikey, and only importing the public key of the master onto the laptops I would use for day-to-day sign/decrypt. This way the master secret key is never on an internet connected machine. This approach is described in more detail here.

I basically gave up on trying to make the yubikey talk to gpg correctly on linux and used a mac (you can read the whole saga after this). So I followed Trammel's excellent instructions with the following modifications:

1. Disconnect from the network.
2. Follow Trammel's instructions. If you have the Yubikey 4 you can use 4096 bit keys. ykpersonalize didn't work ("no yubikey present"), so I had to install the Yubikey NEO Manager, which for some reason requires a reboot.
3. Using the GUI export the key a second time into a file that is just the public key.
4. Copy pub/private exported key and revocation cert onto USB key.
5. Use "srm -sz" to remove the exported key and cert, leave the exported public key.
6. Delete the key (public and secret) from the GPG keychain using the GUI. The only copy of the master secret key is now on the USB.
7. Import the public key using the GUI.

The command:

gpg --card-status

Should now show "sec#" as described here, to indicate the master secret key isn't present. Now your key is ready to use. I seem to be having similar problems as described here:
https://gpgtools.tenderapp.com/discussions/problems/28634-gpg-agent-stops-working-after-osx-upgrade-to-yosemite
I'll update this post when I know more.

The Linux GPG2 and yubkiey saga

Installing gpg2 (required for yubikey "card" support) turned out to be really painful. Ubuntu ships with gpg 1.4, so I ended up downloading a ton of packages off the gpg ftp server, verifying the signature of each one and doing the configure, make, make install dance. It took ages. Update: I didn't think to look for a gpg2 package, turns out there is one, so this was a big waste of time :)

Then I still had to download and install the yubico tools for interacting with the card. I got ykpersonalize installed, but all the tool ever gave me was this error:

Yubikey core error: no yubikey present

This bug pointed me to the Yubikey NEO manager, which has a PPA! Hooray! Except I couldn't get it to work on trusty, my errors are below. However, I just re-tried in a clean trusty docker container and it seemed to succeed, so I'm not going to file a bug:

ubuntu@ubuntu:~$ sudo apt-get install yubikey-neo-manager
Reading package lists... Done
Building dependency tree      
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
 
The following packages have unmet dependencies:
 yubikey-neo-manager : Depends: libu2f-host0 (>= 0.0) but it is not going to be installed
                       Depends: python-pyside.qtwebkit but it is not installable
                       Recommends: pcscd but it is not installable
E: Unable to correct problems, you have held broken packages.
ubuntu@ubuntu:~$ sudo apt-get install python-pyside.qtwebkit
Reading package lists... Done
Building dependency tree      
Reading state information... Done
Package python-pyside.qtwebkit is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

So at this point I gave up on linux and used a Mac, which was waaay easier.

Once I had the keys on the card, to use them on linux I had to do this dance to stop gnome-keyring from ruining everything. On trusty if you use gpg2 you get this error:

$ gpg2 --card-status
gpg: OpenPGP card not available: No SmartCard daemon

but gpg 1.4 works fine. This appears to be caused by differences in how gpg 1 and 2 are packaged, gpg2 needs more packages to work.

Links to other HOWTOs

Here's a big pile of useful links:

• Yubico Yubikey 4: PGP, U2F and other things
• PGP and SSH keys on a Yubikey NEO
• Yubico on importing keys generated off-device
• Yubikey NEO and PGP
• Why master key expiry is pointless
• Trammel's OS X HOWTO
• Offline GnuPG Master Key and Subkeys on YubiKey NEO Smartcard
• Yubikey NEO and GPG
• Using GPG with smart cards

Return a new return_value for each call when mocking tempfile in python

Here is a python snippet that mocks out tempfile.NamedTemporaryFile() to use a StringIO and add it to an array of file handles.

The somewhat unintuitive bit here is that by passing side_effect a function mock will use the result of the function as the return_value for the mocked tempfile.NamedTemporaryFile(). Or as the docs say: "The function is called with the same arguments as the mock, and unless it returns DEFAULT, the return value of this function is used as the return value."

def _GetTempFile(self, *args, **kwargs):
  tempfile = StringIO.StringIO() 
  tempfile.flush = mock.MagicMock()
  self.output_streams.append(tempfile)
  return tempfile

with mock.patch.object(tempfile, "NamedTemporaryFile", side_effect=self._GetTempFile):
  # mocked code...

Debugging pyinstaller binary install missing import "ImportError: No module named google.protobuf"

We were missing an import in our pyinstaller, the error was:

ImportError: No module named google.protobuf

There is a simple fix for this, you just need to add an __init__.py, but after doing that it seemed pyinstaller still wasn't picking up the library. So I wanted to check what it was actually importing. Here's how you can do that. First unzip the self-extracting installer:

unzip myinstaller.exe

Then inspect it with pyi-archive_viewer (which you should have if you pip installed pyinstaller):

$ pyi-archive_viewer myprogram.exe
 pos, length, uncompressed, iscompressed, type, name
[(0, 5392645, 5392645, 0, 'z', 'out00-PYZ.pyz'),
 (5392645, 174, 239, 1, 'm', 'struct'),
 (5392819, 1161, 2711, 1, 'm', 'pyi_os_path'),
 (5393980, 4803, 13033, 1, 'm', 'pyi_archive'),
 (5398783, 3976, 13864, 1, 'm', 'pyi_importers'),
 (5402759, 1682, 3769, 1, 's', '_pyi_bootstrap'),
 (5404441, 4173, 13142, 1, 's', 'pyi_carchive'),
 (5408614, 316, 646, 1, 's', 'pyi_rth_Tkinter'),
 (5408930, 433, 918, 1, 's', 'pyi_rth_pkgres'),
 (5409363, 981, 2129, 1, 's', 'pyi_rth_win32comgenpy'),
 (5410344, 941, 2291, 1, 's', 'client')]
? O out00-PYZ.pyz
{'BaseHTTPServer': (False, 1702685L, 8486),
 'ConfigParser': (False, 1070260L, 9034),
 'Cookie': (False, 2449334L, 9023),
 'Crypto': (True, 1151586L, 621),
 'Crypto.Cipher': (True, 4004931L, 1053),
 'Crypto.Cipher.AES': (False, 2553735L, 1656),
 'Crypto.Cipher.ARC4': (False, 5042987L, 1762),

[snip]

 'google': (True, 3743324L, 102),
 'google.protobuf': (True, 328350L, 111),
 'google.protobuf.descriptor': (False, 2112676L, 9136),
 'google.protobuf.descriptor_database': (False, 344343L, 1713),
 'google.protobuf.descriptor_pb2': (False, 4695199L, 6549),
 'google.protobuf.descriptor_pool': (False, 2080592L, 6890),
 'google.protobuf.internal': (True, 1403354L, 120),
 'google.protobuf.internal.api_implementation': (False, 5300293L, 697),
 'google.protobuf.internal.containers': (False, 4954671L, 3244),
 'google.protobuf.internal.cpp_message': (False, 1447735L, 8287),
 'google.protobuf.internal.decoder': (False, 1318001L, 7942),
 'google.protobuf.internal.encoder': (False, 5337352L, 7379),
 'google.protobuf.internal.enum_type_wrapper': (False, 5202459L, 1039),
 'google.protobuf.internal.message_listener': (False, 2642102L, 1112),
 'google.protobuf.internal.python_message': (False, 4574735L, 13240),
 'google.protobuf.internal.type_checkers': (False, 4723000L, 3257),
 'google.protobuf.internal.wire_format': (False, 4040106L, 2834),
 'google.protobuf.message': (False, 370977L, 3410),
 'google.protobuf.pyext': (True, 650580L, 116),
 'google.protobuf.pyext.cpp_message': (False, 349637L, 763),
 'google.protobuf.reflection': (False, 2578008L, 2557),
 'google.protobuf.symbol_database': (False, 3971343L, 2075),
 'google.protobuf.text_encoding': (False, 2736366L, 1447),
 'google.protobuf.text_format': (False, 4991343L, 8425),

This shows that google.protobuf was successfully included by pyinstaller. In my case pyinstaller was doing the right thing, I had another step in the installer generation process that was using an older version of the installer that was still missing the proto library.

Patching a python class attribute in a unit test using mock

I often want to change a class attribute in a unit test. While it is sort of documented in the official documentation, it's missing an actual example of this use case. So say you have a class:

class MyClass(object):
  myattr = 1

and you want to test the functionality of the class, but need to change the value of myattr from the default:

import mock
import modulename

with mock.patch.object(modulename.MyClass, "myattr", new=50):
  # do test stuff

Snippet for verifying a SHA256 hash of a downloaded file in bash

This snippet will download openssl and verify the hash against a hard-coded value. This is useful on old operating systems (specifically CentOS 5.11) that can't actually establish an SSL connection to lots of sites.

#!/bin/bash

SSL_VERSION=1.0.2d
SSL_SHA256=671c36487785628a703374c652ad2cebea45fa920ae5681515df25d9f2c9a8c8
RETRIEVED_HASH=$(wget -q -O - http://www.openssl.org/source/openssl-${SSL_VERSION}.tar.gz | tee openssl-${SSL_VERSION}.tar.gz | sha256sum | cut -d' ' -f1)
if [ "${RETRIEVED_HASH}" != "${SSL_SHA256}" ]; then
  echo "Bad hash for openssl-${SSL_VERSION}.tar.gz, quitting"
  exit 1
fi