Squashing git commits into a single commit for a github pull request

There's lots of advice about how to git rebase. But unfortunately almost none of these address the case where you have a pull request on a repo with multiple merges to master. In this case you'll have lots of other people's commits in your history and you can't just do:

git rebase -i HEAD~3

Thankfully github documented the solution. 99% of the time what I want to do is squash all of my commits on a branch relative to the repo master branch. It's super easy once you know:

git rebase -i master

Systemd learnings: templates for multiple services, target vs. service, service grouping

My notes on best practices for creating systemd units for a package with multiple services.

Packages install into /lib/systemd/system/ directory, which I found surprising since there is also /etc/systemd. The /etc/systemd directory is actually used as a way for users to override settings from the original package via "drop-in" unit files.

Attaching your service to the multi-user target like this:

WantedBy=multi-user.target

essentially causes a symlink to be created in the relevant .wants directory when you enable the service. There's some instructions floating around the internet where people create these symlinks manually - there's no need to do that, let systemctl do it for you:

$ ls /lib/systemd/system/multi-user.target.wants/
console-setup.service  getty.target           plymouth-quit-wait.service      systemd-logind.service                systemd-user-sessions.service
dbus.service           plymouth-quit.service  systemd-ask-password-wall.path  systemd-update-utmp-runlevel.service

Running multiple copies of a service and grouping services are all much easier than with System V or upstart. Dependency resolution is powerful but a little confusing: e.g. if you look at the getty.target:

$ cat /lib/systemd/system/getty.target
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Login Prompts
Documentation=man:systemd.special(7) man:systemd-getty-generator(8)
Documentation=http://0pointer.de/blog/projects/serial-console.html

it looks like it doesn't do anything. But if you look at what it wants:

$ ls /lib/systemd/system/getty.target.wants/
getty-static.service

there's something there. And it happens to be a good example of using a template to start multiple copies of a service:

$ cat /lib/systemd/system/getty-static.service 
[Unit]
Description=getty on tty2-tty6 if dbus and logind are not available
ConditionPathExists=/dev/tty2
ConditionPathExists=!/lib/systemd/system/dbus.service

[Service]
Type=oneshot
ExecStart=/bin/systemctl --no-block start getty@tty2.service getty@tty3.service getty@tty4.service getty@tty5.service getty@tty6.service
RemainAfterExit=true

but where did that .wants come from? It's the template itself that creates the dependency:

$ cat /lib/systemd/system/getty@.service

...[snip]

[Install]
WantedBy=getty.target
DefaultInstance=tty1

OpenVPN have done a good job of using systemd to reduce complexity and simplify customization of their server init scripts. They have a template:

$ cat /lib/systemd/system/openvpn@.service 
[Unit]
Description=OpenVPN connection to %i
PartOf=openvpn.service
ReloadPropagatedFrom=openvpn.service
Before=systemd-user-sessions.service
Documentation=man:openvpn(8)
Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO

[Service]
PrivateTmp=true
KillMode=mixed
Type=forking
ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/%i.conf --writepid /run/openvpn/%i.pid
PIDFile=/run/openvpn/%i.pid
ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/etc/openvpn
ProtectSystem=yes
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE
LimitNPROC=10
DeviceAllow=/dev/null rw
DeviceAllow=/dev/net/tun rw

[Install]
WantedBy=multi-user.target

That means you can have multiple openvpn configs and control them as if you had written init scripts for each:

/etc/openvpn/server1
/etc/openvpn/server1

systemctl enable openvpn@server1.service
systemctl enable openvpn@server2.service
systemctl start openvpn@server1.service
systemctl start openvpn@server2.service

And all of those are grouped together using a "openvpn.service" which is referred to in PartOf in the template above so you can operate on them as a block. The ReloadPropagatedFrom tells systemd to reload the individual units when the parent is reloaded:

$ service openvpn start
$ cat openvpn.service 
# This service is actually a systemd target,
# but we are using a service since targets cannot be reloaded.

[Unit]
Description=OpenVPN service
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
ExecReload=/bin/true
WorkingDirectory=/etc/openvpn

[Install]
WantedBy=multi-user.target

The comment in that file is interesting. If you have a group of services it seems you are better off creating a service rather than a target, even though at first glance targets seem to have been created for exactly this purpose. My interpretation is that targets are essentially useful for grouping dependencies to determine execution order (i.e. they were primarily created to replace the runlevel system), but you should use a service if you expect users to want to operate on your services as a block.

You'll want to check your systemd file with systemd-analyze:

$ systemd-analyze verify /lib/systemd/system/my-server@.service 
[/lib/systemd/system/my-server@.service:6] Unknown lvalue 'Environment' in section 'Unit'
[/lib/systemd/system/my-server@.service:13] Executable path is not absolute, ignoring: mkdir -p /var/log/myserver;mkdir -p /var/run/myserver/tmp/%i

Publishing a package on PyPi

First, create a ~/.pypirc like this. You don't need to (and shouldn't!) put your cleartext password in here, you will get a prompt when you actually register.

[distutils]
index-servers =
  pypi
  pypitest

[pypi]
repository=https://pypi.python.org/pypi
username=your_username

[pypitest]
repository=https://testpypi.python.org/pypi
username=your_username

Write your setup.py:

setup(
    name="mypackage",
    version="3.1.0",
    description="My description",
    license="Apache License, Version 2.0",
    url="https://github.com/myhomepage"

Make sure your version number and other info in your setup.py is correct and then test your package on the the test server by registering:

python setup.py register -r pypitest

Then build and upload your actual file content using twine. You can also use setup.py to upload, but this way you get to inspect the build tarball before it gets uploaded:

python setup.py sdist
twine upload -r pypitest dist/*

Check that it looks OK on the test site. And that you can install it:

pip install -i https://testpypi.python.org/pypi mypackage

Then register and upload it on the production pypi server:

python setup.py register -r pypi
twine upload -r pypi dist/*

Externally hosting PyPi python packages to workaround PyPi size limit

PyPi has a limit on the size of the package they are willing to host. This doesn't seem to be documented anywhere, but I've seen people mention 60MB on forums. Our package contains a bunch of compressed binary data and weighs in at 130MB, so we needed to find another solution. The error you get from twine when uploading is this:

HTTPError: 413 Client Error: Request Entity Too Large for url: https://testpypi.python.org/pypi

But since hosting files on cloud services is now cheap and reliable we can work around the problem, as long as you're willing to have people use a custom pip command. If you point pip at a file with links using -f it will look through those links for a suitable install candidate. So if you create an index like this:

<html><head><title>Simple Index</title><meta name="api-version" value="2" /></head><body>
<a href='mypackage-3.1.0.tar.gz#md5=71525271a5fdbf0c72580ce56194d999'>mypackage-3.1.0</a><br/>
<a href='mypackage-3.1.2.tar.gz#md5=71525271a5fdbf0c72580ce56194daaa'>mypackage-3.1.2</a><br/>
</body></html>

And host it somewhere (like google cloud storage), along with the tarballs you get from running:

python setup.py sdist

Then your install command looks like this:

pip install --allow-external mypackage -f https://storage.googleapis.com/mypackage/index.html mypackage

Verify SHA256 SSH RSA key fingerprint

As of OpenSSH 6.8 the defaults is to display base64 encoded SHA256 hashes for SSH host keys, whereas previously it showed MD5 hex digests. While this is a good move for security, it's a PITA to verify host keys now, especially on systems with older OpenSSH.

For systems with modern OpenSSH, you can just ask for the sha256 version:

ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub -E sha256

If you have old ssh, you need to work it out yourself:

awk '{print $2}' /etc/ssh/ssh_host_rsa_key.pub | base64 -d | sha256sum -b | awk '{print $1}' | xxd -r -p | base64

On OS X, same thing but with slightly different options:

awk '{print $2}' /etc/ssh/ssh_host_rsa_key.pub | base64 -D | shasum -a 256 -b | awk '{print $1}' | xxd -r -p | base64

Or if you have access to the server by another means you can get the server to tell you the MD5 fingerprint:

ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub -E md5

Troubleshooting Kubernetes and GCE deployment manager

I've been using the GCE deployment manager to create a GCE deployment of a complex server application running multiple docker containers.

Here's a basic command cheatsheet:

gcloud deployment-manager deployments create my-first-deployment --config test_config.yaml
gcloud deployment-manager deployments describe my-first-deployment

Once you're at the point where the actual deployment works you probably need to debug other issues. Use the GUI to ssh into your container host VM. If you only see the /pause container, something is wrong. If you do "ps -a" you should get a list of containers that have failed to start properly (it will just keep retrying).

sudo docker ps -a

You can see the configuration Kubernetes passed to the container at creation time with "inspect". This is useful for debugging configuration problems:

sudo docker inspect [container id]

You can see STDOUT for the container launch with:

sudo docker logs [container id]

One trap I fell into is that the Kubernetes use of Cmd is different to docker :( I had a custom entrypoint in my Dockerfile and called it like this with docker:

docker run mycontainer command

But in Kubernetes config speak, cmd gets translated to docker entrypoint, and args gets translated to cmd. Ugh. So assuming your entrypoint is specified in the Dockerfile you want to leave that alone and just set the args:

containers:
  - name: mycontainer
    args: ["command"]
    env:
      - name: EXTERNAL_HOSTNAME
        value: localhost
      - name: ADMIN_PASSWORD
        value: demo

When run by Kubernetes it looks something like this:

   "Config": {
        "Hostname": "mydeployment-host",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "EXTERNAL_HOSTNAME=localhost",
            "ADMIN_PASSWORD=demo",
        ],
        "Cmd": [
            "command"
        ],
        "Image": "mydocker:latest",
        "Entrypoint": [
            "/mycustom-entrypoint.sh"
        ],
    }

The horror of signing RPMs that support CentOS 5

If you're reading this you are probably just starting to realise what a shit pile you've stepped in. You just want to have a signed RPM that's installable on CentOS 5+ right? The one you built worked fine on CentOS 7, but on CentOS 5 you saw something like this:

$ sudo rpm -i package.rpm 
error: package.rpm: Header V4 RSA/SHA1 signature: BAD, key ID 1234567
error: package.rpm cannot be installed
$ rpm --version
RPM version 4.4.2.3

It turns out that CentOS 5 doesn't support V4 signatures, is very picky about whether your public key has subkeys, and none of this is documented outside of an ancient bug and a bunch of angry blog posts and stack overflow questions. If you read all of that you'll get a bunch of conflicting advice, so I'll add another shout into the wind that might help someone in the future.  Here's a working setup:


Signing system is Ubuntu trusty:

$ lsb_release -rd
Description: Ubuntu 14.04.1 LTS
Release: 14.04
$ rpmsign --version
RPM version 4.11.1
$ rpmsign --define "%_gpg_name My GPGName" --define "__gpg_sign_cmd %{__gpg} gpg --force-v3-sigs --digest-algo=sha1 --batch --no-verbose --no-armor --passphrase-fd 3 --no-secmem-warning -u \\\"%{_gpg_name}\\\" -sbo %{__signature_filename} %{__plaintext_filename}" --resign package.rpm
$ rpm -Kv package.rpm
package.rpm:
    Header V3 RSA/SHA1 Signature, key ID 1234567: OK
    Header SHA1 digest: OK (aaaaaaaaaaaaaaabbbbbbbbbbbb)
    V3 RSA/SHA1 Signature, key ID 1234567: OK
    MD5 digest: OK (aaaaaaaabbbbbbbbb)

Note that your signing key can have subkeys when signing (by default gpg creates a subkey), but if you just export your public key with the subkey as normal and attempt to use it for verification it will look like this (V3 sig, but still marked "BAD") on CentOS 5:

$ rpm -Kv new2.rpm 
new2.rpm:
    Header V3 RSA/SHA1 signature: BAD, key ID 1234567
    Header SHA1 digest: OK (aaaaaaaaaaaaaaabbbbbbbbbbbb)
    V3 RSA/SHA1 signature: BAD, key ID 1234567
    MD5 digest: OK (aaaaaaaabbbbbbbbb)

and since gpg doesn't seem to give you a way to export a master without subkeys, on your Ubuntu signing machine you need to delete the subkey and export again:

$ gpg --edit 1234567
gpg> key 1
gpg> delkey
gpg> save
gpg> quit

gpg --export --armor 1234567 > 1234567_master.pub

Then on your Centos 5 system (I was using 5.11):

$ sudo rpm --import 1234567_master.pub
$ rpm -Kv new2.rpm 
new2.rpm:
    Header V3 RSA/SHA1 signature: OK, key ID 1234567
    Header SHA1 digest: OK (aaaaaaaaaaaaaaabbbbbbbbbbbb)
    V3 RSA/SHA1 signature: OK, key ID 1234567
    MD5 digest: OK (aaaaaaaabbbbbbbbb)

Simple right?