Squashing git commits into a single commit for a github pull request

There's lots of advice about how to git rebase. But unfortunately almost none of these address the case where you have a pull request on a repo with multiple merges to master. In this case you'll have lots of other people's commits in your history and you can't just do:

git rebase -i HEAD~3

Thankfully github documented the solution. 99% of the time what I want to do is squash all of my commits on a branch relative to the repo master branch. It's super easy once you know:

git rebase -i master

Systemd learnings: templates for multiple services, target vs. service, service grouping

My notes on best practices for creating systemd units for a package with multiple services.

Packages install into /lib/systemd/system/ directory, which I found surprising since there is also /etc/systemd. The /etc/systemd directory is actually used as a way for users to override settings from the original package via "drop-in" unit files.

Attaching your service to the multi-user target like this:

WantedBy=multi-user.target

essentially causes a symlink to be created in the relevant .wants directory when you enable the service. There's some instructions floating around the internet where people create these symlinks manually - there's no need to do that, let systemctl do it for you:

$ ls /lib/systemd/system/multi-user.target.wants/
console-setup.service  getty.target           plymouth-quit-wait.service      systemd-logind.service                systemd-user-sessions.service
dbus.service           plymouth-quit.service  systemd-ask-password-wall.path  systemd-update-utmp-runlevel.service

Running multiple copies of a service and grouping services are all much easier than with System V or upstart. Dependency resolution is powerful but a little confusing: e.g. if you look at the getty.target:

$ cat /lib/systemd/system/getty.target
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Login Prompts
Documentation=man:systemd.special(7) man:systemd-getty-generator(8)
Documentation=http://0pointer.de/blog/projects/serial-console.html

it looks like it doesn't do anything. But if you look at what it wants:

$ ls /lib/systemd/system/getty.target.wants/
getty-static.service

there's something there. And it happens to be a good example of using a template to start multiple copies of a service:

$ cat /lib/systemd/system/getty-static.service 
[Unit]
Description=getty on tty2-tty6 if dbus and logind are not available
ConditionPathExists=/dev/tty2
ConditionPathExists=!/lib/systemd/system/dbus.service

[Service]
Type=oneshot
ExecStart=/bin/systemctl --no-block start getty@tty2.service getty@tty3.service getty@tty4.service getty@tty5.service getty@tty6.service
RemainAfterExit=true

but where did that .wants come from? It's the template itself that creates the dependency:

$ cat /lib/systemd/system/getty@.service

...[snip]

[Install]
WantedBy=getty.target
DefaultInstance=tty1

OpenVPN have done a good job of using systemd to reduce complexity and simplify customization of their server init scripts. They have a template:

$ cat /lib/systemd/system/openvpn@.service 
[Unit]
Description=OpenVPN connection to %i
PartOf=openvpn.service
ReloadPropagatedFrom=openvpn.service
Before=systemd-user-sessions.service
Documentation=man:openvpn(8)
Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO

[Service]
PrivateTmp=true
KillMode=mixed
Type=forking
ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/%i.conf --writepid /run/openvpn/%i.pid
PIDFile=/run/openvpn/%i.pid
ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/etc/openvpn
ProtectSystem=yes
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE
LimitNPROC=10
DeviceAllow=/dev/null rw
DeviceAllow=/dev/net/tun rw

[Install]
WantedBy=multi-user.target

That means you can have multiple openvpn configs and control them as if you had written init scripts for each:

/etc/openvpn/server1
/etc/openvpn/server1

systemctl enable openvpn@server1.service
systemctl enable openvpn@server2.service
systemctl start openvpn@server1.service
systemctl start openvpn@server2.service

And all of those are grouped together using a "openvpn.service" which is referred to in PartOf in the template above so you can operate on them as a block. The ReloadPropagatedFrom tells systemd to reload the individual units when the parent is reloaded:

$ service openvpn start
$ cat openvpn.service 
# This service is actually a systemd target,
# but we are using a service since targets cannot be reloaded.

[Unit]
Description=OpenVPN service
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
ExecReload=/bin/true
WorkingDirectory=/etc/openvpn

[Install]
WantedBy=multi-user.target

The comment in that file is interesting. If you have a group of services it seems you are better off creating a service rather than a target, even though at first glance targets seem to have been created for exactly this purpose. My interpretation is that targets are essentially useful for grouping dependencies to determine execution order (i.e. they were primarily created to replace the runlevel system), but you should use a service if you expect users to want to operate on your services as a block.

You'll want to check your systemd file with systemd-analyze:

$ systemd-analyze verify /lib/systemd/system/my-server@.service 
[/lib/systemd/system/my-server@.service:6] Unknown lvalue 'Environment' in section 'Unit'
[/lib/systemd/system/my-server@.service:13] Executable path is not absolute, ignoring: mkdir -p /var/log/myserver;mkdir -p /var/run/myserver/tmp/%i

Publishing a package on PyPi

First, create a ~/.pypirc like this. You don't need to (and shouldn't!) put your cleartext password in here, you will get a prompt when you actually register.

[distutils]
index-servers =
  pypi
  pypitest

[pypi]
repository=https://pypi.python.org/pypi
username=your_username

[pypitest]
repository=https://testpypi.python.org/pypi
username=your_username

Write your setup.py:

setup(
    name="mypackage",
    version="3.1.0",
    description="My description",
    license="Apache License, Version 2.0",
    url="https://github.com/myhomepage"

Make sure your version number and other info in your setup.py is correct and then test your package on the the test server by registering:

python setup.py register -r pypitest

Then build and upload your actual file content using twine. You can also use setup.py to upload, but this way you get to inspect the build tarball before it gets uploaded:

python setup.py sdist
twine upload -r pypitest dist/*

Check that it looks OK on the test site. And that you can install it:

pip install -i https://testpypi.python.org/pypi mypackage

Then register and upload it on the production pypi server:

python setup.py register -r pypi
twine upload -r pypi dist/*

Externally hosting PyPi python packages to workaround PyPi size limit

PyPi has a limit on the size of the package they are willing to host. This doesn't seem to be documented anywhere, but I've seen people mention 60MB on forums. Our package contains a bunch of compressed binary data and weighs in at 130MB, so we needed to find another solution. The error you get from twine when uploading is this:

HTTPError: 413 Client Error: Request Entity Too Large for url: https://testpypi.python.org/pypi

But since hosting files on cloud services is now cheap and reliable we can work around the problem, as long as you're willing to have people use a custom pip command. If you point pip at a file with links using -f it will look through those links for a suitable install candidate. So if you create an index like this:

<html><head><title>Simple Index</title><meta name="api-version" value="2" /></head><body>
<a href='mypackage-3.1.0.tar.gz#md5=71525271a5fdbf0c72580ce56194d999'>mypackage-3.1.0</a><br/>
<a href='mypackage-3.1.2.tar.gz#md5=71525271a5fdbf0c72580ce56194daaa'>mypackage-3.1.2</a><br/>
</body></html>

And host it somewhere (like google cloud storage), along with the tarballs you get from running:

python setup.py sdist

Then your install command looks like this:

pip install --allow-external mypackage -f https://storage.googleapis.com/mypackage/index.html mypackage

Verify SHA256 SSH RSA key fingerprint

As of OpenSSH 6.8 the defaults is to display base64 encoded SHA256 hashes for SSH host keys, whereas previously it showed MD5 hex digests. While this is a good move for security, it's a PITA to verify host keys now, especially on systems with older OpenSSH.

For systems with modern OpenSSH, you can just ask for the sha256 version:

ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub -E sha256

If you have old ssh, you need to work it out yourself:

awk '{print $2}' /etc/ssh/ssh_host_rsa_key.pub | base64 -d | sha256sum -b | awk '{print $1}' | xxd -r -p | base64

On OS X, same thing but with slightly different options:

awk '{print $2}' /etc/ssh/ssh_host_rsa_key.pub | base64 -D | shasum -a 256 -b | awk '{print $1}' | xxd -r -p | base64

Or if you have access to the server by another means you can get the server to tell you the MD5 fingerprint:

ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub -E md5