Wednesday, May 4, 2016

Detecting prelinking breakage

A long time ago RHEL made a bad decision to have prelink enabled by default. This has caused various forms of hard-to-debug heartache for people including me, as a maintainer of a pyinstaller package. The performance gains are dubious, and it causes problems with ASLR and rpm verify (since binaries are modified post-install). Thankfully I believe it is off by default in new versions of Red Hat.

Here's a quick troubleshooting guide to see if prelinking is causing unexpected modification of your binaries.

First check to see if it is enabled:
grep ^PRELINK /etc/sysconfig/prelink
You can also check to see if the binary itself is prelinked using readelf.

To disable it system-wide set "PRELINKING=no" in /etc/sysconfig/prelink and run /etc/cron.daily/prelink as root.

Tuesday, April 26, 2016

Squashing git commits into a single commit for a github pull request

There's lots of advice about how to git rebase. But unfortunately almost none of these address the case where you have a pull request on a repo with multiple merges to master. In this case you'll have lots of other people's commits in your history and you can't just do:
git rebase -i HEAD~3
Thankfully github documented the solution. 99% of the time what I want to do is squash all of my commits on a branch relative to the repo master branch. It's super easy once you know:
git rebase -i master

Friday, April 22, 2016

Systemd learnings: templates for multiple services, target vs. service, service grouping

My notes on best practices for creating systemd units for a package with multiple services.

Packages install into /lib/systemd/system/ directory, which I found surprising since there is also /etc/systemd. The /etc/systemd directory is actually used as a way for users to override settings from the original package via "drop-in" unit files.

Attaching your service to the multi-user target like this:
WantedBy=multi-user.target
essentially causes a symlink to be created in the relevant .wants directory when you enable the service. There's some instructions floating around the internet where people create these symlinks manually - there's no need to do that, let systemctl do it for you:
$ ls /lib/systemd/system/multi-user.target.wants/
console-setup.service  getty.target           plymouth-quit-wait.service      systemd-logind.service                systemd-user-sessions.service
dbus.service           plymouth-quit.service  systemd-ask-password-wall.path  systemd-update-utmp-runlevel.service
Running multiple copies of a service and grouping services are all much easier than with System V or upstart. Dependency resolution is powerful but a little confusing: e.g. if you look at the getty.target:
$ cat /lib/systemd/system/getty.target
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Login Prompts
Documentation=man:systemd.special(7) man:systemd-getty-generator(8)
Documentation=http://0pointer.de/blog/projects/serial-console.html
it looks like it doesn't do anything. But if you look at what it wants:
$ ls /lib/systemd/system/getty.target.wants/
getty-static.service
there's something there. And it happens to be a good example of using a template to start multiple copies of a service:
$ cat /lib/systemd/system/getty-static.service 
[Unit]
Description=getty on tty2-tty6 if dbus and logind are not available
ConditionPathExists=/dev/tty2
ConditionPathExists=!/lib/systemd/system/dbus.service

[Service]
Type=oneshot
ExecStart=/bin/systemctl --no-block start getty@tty2.service getty@tty3.service getty@tty4.service getty@tty5.service getty@tty6.service
RemainAfterExit=true
but where did that .wants come from? It's the template itself that creates the dependency:
$ cat /lib/systemd/system/getty@.service

...[snip]

[Install]
WantedBy=getty.target
DefaultInstance=tty1
OpenVPN have done a good job of using systemd to reduce complexity and simplify customization of their server init scripts. They have a template:
$ cat /lib/systemd/system/openvpn@.service 
[Unit]
Description=OpenVPN connection to %i
PartOf=openvpn.service
ReloadPropagatedFrom=openvpn.service
Before=systemd-user-sessions.service
Documentation=man:openvpn(8)
Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO

[Service]
PrivateTmp=true
KillMode=mixed
Type=forking
ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/%i.conf --writepid /run/openvpn/%i.pid
PIDFile=/run/openvpn/%i.pid
ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/etc/openvpn
ProtectSystem=yes
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE
LimitNPROC=10
DeviceAllow=/dev/null rw
DeviceAllow=/dev/net/tun rw

[Install]
WantedBy=multi-user.target
That means you can have multiple openvpn configs and control them as if you had written init scripts for each:
/etc/openvpn/server1
/etc/openvpn/server1

systemctl enable openvpn@server1.service
systemctl enable openvpn@server2.service
systemctl start openvpn@server1.service
systemctl start openvpn@server2.service
And all of those are grouped together using a "openvpn.service" which is referred to in PartOf in the template above so you can operate on them as a block. The ReloadPropagatedFrom tells systemd to reload the individual units when the parent is reloaded:
$ service openvpn start
$ cat openvpn.service 
# This service is actually a systemd target,
# but we are using a service since targets cannot be reloaded.

[Unit]
Description=OpenVPN service
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
ExecReload=/bin/true
WorkingDirectory=/etc/openvpn

[Install]
WantedBy=multi-user.target

The comment in that file is interesting. If you have a group of services it seems you are better off creating a service rather than a target, even though at first glance targets seem to have been created for exactly this purpose. My interpretation is that targets are essentially useful for grouping dependencies to determine execution order (i.e. they were primarily created to replace the runlevel system), but you should use a service if you expect users to want to operate on your services as a block.

You'll want to check your systemd file with systemd-analyze:
$ systemd-analyze verify /lib/systemd/system/my-server@.service 
[/lib/systemd/system/my-server@.service:6] Unknown lvalue 'Environment' in section 'Unit'
[/lib/systemd/system/my-server@.service:13] Executable path is not absolute, ignoring: mkdir -p /var/log/myserver;mkdir -p /var/run/myserver/tmp/%i

Monday, April 11, 2016

Publishing a package on PyPi

First, create a ~/.pypirc like this. You don't need to (and shouldn't!) put your cleartext password in here, you will get a prompt when you actually register.
[distutils]
index-servers =
  pypi
  pypitest

[pypi]
repository=https://pypi.python.org/pypi
username=your_username

[pypitest]
repository=https://testpypi.python.org/pypi
username=your_username

Write your setup.py:
setup(
    name="mypackage",
    version="3.1.0",
    description="My description",
    license="Apache License, Version 2.0",
    url="https://github.com/myhomepage"
Make sure your version number and other info in your setup.py is correct and then test your package on the the test server by registering:
python setup.py register -r pypitest
Check that it looks OK on the test site. And that you can install it:
pip install -i https://testpypi.python.org/pypi mypackage
Then register it on the production pypi server:
python setup.py register -r pypi
Then upload your actual file content using twine. You can also use setup.py to upload, but this way you get to inspect the build tarball before it gets uploaded:
twine upload dist/*

Friday, April 8, 2016

Externally hosting PyPi python packages to workaround PyPi size limit

PyPi has a limit on the size of the package they are willing to host. This doesn't seem to be documented anywhere, but I've seen people mention 60MB on forums. Our package contains a bunch of compressed binary data and weighs in at 130MB, so we needed to find another solution. The error you get from twine when uploading is this:
HTTPError: 413 Client Error: Request Entity Too Large for url: https://testpypi.python.org/pypi
But since hosting files on cloud services is now cheap and reliable we can work around the problem, as long as you're willing to have people use a custom pip command. If you point pip at a file with links using -f it will look through those links for a suitable install candidate. So if you create an index like this:
<html><head><title>Simple Index</title><meta name="api-version" value="2" /></head><body>
<a href='mypackage-3.1.0.tar.gz#md5=71525271a5fdbf0c72580ce56194d999'>mypackage-3.1.0</a><br/>
<a href='mypackage-3.1.2.tar.gz#md5=71525271a5fdbf0c72580ce56194daaa'>mypackage-3.1.2</a><br/>
</body></html>
And host it somewhere (like google cloud storage), along with the tarballs you get from running:
python setup.py sdist
Then your install command looks like this:
pip install --allow-external mypackage -f https://storage.googleapis.com/mypackage/index.html mypackage

Tuesday, April 5, 2016

Verify SHA256 SSH RSA key fingerprint

As of OpenSSH 6.8 the defaults is to display base64 encoded SHA256 hashes for SSH host keys, whereas previously it showed MD5 hex digests. While this is a good move for security, it's a PITA to verify host keys now, especially on systems with older OpenSSH.

For systems with modern OpenSSH, you can just ask for the sha256 version:
ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub -E sha256
If you have old ssh, you need to work it out yourself:
awk '{print $2}' /etc/ssh/ssh_host_rsa_key.pub | base64 -d | sha256sum -b | awk '{print $1}' | xxd -r -p | base64
On OS X, same thing but with slightly different options:
awk '{print $2}' /etc/ssh/ssh_host_rsa_key.pub | base64 -D | shasum -a 256 -b | awk '{print $1}' | xxd -r -p | base64

Wednesday, March 9, 2016

Troubleshooting Kubernetes and GCE deployment manager

I've been using the GCE deployment manager to create a GCE deployment of a complex server application running multiple docker containers.

Here's a basic command cheatsheet:
gcloud deployment-manager deployments create my-first-deployment --config test_config.yaml
gcloud deployment-manager deployments describe my-first-deployment
Once you're at the point where the actual deployment works you probably need to debug other issues. Use the GUI to ssh into your container host VM. If you only see the /pause container, something is wrong. If you do "ps -a" you should get a list of containers that have failed to start properly (it will just keep retrying).
sudo docker ps -a
You can see the configuration Kubernetes passed to the container at creation time with "inspect". This is useful for debugging configuration problems:
sudo docker inspect [container id]
You can see STDOUT for the container launch with:
sudo docker logs [container id]
One trap I fell into is that the Kubernetes use of Cmd is different to docker :( I had a custom entrypoint in my Dockerfile and called it like this with docker:
docker run mycontainer command
But in Kubernetes config speak, cmd gets translated to docker entrypoint, and args gets translated to cmd. Ugh. So assuming your entrypoint is specified in the Dockerfile you want to leave that alone and just set the args:
containers:
  - name: mycontainer
    args: ["command"]
    env:
      - name: EXTERNAL_HOSTNAME
        value: localhost
      - name: ADMIN_PASSWORD
        value: demo

When run by Kubernetes it looks something like this:
   "Config": {
        "Hostname": "mydeployment-host",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "EXTERNAL_HOSTNAME=localhost",
            "ADMIN_PASSWORD=demo",
        ],
        "Cmd": [
            "command"
        ],
        "Image": "mydocker:latest",
        "Entrypoint": [
            "/mycustom-entrypoint.sh"
        ],
    }