Troubleshooting Kubernetes and GCE deployment manager

I've been using the GCE deployment manager to create a GCE deployment of a complex server application running multiple docker containers.

Here's a basic command cheatsheet:

gcloud deployment-manager deployments create my-first-deployment --config test_config.yaml
gcloud deployment-manager deployments describe my-first-deployment

Once you're at the point where the actual deployment works you probably need to debug other issues. Use the GUI to ssh into your container host VM. If you only see the /pause container, something is wrong. If you do "ps -a" you should get a list of containers that have failed to start properly (it will just keep retrying).

sudo docker ps -a

You can see the configuration Kubernetes passed to the container at creation time with "inspect". This is useful for debugging configuration problems:

sudo docker inspect [container id]

You can see STDOUT for the container launch with:

sudo docker logs [container id]

One trap I fell into is that the Kubernetes use of Cmd is different to docker :( I had a custom entrypoint in my Dockerfile and called it like this with docker:

docker run mycontainer command

But in Kubernetes config speak, cmd gets translated to docker entrypoint, and args gets translated to cmd. Ugh. So assuming your entrypoint is specified in the Dockerfile you want to leave that alone and just set the args:

containers:
  - name: mycontainer
    args: ["command"]
    env:
      - name: EXTERNAL_HOSTNAME
        value: localhost
      - name: ADMIN_PASSWORD
        value: demo

When run by Kubernetes it looks something like this:

   "Config": {
        "Hostname": "mydeployment-host",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "EXTERNAL_HOSTNAME=localhost",
            "ADMIN_PASSWORD=demo",
        ],
        "Cmd": [
            "command"
        ],
        "Image": "mydocker:latest",
        "Entrypoint": [
            "/mycustom-entrypoint.sh"
        ],
    }

The horror of signing RPMs that support CentOS 5

If you're reading this you are probably just starting to realise what a shit pile you've stepped in. You just want to have a signed RPM that's installable on CentOS 5+ right? The one you built worked fine on CentOS 7, but on CentOS 5 you saw something like this:

$ sudo rpm -i package.rpm 
error: package.rpm: Header V4 RSA/SHA1 signature: BAD, key ID 1234567
error: package.rpm cannot be installed
$ rpm --version
RPM version 4.4.2.3

It turns out that CentOS 5 doesn't support V4 signatures, is very picky about whether your public key has subkeys, and none of this is documented outside of an ancient bug and a bunch of angry blog posts and stack overflow questions. If you read all of that you'll get a bunch of conflicting advice, so I'll add another shout into the wind that might help someone in the future.  Here's a working setup:


Signing system is Ubuntu trusty:

$ lsb_release -rd
Description: Ubuntu 14.04.1 LTS
Release: 14.04
$ rpmsign --version
RPM version 4.11.1
$ rpmsign --define "%_gpg_name My GPGName" --define "__gpg_sign_cmd %{__gpg} gpg --force-v3-sigs --digest-algo=sha1 --batch --no-verbose --no-armor --passphrase-fd 3 --no-secmem-warning -u \\\"%{_gpg_name}\\\" -sbo %{__signature_filename} %{__plaintext_filename}" --resign package.rpm
$ rpm -Kv package.rpm
package.rpm:
    Header V3 RSA/SHA1 Signature, key ID 1234567: OK
    Header SHA1 digest: OK (aaaaaaaaaaaaaaabbbbbbbbbbbb)
    V3 RSA/SHA1 Signature, key ID 1234567: OK
    MD5 digest: OK (aaaaaaaabbbbbbbbb)

Note that your signing key can have subkeys when signing (by default gpg creates a subkey), but if you just export your public key with the subkey as normal and attempt to use it for verification it will look like this (V3 sig, but still marked "BAD") on CentOS 5:

$ rpm -Kv new2.rpm 
new2.rpm:
    Header V3 RSA/SHA1 signature: BAD, key ID 1234567
    Header SHA1 digest: OK (aaaaaaaaaaaaaaabbbbbbbbbbbb)
    V3 RSA/SHA1 signature: BAD, key ID 1234567
    MD5 digest: OK (aaaaaaaabbbbbbbbb)

and since gpg doesn't seem to give you a way to export a master without subkeys, on your Ubuntu signing machine you need to delete the subkey and export again:

$ gpg --edit 1234567
gpg> key 1
gpg> delkey
gpg> save
gpg> quit

gpg --export --armor 1234567 > 1234567_master.pub

Then on your Centos 5 system (I was using 5.11):

$ sudo rpm --import 1234567_master.pub
$ rpm -Kv new2.rpm 
new2.rpm:
    Header V3 RSA/SHA1 signature: OK, key ID 1234567
    Header SHA1 digest: OK (aaaaaaaaaaaaaaabbbbbbbbbbbb)
    V3 RSA/SHA1 signature: OK, key ID 1234567
    MD5 digest: OK (aaaaaaaabbbbbbbbb)

Simple right?