Systemd learnings: templates for multiple services, target vs. service, service grouping

My notes on best practices for creating systemd units for a package with multiple services.

Packages install into /lib/systemd/system/ directory, which I found surprising since there is also /etc/systemd. The /etc/systemd directory is actually used as a way for users to override settings from the original package via "drop-in" unit files.

Attaching your service to the multi-user target like this:

WantedBy=multi-user.target

essentially causes a symlink to be created in the relevant .wants directory when you enable the service. There's some instructions floating around the internet where people create these symlinks manually - there's no need to do that, let systemctl do it for you:

$ ls /lib/systemd/system/multi-user.target.wants/
console-setup.service  getty.target           plymouth-quit-wait.service      systemd-logind.service                systemd-user-sessions.service
dbus.service           plymouth-quit.service  systemd-ask-password-wall.path  systemd-update-utmp-runlevel.service

Running multiple copies of a service and grouping services are all much easier than with System V or upstart. Dependency resolution is powerful but a little confusing: e.g. if you look at the getty.target:

$ cat /lib/systemd/system/getty.target
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Login Prompts
Documentation=man:systemd.special(7) man:systemd-getty-generator(8)
Documentation=http://0pointer.de/blog/projects/serial-console.html

it looks like it doesn't do anything. But if you look at what it wants:

$ ls /lib/systemd/system/getty.target.wants/
getty-static.service

there's something there. And it happens to be a good example of using a template to start multiple copies of a service:

$ cat /lib/systemd/system/getty-static.service 
[Unit]
Description=getty on tty2-tty6 if dbus and logind are not available
ConditionPathExists=/dev/tty2
ConditionPathExists=!/lib/systemd/system/dbus.service

[Service]
Type=oneshot
ExecStart=/bin/systemctl --no-block start getty@tty2.service getty@tty3.service getty@tty4.service getty@tty5.service getty@tty6.service
RemainAfterExit=true

but where did that .wants come from? It's the template itself that creates the dependency:

$ cat /lib/systemd/system/getty@.service

...[snip]

[Install]
WantedBy=getty.target
DefaultInstance=tty1

OpenVPN have done a good job of using systemd to reduce complexity and simplify customization of their server init scripts. They have a template:

$ cat /lib/systemd/system/openvpn@.service 
[Unit]
Description=OpenVPN connection to %i
PartOf=openvpn.service
ReloadPropagatedFrom=openvpn.service
Before=systemd-user-sessions.service
Documentation=man:openvpn(8)
Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO

[Service]
PrivateTmp=true
KillMode=mixed
Type=forking
ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/%i.conf --writepid /run/openvpn/%i.pid
PIDFile=/run/openvpn/%i.pid
ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/etc/openvpn
ProtectSystem=yes
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE
LimitNPROC=10
DeviceAllow=/dev/null rw
DeviceAllow=/dev/net/tun rw

[Install]
WantedBy=multi-user.target

That means you can have multiple openvpn configs and control them as if you had written init scripts for each:

/etc/openvpn/server1
/etc/openvpn/server1

systemctl enable openvpn@server1.service
systemctl enable openvpn@server2.service
systemctl start openvpn@server1.service
systemctl start openvpn@server2.service

And all of those are grouped together using a "openvpn.service" which is referred to in PartOf in the template above so you can operate on them as a block. The ReloadPropagatedFrom tells systemd to reload the individual units when the parent is reloaded:

$ service openvpn start
$ cat openvpn.service 
# This service is actually a systemd target,
# but we are using a service since targets cannot be reloaded.

[Unit]
Description=OpenVPN service
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
ExecReload=/bin/true
WorkingDirectory=/etc/openvpn

[Install]
WantedBy=multi-user.target

The comment in that file is interesting. If you have a group of services it seems you are better off creating a service rather than a target, even though at first glance targets seem to have been created for exactly this purpose. My interpretation is that targets are essentially useful for grouping dependencies to determine execution order (i.e. they were primarily created to replace the runlevel system), but you should use a service if you expect users to want to operate on your services as a block.

You'll want to check your systemd file with systemd-analyze:

$ systemd-analyze verify /lib/systemd/system/my-server@.service 
[/lib/systemd/system/my-server@.service:6] Unknown lvalue 'Environment' in section 'Unit'
[/lib/systemd/system/my-server@.service:13] Executable path is not absolute, ignoring: mkdir -p /var/log/myserver;mkdir -p /var/run/myserver/tmp/%i