Choosing an init process for multi-process containers

If you are developing containers you must have heard the “single process per container” mantra. Inherently, there’s nothing wrong¹ with running multiple processes in a container, as long as your ENTRYPOINT is a proper init process. Some use cases are having processes are aiding each other (such as a sidecar proxy process) or porting legacy applications.

Recently, I had to spawn a sidecar process inside a container. Docker’s own tutorial for running multiple processes in a container is a good place to start, but not production-ready. So I outsourced my quest on Twitter to find an init replacement that can:

1. run multiple child processes, but do not restart them
2. exit as soon as a child process terminates (no point of restarting child processes, let the container crash to be restarted by docker or Kubernetes)
3. fulfill PID 1 (init process) responsibilities like zombie child reaping and signal forwarding.

In this article I explored pros and cons of some of the options like supervisord, runit, monit, tini/dumb-init, s6 (audience favorite), and tini+bash4.x combo (personal favorite).

supervisord

Supervisord is from a times when we people actively managed Linux servers and what daemons we run on them. It gives a higher level abstraction than writing upstart/systemd scripts for supervising services.

Pros:

• easy to learn and use service configuration language

Cons:

• requires Python runtime in the container image (usually a no-go)
• requires you to write a config file (supervisord.conf)
• doesn’t let you exit when a supervised process terminates, as supervisord itself is meant to run indefinitely.

monit

Monit is pretty decent but it requires a configuration file written in a fairly complex DSL, and its child process supervision is pidfile-oriented which is an overkill for what I am trying to do.

Pros:

• small (written in C), suitable for bundling in a container image

Cons:

• requires you to write a config file
• checks on child health by polling at intervals with pidfiles (and not through SIGCHLD signal)
• doesn’t let you exit when a process terminates (tends to restart processes)

runit

runit is a legitimate replacement for OS-level /sbin/init processes like sysvinit, upstart or systemd –and it’s an overkill for containers.

s6 (audience favorite)

With s6-overlay project and its awesome documentation, this came up on many twitter responses I got from Andrej Baran, Joe Miller, Paul Tinsley and Wadim Kruse. I will write a separate article on how to use this in the future.

Pros:

• small (written in C), suitable for bundling in a container image
• thanks to s6-overlay, optimized for docker container images
• declarative service model by creating /etc/services/{name}/ directories that have run and finish scripts.
• can terminate itself when a child process exits

tini or dumb-init

tini and dumb-init both are written for Docker containers specifically, because most container entrypoint processes (like python, java) can’t fulfill the PID 1 responsibilities properly.

Today, when you do docker run command with --init option, your container’s original entrypoint will be replaced by tini, and executed as its subprocess.

Pros:

• small (written in C) and container-optimized
• production-hardened to work well in containers

Cons:

• can’t run multiple direct child processes as it’s designed to have 1 child process (which is docker container’s original entrypoint)

tini + bash 4.x

This is not a complete solution, but gets the job done if you don’t care about graceful termination (through signal forwarding to children).

Since tini(1) alone is not capable of running multiple child processes, bash gives us an escape hatch: Have a bash script entrypoint where you start processes in the background and exit immediately when one of the background processes terminate using the bash 4.x builtin wait -n command:

#!/usr/bin/env bash
set -e

program1 &
program2 &
wait -n

Then in your Dockerfile, modify the entrypoint:

ENTRYPOINT ["/bin/tini", "--", "entrypoint.sh"]

Pros:

• simple, tini(1) is container-optimized and small, handles zombie reaping etc.
• easily terminates when a child process exits (while preserving exit code)

Cons:

• no signal forwarding: your container will still exit, but you lose the graceful termination opportunity.
• you have to write a small custom bash script entrypoint and ship bash 4.x
• similarly, when a subprocess terminates, the other process will not get a graceful termination notice as bash will just exit.

I learned about init processes a lot from this Twitter thread I started. Thanks a lot to those participated in the discussion (Tibor Vass, Andrej Baran, Ricardo Katz, Joe Miller, Paul Tinsley, Berk Ülsoy, and Wadim Kruse and Vincent Demeester).

I’m hoping to follow up with a new blog post with an example of using s6-overlay to run multiple processes in a container image.