semi/signal

I’ve been working a lot with Kubernetes this year and an interesting problem surfaced when attempting to get a container to terminate gracefully.

What Kubernetes does

The Kubernetes termination lifecycle is detail in Kubernetes best practices: terminating with grace, but the most important bits are:

• When a pod is set to the Terminating state, all containers are sent SIGTERM
• Kubernetes waits a grace period (default is 30s) for containers to handle SIGTERM
  • If a container process has no handler for SIGTERM, the Linux kernel will kill the process immediately
  • If a container process does have a handler for SIGTERM, the handler can do whatever is needed to wrap up, then exit
• At the end of the grace period, any container still alive is sent SIGKILL and deleted

This is all very reasonable but it does depends on container processes handling SIGTERM or not handling SIGTERM and letting the kernel kill the process.

Not having a handler for SIGTERM

In general, without an explicit handler for SIGTERM, the kernel will kill the process. However, there is one very important exception, a process having Process ID (PID) 1 will not be killed, as PID 1 is not killable via signals.

Note that this is true for all termination signals, SIGKILL won’t have an impact either

When a container is run (e.g. via docker run), whatever process is started, from the declared entrypoint, is PID 1.

As for why PID 1 is unkillable, I couldn’t find an exact reason, but given that PID 1 is usually for an init daemon, I’d wager it’s due to the importance of init, as it’s the ancestor of all userspaces processes. That said, while this all makes sense in the context of a full-featured Linux distro, this protection is questionable when it comes to running processes within a container.

Responding to SIGTERM

So, for processes without an explicit signal handler, when Kubernetes issues SIGTERM, nothing will happen. The process will simply keep running. Only after the termination grace period, when Kubernetes forcible deletes the container, will the process be killed. In some cases, this is not a problem (the termination grace period is doesn’t matter and graceful termination isn’t a concern) but for programs that handle long-running tasks, it can certainly be an issue.

If the process running is an application or script where the source code is available, the solution is obvious, write a handler for SIGTERM.

If writing a handler isn’t possible, add an init program to the container and use it to run the application or script. tini works great here, as it’s lightweight and designed for containers (it’s also what docker run uses when the --init flag specified).

I was playing around a bit with Desktop Entry files which provide a nice facade for hiding the execution details of a desktop application. However, a somewhat odd limitation is that relative paths are not supported. At least for the Exec key, I found a nice solution which makes use of bash and the %k field code allowed for the Exec value.

The code above presumes that the application to run is at runner-linux-x86_64/dist/bin/run, relative to the location of the .desktop file.

I got the above error while attempting to install updates on a Linux Mint system. This thread provided a few possible solutions, none of which worked for me (though I was glad I discovered the grub » recovery mode » repair broken packages option, which might come in handy in the future). Digging a bit, I ran:

and noticed,

There was no /var/cache/debconf/ directory and, as indicated here, simply creating it would provide a fix. So a simple:

… and I was good to go!

Like many (I’m guessing) I was under the assumption that processes incurred a higher cost on performance compared to threads. On Linux, at least, this appears to not be the case,

Linux uses a 1-1 threading model, with (to the kernel) no distinction between processes and threads — everything is simply a runnable task…