semi/signal

I’ve been working a lot with Kubernetes this year and an interesting problem surfaced when attempting to get a container to terminate gracefully.

What Kubernetes does

The Kubernetes termination lifecycle is detail in Kubernetes best practices: terminating with grace, but the most important bits are:

• When a pod is set to the Terminating state, all containers are sent SIGTERM
• Kubernetes waits a grace period (default is 30s) for containers to handle SIGTERM
  • If a container process has no handler for SIGTERM, the Linux kernel will kill the process immediately
  • If a container process does have a handler for SIGTERM, the handler can do whatever is needed to wrap up, then exit
• At the end of the grace period, any container still alive is sent SIGKILL and deleted

This is all very reasonable but it does depends on container processes handling SIGTERM or not handling SIGTERM and letting the kernel kill the process.

Not having a handler for SIGTERM

In general, without an explicit handler for SIGTERM, the kernel will kill the process. However, there is one very important exception, a process having Process ID (PID) 1 will not be killed, as PID 1 is not killable via signals.

Note that this is true for all termination signals, SIGKILL won’t have an impact either

When a container is run (e.g. via docker run), whatever process is started, from the declared entrypoint, is PID 1.

As for why PID 1 is unkillable, I couldn’t find an exact reason, but given that PID 1 is usually for an init daemon, I’d wager it’s due to the importance of init, as it’s the ancestor of all userspaces processes. That said, while this all makes sense in the context of a full-featured Linux distro, this protection is questionable when it comes to running processes within a container.

Responding to SIGTERM

So, for processes without an explicit signal handler, when Kubernetes issues SIGTERM, nothing will happen. The process will simply keep running. Only after the termination grace period, when Kubernetes forcible deletes the container, will the process be killed. In some cases, this is not a problem (the termination grace period is doesn’t matter and graceful termination isn’t a concern) but for programs that handle long-running tasks, it can certainly be an issue.

If the process running is an application or script where the source code is available, the solution is obvious, write a handler for SIGTERM.

If writing a handler isn’t possible, add an init program to the container and use it to run the application or script. tini works great here, as it’s lightweight and designed for containers (it’s also what docker run uses when the --init flag specified).

Like many (I’m guessing) I was under the assumption that processes incurred a higher cost on performance compared to threads. On Linux, at least, this appears to not be the case,

Linux uses a 1-1 threading model, with (to the kernel) no distinction between processes and threads — everything is simply a runnable task…

So, a little story. I was working on some code that periodically spawns off a child process to do its thing, then read the results in the output stream from the parent process. Now, this periodic spawning was being done asynchronously in the parent process via. a thread. All was well and good, but there was also another thread in the parent process which created a file with a temporary name, downloaded and wrote a bunch of bytes into it, then renamed the file to a proper name. Unfortunately, I began to notice that the rename operation was failing as the parent process couldn’t get access rights to the file that needed to be renamed due to a sharing violation. I checked, double checked, and triple checked that I was closing the file, and everything looked perfect. After a bit of headbanging, I figured I should probably check to verify that another process is not holding the file handle. I then ran a wonderful utility called Handle to see exactly what processes had the file handle and, to my surprise, it was a child process that was spawned. Now, this was weird as hell as the child process did nothing with this file or directory or any file i/o whatsoever. After a bit more headbanging, I made the unpleasant discovery of child process inheritance. This forum thread, discussing a similar issue with a .NET’s TcpListener, actually pointed me to the issue.

Now most of this was in C#. One tricky aspect of this issue is that the file i/o mentioned was done in a bit of native code using fopen/fclose. I never encountered the issue with similar code using a C# FileStream. My assumption is that the file handles from the C# functions were explicitly not inheritable, while those created by fopen were. The underlying Win32 CreateFile API function does provide for this feature, but it’s not exposed via. fopen.

If this parameter is NULL, the handle returned by CreateFile cannot be inherited by any child processes the application may create and the file or device associated with the returned handle gets a default security descriptor.

CreateFile ignores the lpSecurityDescriptor member when opening an existing file or device, but continues to use the bInheritHandle member.

The bInheritHandle member of the structure specifies whether the returned handle can be inherited.

Now it was time for the really tricky part, fixing this. I didn’t want to change the native code (which in retrospect may have been an appropriate course of action and much easier to do), so instead I tried to see if I could prevent the process from inheriting the handle. The Win32 CreateProcess function has a bInheritHandles argument that can be set to false to prevent child processes from inheriting handles. Unfortunately, I was using the C# Process class and it provides no means to set such a flag. I eventually P/Invoked the CreateProcess function (this blog entry helped a great deal), but faced disappointment as I discovered that I can’t redirect standard output from the child process without bInheritHandles being set to true. I eventually altered the child process’ code to write its output to a file (which was actually better behavior for the app) and finally closed the door on this issue.