This post is about a program hang. The hang was in the Python process that was running Ansible scripts. The problem was hard to debug and had me go back to Unix textbook.
This quiz is about Unix signals.
Let’s say you’ve set SIGTERM to be ignored in your program. You invoke a shell script and the script itself invokes, say, python.
Can you quickly tell if you can kill the python process via SIGTERM?
As you might have guessed by now, the answer is no.
The signal disposition of your program is carried forward to your child and beyond, unless the child program resets it at startup.
Signals in Unix
Unix allows you to deal with signals in two ways.
You can ignore a signal, and the system will not deliver it to you. On the other hand, you can also specify a function that will handle the signal. The relevant system call is sigaction().
You can specify a signal mask: a series of bits that tell the system which signals not to deliver. The signals are said to be blocked, until a system call is specifically made to unblock them. The relevant system call is sigprocmask().
Signals across program copies
The crucial question is: what happens to these when you make a copy of yourself via fork, or load a different program via exec?
Here’s what happens.
If you’ve asked a signal to be ignored, the forked and execed programs will inherit it.
If you’ve set up a signal handler, the forked program retains this handler, but the exec’ed program does not. (It cannot, because the new program is entirely different code, right?)
If you’ve set up a signal mask for blocking, the forked and execed programs will inherit it.
The manual pages are quite dense about this:
After a fork(2) or vfork(2) all signals, the signal mask, the signal stack, and the restart/interrupt flags are inherited by the child.
The execve(2) system call reinstates the default action for all signals which were caught and resets all signals to be caught on the user stack. Ignored signals remain ignored; the signal mask remains the same; signals that restart pending system calls continue to do so.
ps, the standard utility to observe process state, has the following flags to check the signal details for a program. The commands below come from FreeBSD, but it will be similar on Linux or Mac OS X.
The output below is quite wide, so you may have to scroll a bit to the right.
$ ps axjf -O pending,ignored,blocked USER PID PPID PGID SID JOBC STAT TT TIME COMMAND PID PENDING IGNORED BLOCKED TT STAT TIME COMMAND ... pgxl 27053 1 26638 11566 0 I ?? 0:00.58 /usr/local/bin/p 27053 0 1948d007 5004 ?? I 0:00.58 /usr/local/bin/python -u /va ...
Note the 3 columns with headers PENDING, IGNORED and BLOCKED. They show which signals are pending delivery to the process, which are being ignored by the process, and which have been blocked.
This particular process has no pending signals (0). It is ignoring some signals, as specified by the mask 0x1948d007. It is also blocking some signals, vide the mask 0x5004.
You can decipher the mask by looking at the man page for signal, but the easier way is to use the procstat command. With the -i option, it shows which signals are being ignored (I), and with the -j option, it shows which signals are being blocked (B).
$ procstat -i 38540 PID COMM SIG FLAGS 38540 nsulfd HUP -I- 38540 nsulfd INT -I- 38540 nsulfd QUIT -I- 38540 nsulfd ILL --- 38540 nsulfd TRAP --- ... $ procstat -j 38540 PID TID COMM SIG FLAGS 38540 101220 nsulfd HUP -- 38540 101220 nsulfd INT -- 38540 101220 nsulfd QUIT -B 38540 101220 nsulfd ILL -- 38540 101220 nsulfd TRAP -- ...
The Ansible problem
With this introduction, I can now analyze the problem I had. Ansible forks a bunch of worker processes and they communicate via a queue. When the work is all done, the driver sends them a TERM signal.
But with the above setup of SIGTERM being ignored by some parent process which invoked Ansible, they never get the TERM … and they hang!
I spent a lot of time over the details of Python’s multiprocessing, decoding Python’s call stack via gdb, and of course, reading similar Ansible bug reports. The actual solution came when I noticed that another program from the same parent was also not terminating. A sharp colleague of mine connected the dots and thought this might have something to do with signal inheritance. Bingo!
View from the other side
It is possible to verify some of this behavior from the kernel sources. Here is the relevant code in the implementation of fork() in FreeBSD 8.4. You can see the mask of blocked signals, as well as handlers being passed from parent to child.
(kern/fork.c. Comments mine.)
Here is the relevant code in exec(), where the system resets signal handlers.
(kern/kern_sig.c. Comments mine.)
What’s interesting about the exec code is what it does not do, and therefore what I cannot show: it does not clear signals that are already being ignored (ps_sigignore). Also, it does not clear the signal mask (td_sigmask).
This means both of these remain the same on the exec’ed code!
There is more to Unix signals than what a blog post can discuss. For a very good treatment of the nuances in a dedicated chapter, I recommend the book Computer Systems: A Programmer’s Perspective by Bryant and O’Hallaron.