“What is Astcanary?”

basically, it is a little application that provides assurance to Asterisk that there are no threads that have gone into runaway mode, thus hogging the CPU, and making the Asterisk machine seem to be unresponsive.

It works by creating a file when Asterisk is run with a realtime priority (-p).   This file must continue to exist, and the astcanary process must be allowed to continue running, or else the Asterisk process will, within a short period of time, slow itself down to regular priority.

Technical Details:

The technical explanation for this file is to provide an assurance to Asterisk that there are no threads that have gone into runaway mode, thus hogging the CPU, and making the Asterisk machine seem to be unresponsive.  When that happens, the astcanary process will be unable to update the timestamp on this file, and Asterisk will notice within 120 seconds and react.  Slowing the Asterisk process down to regular priority will permit an administrator to intervene, thus avoiding a need to reboot the entire machine.

Where does that idea comes from?

At one time, canaries were carried along with coal miners down into a mine. Their purpose was to alert the miners when they had drilled into a pocket of methane gas or another noxious substance. The canary, being the most sensitive animal, would immediately fall over.  Seeing this, the miners could take action to escape the mine, seeing an imminent danger.

This process serves a similar purpose, though with the realtime priority being the reason.  When a thread starts running away with the processor, it is typically difficult to tell what thread caused the problem, as the machine acts as if it is locked up (in fact, what has happened is that Asterisk runs at a higher priority than even the login shell), so the runaway thread hogs all available CPU time. If that happens, this canary process will cease to get any process time, which we can monitor with a multi-threaded in Asterisk.  Should that happen, that monitoring thread may take immediate action to slow down Asterisk to regular priority, thus allowing an administrator to login to the system and restart Asterisk or perhaps take another course of action (such as retrieving a backtrace to let the developers know what precisely went wrong).

Note that according to POSIX, all threads inside a single process must share the same priority, so when the monitoring thread deprioritizes itself, it deprioritizes all threads at the same time.  This is also why this canary must exist as a completely separate process and not simply as a thread within Asterisk itself.

Quote:
“The nice value set with setpriority() shall be applied to the process. If the process is multi-threaded, the nice value shall affect all system scope threads in the process.”

Source:
http://www.opengroup.org/onlinepubs/000095399/functions/setpriority.html

Then, In answer to the question, “what aren’t system scope threads?”, the answer is, in Asterisk, nothing.  Process scope thread are the alternative, but they aren’t supported in Linux.

If you want an updated UNIX classic that covers everything you’ll ever need for threads, TCP/IP and RPC programming-with reusable code examples that explain syntax along the way, then you must read this book:

• UNIX Systems Programming: Communication, Concurrency and Threads