Advices When Asterisk Segfaults And Nothing Useful In Logs

Home » Asterisk Users » Advices When Asterisk Segfaults And Nothing Useful In Logs
Asterisk Users 6 Comments

Hello,

I’ve got a 13.13.1 system using PJSIP stack on debian Jessie. It runs from 50 to 100 simultaneous calls (so 100 to 200 PJSIP channels)
all day long. From time to time, roughly meaning once a month, it segfaults with lines
(from dmesg -T output) like this:
asterisk[1160]: segfault at 7efffffe ip 00000000005881d6 sp
00007fec95c33910 error 4 in asterisk[400000+2a2000]

Debug level was unfortunately not set in asterisk.conf but verbose level was set to 5. Asterisk runs with:
/usr/sbin/asterisk -U asterisk -G asterisk -g

Asterisk is compiled with DONT_OPTIMIZE and BETTER_BACKTRACES options.

“core show settings” outputs:
* Directories
———–

6 thoughts on - Advices When Asterisk Segfaults And Nothing Useful In Logs

  • Yes if -g is set and the user that’s running asterisk has permissions to set ulimit -c.

    No because it’s the system that determines where a coredump goes and actually writes it, not asterisk. It’s the sysctl kernel.core_pattern setting.

    See above.

    Nope. Not a thing. Sorry.

    If you can at least get the system to write a coredump file, there are new utilities in /var/lib/asterisk/scripts, namely ast_coredumper which can help create the backtraces if it can at least find the core file. Just run
    “./ast_coredumper –help” for more info. You should also be able to use those utilities with earlier Asterisk 13 versions.

  • Oh yeah, and it’s on my list to publish instructions on how ot use those utilities but they were just released yesterday.

  • Hi George,

    How does ast_coredumper compare to ast_grab_core ) ?
    Is it worth learning to use both or shall favor one ?

    PS: As I don’t know either program, yet, my question may seem silly. Please, forgive me for this

    2017-02-14 22:52 GMT+01:00 George Joseph :

  • Not silly at all.

    ast_grab_core actually kills asterisk to get the core file while ast_coredumper dumps the core file and let’s asterisk continue. If asterisk is truly deadlocked this may not matter but in some situations you might not want to kill asterisk.

    If asterisk was compiled with DEBUG_THREADS, ast_coredumper dumps the locks table which is critical in debugging deadlock scenarios. It does this from the coredump, as opposed to running “core show locks”, which makes the locks table consistent with the rest of the backtraces.

    ast_coiredumper can also find and process existing coredumps and it can create a tarball containing the backtraces and lock tables.

  • 2017-02-17 14:39 GMT+01:00 George Joseph :

    Would you then advise to run an Asterisk server in production with DEBUG_THREADS enabled ?
    Page [1] does not mention to do so nor to avoid it.

    In the production environment I’m thinking about, Asterisk is compiled with DONT_OPTIMIZE and BETTER_BACKTRACES
    hoping that if Asterisk ever fails, sysadmins would have everything needed to track the issue no matter if those would increase resources needed to run Asterisk properly.

    [1]
    https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-GettingInformationForADeadlock

  • No, you should not leave DEBUG_THREADS on in a normal production environment. Only enable it when you are actually hunting for a deadlock. DEBUG_THREADS causes a noticeable drop in performance.

    Richard