Detected time shift.

Home » Asterisk Users » Detected time shift.
Asterisk Users 6 Comments

Hello list,

I’m experiencing a lot of server freezes lately. The server just… freezes.

I notice in the log files (/var/log/asterisk/messages &
/var/log/messages) that logging stops at the time the server hangs.
Logging continues when the server has been restarted (which is the only
solution).

So it is not a proces that hangs, it’s the entire server (CentOS5.5 +
Asterisk + MySQL).

I really have no idea what can be causing these sudden freezes. Memory
stays mostly at 250MB of 512 MB total, CPU is 97% to 100% idle…

/var/log/asterisk/debug tells me nothing, no lines that indicate
something strange before the freeze (debug level 9).

I have no core.pid file in /tmp, when I look after rebooting the server.

The only thing I have is a high level of mentionning of “kernel: dahdi:
Detected time shift.” in /var/log/messages.

What is causing this kernel message ? Could this be the cause of the
server freeze ?

Thank you for every feedback you can give me.

Kind regards,
Jonas.

6 thoughts on - Detected time shift.

  • When a system does not have a hardware device installed DAHDI uses the
    system time in order to approximate a telephony clock source (what was
    previously referred to as dahdi_dummy). Since it uses the system time
    to accomplish this, if there is a large gap in the time (typically
    because NTP is adjusting the time) DAHDI just reports that it thinks
    it’s very far behind or ahead of where it should be, and doesn’t even
    try to “mix” any audio for that interval.

    So, I think the “detected time shift” is more a symptom of something
    else causing locks as opposed to the source.

    Is your server keeping accurate wall-time?

  • On Wed, Nov 24, 2010 at 11:43 AM, Jonas Kellens
    wrote:

    Jonas,

    Do you have a monitor attached to the server? If the kernel is
    crashing, you might be able to catch the stack trace there.

    -M

  • Another case where I regularly see “detected time shift” is when I’m
    running DAHDI in a virtual machine and go through a suspend / resume
    cycle. On resume when the system clock is updated DAHDI will notice the
    shift.

  • I know there was a patch for dahdi to fix server lockups on time shift. (not
    sure what version, but if you changed the time, the server would just go
    crash.)

    Do you have the latest version ?

    Check your ntpd settings to make sure your time isn’t bouncing all over the
    place.

    William Stillwell

    href=”mailto:asterisk-users-bounces@lists.digium.com”>asterisk-users-bounces@lists.digium.com
    [mailto:asterisk-users-bounces@lists.digium.com] On Behalf Of Jonas Kellens
    Sent: Wednesday, November 24, 2010 11:44 AM

    Hello list,

    I’m experiencing a lot of server freezes lately. The server just… freezes.

    I notice in the log files (/var/log/asterisk/messages & /var/log/messages)
    that logging stops at the time the server hangs. Logging continues when the
    server has been restarted (which is the only solution).

    So it is not a proces that hangs, it’s the entire server (CentOS5.5 +
    Asterisk + MySQL).

    I really have no idea what can be causing these sudden freezes. Memory stays
    mostly at 250MB of 512 MB total, CPU is 97% to 100% idle…

    /var/log/asterisk/debug tells me nothing, no lines that indicate something
    strange before the freeze (debug level 9).

    I have no core.pid file in /tmp, when I look after rebooting the server.

    The only thing I have is a high level of mentionning of “kernel: dahdi:
    Detected time shift.” in /var/log/messages.

    What is causing this kernel message ? Could this be the cause of the server
    freeze ?

    Thank you for every feedback you can give me.

    Kind regards,
    Jonas.

  • Hello.

    Thank you for the feedback.

    To reply to all the information :

    @ Shaun Ruffell : What do you mean by Wall time ?
    This server is indeed also time server (ntpd is running)

    @ Mark Deneen : No, no monitor attached. This is a Xen VPS. I do have a
    VPS interface, but this is also frozen when the server hangs…

    @ William Stillwell : I run Asterisk 1.6.2.10 with Dahdi 2.4.0 (as
    timing source). Time on my server seems very consistent. When doing
    /usr/sbin/ntpdate (once a month) there is a very very small offset.

    Kind regards,
    Jonas.

    href=”mailto:asterisk-users-bounces@lists.digium.com”>asterisk-users-bounces@lists.digium.com

  • Basically that the time on the server matches up with the time actual
    time you would see on a wall clock. Based on your response to Willaim
    Stillwell below, it sounds like it is keeping accurate wall time.

    How often does this happen? Daily, Weekly, etc? Also, when you say the
    server freezes is it completely locked or can you make the caps lock key
    LED go on and off on your keyboard?

    Also, you said that this started occurring lately. Was there some other
    system change that was made about the time you noticed the problems start?