11 thoughts on - Strange Issue: Asterisk Deleted

• Am 26.11.2014 11:37, schrieb Antoine Megalla:

  Hi,

  you write “Also if it was really deleted ..” – did you looked at it via
  “ls /usr/sbin/asterisk”?

  You compiled asterisk (make / make install) as root I think. Perhaps access rights are not set properly? root is owner but you try to start the daemon as “normal” user?

  You write “the process is killed”. Where do you now? Did you get a message on your terminal? Did you take a look at /var/log/syslog?

  Best regards
  -Thorsten-

• If you suspect that something is being run from the asterisk process or one of its children, run it under ‘strace -f’ and look for hints (e.g.:
  ‘unlink’) in the generated log.

• Hi,

  I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there.

  I know that the process is killed because when I start asterisk using the command asterisk -vvvvc it starts and then it exits and the word killed is wrote on the console.

  Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too.

  Now I used the strace command on asterisk and I can clearly see at the end of the strace the line : killed by SIGKILL
  This means that something or someone is actually and purposely killing asterisk but I do not know what or who is doing that also I know that I am the only user on the system.

  Again any indicators to solve this very weird issue are welcomed.

  Regards, Antoine Megalla

  Sent from my iPhone

• I don’t know if there’s any way to see where the signal comes from. But I think it would have to be another process. Is this a hosted machine? Could it be that your hosting provider doesn’t allow asterisk? This would be a good way to enforce that rule. Otherwise, it could be a root kit or a virus.

  Or it could be that you (or someone else) wanted to make sure asterisk wasn’t running at some point and left “while true; do killall -9
  asterisk; done” running in a shell, and forgot about it.

  You can list all the processes with the command “ps -ef”

  And to see if anyone else (or yourself) is logged in, run “w”. That will show every individual session and where they’re connected from.

• Interesting problem, I’m quite curious what the cause is.

  Are you 100% sure that the asterisk your are running is in /usr/sbin? Try ‘which asterisk’ to see what your shell is running and/or start asterisk with a full path as /usr/sbin/asterisk -vvvvc.

  You could also try renaming the binary to find out if indeed something kills Asterisk by name.

  There’s a tool called SystemTap which could give you information which process sent the SIGKILL:
  https://sourceware.org/systemtap/
  http://www.percona.com/blog/2014/07/18/systemtap-solves-phantom-mysqld-sigterm-sigkill-issue/

• Did you take a look at /var/log/syslog?

  Am 26.11.2014 21:08, schrieb Antoine Megalla:

• Yes I did, and there is nothing about asterisk in the /var/log folder

  I am starting to think that the server on compromised.

  Sent from my iPhone

• It sounds as though your server might have been compromised.

  Get another machine of the same bit architecture and perform a fresh install of exactly the same OS as your Asterisk box on that. Install busybox too
  (it’s usually there anyway, as it’s required for building the initial RAMdisks used by most distros for booting). Using a USB stick (preferrably one that can be set read-only), copy at least the `ls`, `ps`, `netstat`, `w`,
  `lsattr`, `md5sum`, `cat`, `diff` and `busybox` binaries over (to somewhere that isn’t /usr/bin/). Use both the existing installed and the newly-copied md5sum and diff to check each system binary against the known-good ones. You can use busybox to replicate commands you haven’t copied (but note that busybox versions are rather cut-down as compared to the GNU tools you know and love. Come to think of it, they’re cut-down as compared to the BSD tools everyone replaces with GNU versions once they have a C compiler up and running).

  Compare /etc/inittab between the two machines.

  Many rootkits mess with ext[2-4]fs attributes, presumably to stop you overwriting their overwritten system binaries; so use a known good lsattr to check the attributes of everything in /bin/, /sbin/, /usr/bin/ and /usr/sbin/
  — watch out for anything set immutable.

  Getting rid of the compromise fortunately is reasonably easy, especially if your /home folder is on its own partition. Just ignore that partition during reinstallation, edit your /etc/fstab afterwards and reboot — your original
  /home will be preserved intact. If not, use systemrescuecd or something similar to boot a known-good system. Use mv to rename /home to a new name. Shrink a disk partition and create a new small partition. Use that for your
  /home during the reinstall. Then again edit /etc/fstab, unmount /home, mv your old /home back to /home and reboot.

• Hi

  Thank you for your support. The server is actually compromised, I discovered that after making a deep trace using the audit daemon and looking for the kill signal (SIGKILL) that terminates asterisk. I discovered that there is an executable with a random name in the /boot folder that is killing and deleting asterisk !!!

  This executable is launched by a service in /etc/rc.d/ with the same random name. When I stopped this service, a new service was created with another different random name and it too is killing and deleting asterisk. This was the evidence i needed to be convinced that the server has a virus and is compromised.

  The good thing is that this is a fresh install and hence there are no sensitive data or a lot of work done on it so i will reinstall the OS and start over. The bad thing is that I spent more than 4 days trying to understand what was going on.

  Again, thank you for your support.

  Regards, Antoine Megalla

  Sent from my iPhone

• Question remains, how was it compromised?
  In the original install ?
  A “fresh” install perhaps from another source?

  Best you determine HOW before spending more time going down another rabbit hole!

  John Novack

  Antoine Megalla wrote:
  

• —– Original Message —