Fail2ban integration issues with Asterisk 1.4.21 under Debian Lenny

Home » Asterisk Users » Fail2ban integration issues with Asterisk 1.4.21 under Debian Lenny
Asterisk Users 10 Comments

I’ve recently had a fairly prolonged SIP registration attack, 18 hours in this case and often with 200 attempts per second, and suspect I’ve had a number of these in the past. The main symptom I noticed previously was, because Asterisk was responding to each registration request it received, it was very quickly using up my 448 kbps upload limit for my home ADSL connection: any further traffic (i.e. anything I did) was then experiencing significant packet loss.

Anyway, I’ve now implemented the “7 steps to better Asterisk security” that I found on the Digium website (deny/permit, alwaysauthreject etc.), and have been looking at fail2ban. However, when I attempted to install it (following the instructions I found on a page about fail2ban with Asterisk), I ran into a couple of issues.

FWIW, I’m using Asterisk 1.4.21.2~dfsg-3+lenny1 on Debian.

First, I tried uncommenting the line in /etc/asterisk/logger.conf, i.e. dateformat=%F %T and verified that the date format in /var/log/asterisk/full had, indeed, changed (after I did an asterisk -rx ‘logger reload’, of course). It had changed: it now started with the year, instead of Aug; however, the parentheses were still there, whereas the instructions seemed to indicate that they’d disappear when this line was used in logger.conf.

At that point, I presumed I’d have to use syslog, after all, as that was given as the only alternative if the date format couldn’t be fixed properly. That wasn’t my preference, but it was still workable.

The second snag I found was that, after I fixed sip.conf to include appropriate deny= and permit= lines and alwaysauthreject=yes, the failed
registration attempts were no longer being logged in /var/log/asterisk/full at all, despite my having the line full => notice,warning,error,debug,verbose in the logfiles section of logger.conf.

It seems that the attack was coming from a region that was denied in sip.conf. This is obviously no problem from the security point of view,
as the attempt would inevitably fail; however, my issue isn’t that the attack might succeed, but rather, that by responding to the attack at all,
Asterisk is grinding my internet connection to a halt. And Asterisk is, indeed, still responding, rather than just ignoring the attempts.

Is there a way to get Asterisk to log failed SIP registration attempts that come from a denied IP address? Or a way to get it to simply ignore such attempts?

I have a feeling that a major Debian release has come out recently, and passed me by. I’m wondering if that contains Asterisk 1.6, and, if so,
whether all these issues (date format as well as logging sip registration attempts from denied IP addresses) might be present in that release. That would certainly present a neat solution – just upgrade my machine!

Any input very welcome.

Oh, if it’s of any interest: I worked out what was going on by using tshark (terminal version of wireshark). In 20 seconds, it captured well
over 7000 packets, rather than the 30 or so I was expecting – and these included about 4000 packets arriving from one host with SIP registration attempts, fully 200 per second.

Best,

Nikhil.

10 thoughts on - Fail2ban integration issues with Asterisk 1.4.21 under Debian Lenny

  • Almost everyone has – read the fine archives, then google for sipvicious
    because that’s what they’re using.

    18-hours eh? This recent one broke my record – just under 3 days. See
    this:

    http://unicorn.drogon.net/hack1.png

    green is inbound – all from one site belonging to a Romanian telephone/ISP

    The real issue here is that most of the hackers I’ve had attack me and my
    clients are using an older version of sipvicious – and the problem with
    that is that it will not go away when you firewall it, so using fail2ban,
    etc. is a waste of time against it – it might protect your asterisk box,
    but it won’t protect your network. In the above case I had, I added
    firewalling into the router where the blue-line output line went to zero
    on Thursday evening (I was playing with it on Friday morning which is why
    there’s output from it then)

    However because the ISP in this case counts all traffic coming in, it
    counted against their monthly allowance – that for me and my clients is
    going to be the killer more than anything else – we can firewall against
    these things, but sipvicious doesn’t care – it just keeps on pumping the
    data towards you and your ISP keeps on incrementing the counters and
    billing you for it (and I’ve yet to find a UK ISP who will put in a block
    at their border against this sort of thing – actually, I know one, but
    they’re too expensive for most).

    At least the older versions of sipvicious behave this way, but when do
    criminals bother to upgrade their software? They don’t seem to care –
    they’ve already stolen resources, so it’s no big issue to them.

    This problems is not going to go away – if anything, I reckon it will get
    worse in the near future. Fail2ban, etc. is not going to protect you from
    broken versions of sipvicious. Anyone who can not firewall their inbound
    SIP port to a known set of IP addresses is inviting attack, and they will
    be attacked. The sipvicious tools make scanning very easy indeed, so you
    will have to take additional measures if you want to save your bandwidth
    and sanity.

    So.. Get a copy of the sipvicious code from http://blog.sipvicious.org/
    (or directly from http://code.google.com/p/sipvicious/ ) and learn how to
    use svcrash.py as that’s the only thing that’s going to ultimately stop a
    long-term attack on your site. For now, anyway.

    Gordon

  • Gordon Henderson wrote:

    You’re wrong when you state: “that’s the only thing that’s going to
    ultimately stop” The fact of the matter is, its quite simple to block
    attackers without relying on anything other than good old fashioned
    systems/network administration.

    should be the Golden Rule on any environment however, in the real world
    there are times when we can’t just do something as simple as that. So
    what’s the next best thing? Good old fashioned administration:

    # tail -n 10 /var/log/asterisk/messages
    [Aug 29 19:13:36] NOTICE[21056] chan_sip.c: Registration from
    ‘”yzlj”‘ failed for ‘69.72.242.170’ – No
    matching peer found
    [Aug 29 19:13:36] NOTICE[21056] chan_sip.c: Registration from
    ‘”zdcu”‘ failed for ‘69.72.242.170’ – No
    matching peer found
    [Aug 29 19:13:36] NOTICE[21056] chan_sip.c: Registration from
    ‘”zdur”‘ failed for ‘69.72.242.170’ – No
    matching peer found
    [Aug 29 19:13:36] NOTICE[21056] chan_sip.c: Registration from
    ‘”zmug”‘ failed for ‘69.72.242.170’ – No
    matching peer found
    [Aug 29 19:13:36] NOTICE[21056] chan_sip.c: Registration from
    ‘”zoej”‘ failed for ‘69.72.242.170’ – No
    matching peer found
    [Aug 29 19:13:36] NOTICE[21056] chan_sip.c: Registration from
    ‘”zpcp”‘ failed for ‘69.72.242.170’ – No
    matching peer found
    [Aug 29 19:13:36] NOTICE[21056] chan_sip.c: Registration from
    ‘”zxnj”‘ failed for ‘69.72.242.170’ – No
    matching peer found
    [Aug 29 19:13:36] NOTICE[21056] chan_sip.c: Registration from
    ‘”zygq”‘ failed for ‘69.72.242.170’ – No
    matching peer found
    [Aug 29 19:13:36] NOTICE[21056] chan_sip.c: Registration from
    ‘”zyjb”‘ failed for ‘69.72.242.170’ – No
    matching peer found
    [Aug 29 19:13:36] NOTICE[21056] chan_sip.c: Registration from
    ‘”zynh”‘ failed for ‘69.72.242.170’ – No
    matching peer found

    How about a little cron script without having to install anything? You
    could run it off the hour:

    rightnow=`date “+%Y-%m-%d %k”`

    grep $rightnow /var/log/asterisk/messages |
    awk ‘/No matching peer/’ | sed’s:”’::g’ |
    uniq | awk ‘{print “iptables -A INPUT -s “$1″ -j DROP”}’| sh

    I’ve done my own IPS/IDS and honeypots on Asterisk and I can tell you
    there are other ways to minimize the attempts and the attacks without
    even running ANYTHING against your machine. I can tell you from
    EXPERIENCE and watching and analyze about 2-3 years worth of VoIP
    attacks, you’d be extremely wrong to think that sipvicious is the only
    tool in someone’s arsenal. Secondly, I’ve seen patient attackers test
    accounts 1 at a time so don’t think for a moment that by solely running
    sipvicious and checking the results, you’re in the clear.

    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-08 | 09:15:03 |
    2010-08-08 | 09:15:03 | 125.71.212.123 | 1 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-23 | 01:28:45 |
    2010-08-23 | 01:28:45 | 82.201.218.31 | 1 |

    mysql> use arkeos

    Database changed
    mysql> select * from bruteforcers where start_date like ‘%2010-08%’;
    +——————————————+————+————+————+———–+—————–+———-+
    | hostid | start_date | start_time |
    stop_date | stop_time | attacker | attempts |
    +——————————————+————+————+————+———–+—————–+———-+
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-02 | 12:28:22 |
    2010-08-02 | 12:58:27 | 88.42.207.98 | 54644 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-04 | 11:46:29 |
    2010-08-04 | 11:48:18 | 93.35.113.170 | 9975 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-04 | 13:08:48 |
    2010-08-04 | 13:09:16 | 210.22.14.113 | 4187 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-06 | 01:51:15 |
    2010-08-06 | 02:26:43 | 187.63.73.3 | 142904 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-08 | 09:15:03 |
    2010-08-08 | 09:15:03 | 125.71.212.123 | 1 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-08 | 15:42:59 |
    2010-08-08 | 17:07:54 | 217.174.169.29 | 108120 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-08 | 18:20:40 |
    2010-08-08 | 18:53:58 | 61.218.212.75 | 79195 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-08 | 19:07:25 |
    2010-08-08 | 19:39:52 | 72.166.143.8 | 50073 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-10 | 19:20:27 |
    2010-08-10 | 19:21:02 | 61.164.41.144 | 2797 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-11 | 10:54:14 |
    2010-08-11 | 12:24:36 | 222.73.93.143 | 128352 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-11 | 16:07:32 |
    2010-08-11 | 16:20:12 | 218.249.33.23 | 2029 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-14 | 02:42:13 |
    2010-08-14 | 02:42:49 | 85.25.20.51 | 3631 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-15 | 12:50:13 |
    2010-08-15 | 12:50:13 | 220.128.103.139 | 1 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-15 | 15:55:48 |
    2010-08-15 | 17:10:28 | 64.15.159.171 | 148217 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-16 | 09:40:00 |
    2010-08-16 | 09:53:25 | 91.121.132.176 | 3039 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-20 | 21:21:48 |
    2010-08-20 | 21:30:44 | 115.146.19.233 | 32018 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-21 | 22:59:17 |
    2010-08-21 | 23:56:59 | 66.246.127.233 | 110170 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-22 | 14:35:34 |
    2010-08-22 | 14:58:35 | 210.17.189.84 | 83977 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-22 | 15:06:26 |
    2010-08-22 | 16:27:03 | 209.172.57.41 | 144106 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-23 | 01:28:45 |
    2010-08-23 | 01:28:45 | 82.201.218.31 | 1 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-23 | 21:54:40 |
    2010-08-23 | 23:14:47 | 64.22.82.135 | 167086 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-24 | 01:23:09 |
    2010-08-24 | 01:23:09 | 62.84.34.18 | 1 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-25 | 07:54:02 |
    2010-08-25 | 07:55:54 | 38.99.168.133 | 16022 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-25 | 17:19:20 |
    2010-08-25 | 17:49:20 | 218.18.9.155 | 88302 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-26 | 18:01:01 |
    2010-08-26 | 19:36:12 | 208.86.252.86 | 166780 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-26 | 19:32:37 |
    2010-08-26 | 21:08:50 | 86.122.211.134 | 113078 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-26 | 21:02:34 |
    2010-08-26 | 21:02:50 | 173.1.78.157 | 2535 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-26 | 21:33:23 |
    2010-08-26 | 23:21:33 | 91.203.134.34 | 167334 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-26 | 22:47:08 |
    2010-08-26 | 23:57:03 | 91.202.26.233 | 76167 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-27 | 09:57:44 |
    2010-08-27 | 10:48:07 | 66.197.145.85 | 228134 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-27 | 13:50:45 |
    2010-08-27 | 13:50:47 | 119.255.6.100 | 315 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-27 | 13:56:36 |
    2010-08-27 | 14:16:48 | 119.145.9.190 | 96698 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-27 | 21:23:11 |
    2010-08-27 | 23:01:52 | 84.23.73.232 | 105549 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-29 | 06:50:41 |
    2010-08-29 | 06:54:53 | 64.199.151.238 | 0 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-29 | 06:50:41 |
    2010-08-29 | 06:54:53 | 64.199.151.238 | 6168 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-29 | 19:19:22 |
    2010-08-29 | 19:36:39 | 69.72.242.170 | 115256 |
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-29 | 19:19:22 |
    2010-08-29 | 19:36:39 | 69.72.242.170 | 6168 |
    +——————————————+————+————+————+———–+—————–+———-+

  • Your script is fine, but I think you are missing the point I made which is
    that early versions of sipvicious are badly broken in that they will
    continue to send their attacks for days after you firewall them out.

    I also posted a very effective iptables script some weeks ago if you care
    to search the archives. It works and is extremely effective in blocking
    these types of attacks – however, it will not stop a broken sipvicious
    from continuing to send data to your server, and that’s the issue I have
    at present.

    Any attacking site is trivial to firewall out once you’ve detected it, but
    firewalling it out is not going to protect you or your customers from the
    incoming data if you have to pay for it, or have a capped Internet
    connection, and the attacking site doesn’t give up, even when it’s
    firewalled.

    A typical setup I might use for a customer in the UK is to set them up
    with an ADSL service with a 15GB/month cap. That’s OK for a small office,
    or a bigger one with a line dedicated to VoIP. The attack I posted about
    earlier used up nearly 15GB of data in 3 days. Fortunately most of it was
    off-peak/weekend when it’s unmetered. I arranged for their router to drop
    the packets, but the ISP still counted them.

    Try it for yourself – get an early copy of SV, point it at one of your
    servers, then firewall the server against your attacking machine.
    svcrack.py will not give up.

    What your script needs to do is not only get the IP address, but also the
    calling port, then launch an svcrash against that site…

    … and hope that the hackers aren’t getting clever and putting svcrack in
    a script that automatically re-starts it… (which I think some of them
    are now)

    Gordon

  • Gordon Henderson wrote:

    Alright, so I’m slightly confused maybe I’m reading this wrong…

    Someone using an older version of sipvicious was blocked and the
    “blocking” of the traffic still carried a load?

    If so then you should have logged into your router and simply sinkholed
    him. There is nothing you can do against a flood whether or not its
    sipvicious or any other program. It’s the “golf ball through the water
    hose” effect.

    Did you try:

    1) sinkholing from your router
    2) Contacting your upstream to inform them of the DoS to see if they’d
    sinkhole it
    3) Contact the UPSTREAM of the attacking host?

    +——————————————+————+————+————+———–+—————–+———-+
    | hostid | start_date | start_time |
    stop_date | stop_time | attacker | attempts |
    +——————————————+————+————+————+———–+—————–+———-+
    | e3d8862a1f1457b8722646dbec79d0f4b7e1b2ab | 2010-08-25 | 07:54:02 |
    2010-08-25 | 07:55:54 | 38.99.168.133 | 16022 |

    8K attempts in a minute. There were times last month I’d see upwards of
    40-60k per minute WHILE I played around with some of these guys in a
    separate Asterisk based honeypot I created. So my confusion: “it will
    not stop a broken sipvicious from continuing to send data to your
    server” Even CURRENT versions of sipvicious won’t stop sending data just
    because you firewalled them out.

    There is a pattern that many don’t see unless your constantly monitoring
    and watching what’s going on with your logs/devices. What I see
    firsthand is, there are “bruteforcers” and there are the “toll
    fraudsters.” Since this is a public list, I care not to discuss findings
    for obvious reasons however, for those interested in that information,
    feel free to send me a “non-free-mail” (meaning no Gmail, no Hotmail,
    etc) message. If I get around to seeing I should share this information,
    I’d gladly do so… Otherwise I won’t disclose anything about honeypots,
    analysis, traffic patterns, etc. Its already surprising I posted
    attacker information on the forum. 😉 I see all sorts of attackers,
    attack vectors, numbers dialed, etc., from many of these attackers.
    You’d be surprised how STUPID some are and how SMART others are.

    As for your comment though, its confusing to me because if you blocked
    them and they’re still overwhelming you, sounds like a) you need more
    bandwidth because you’re on a slow connection (I’m on a DS3) or b)
    server is misconfigured. On Linux tc can be your friend

  • Yes. It’s UDP, they just keep on sending.

    Yes. works fine until they can send faster than the router/incoming line
    can handle the load. With a good VPS host you can trivially max-out a
    typical UK ADSL line.

    Yes.

    My (ADSL) upstream will not block inbound floods like this. They have a
    financial incentive not to – they get paid for the data the allow into
    their network and through to you.

    I only know of one UK broadband ISP that will actively block inbound
    traffic for you and they’re technically superb, but that comes with a
    price which is more than your average small business is wiling to pay.
    None of the others I know and have used will block an inbound flood of
    anything for you.

    My main hosting upstream will only block such attacks when it has a
    detrimental effect on their network (and then they’re very good at it) –
    last time my hosted servers got hit, they soaked up just over 30GB from a
    single VPS site in France in a 12-hour period.

    Yes. No reply. And in the few times I’ve tried, I’ve only ever had a reply
    from Amazon – some 18 hours after the flood started and then it took
    another 12 hours for them to stop it (well documented here in the archives
    by myself and others)

    The reality is that most bulk VPS providers just don’t care, or you’ve got
    to go through layes of their own (semi-automated) protocol to get anywhere
    (cf. Amazon)

    Basically if you have to pay for inbound traffic in any shape or form
    (monthly cap, daily limit, etc.) then you’re fucked when this happens.

    That’s why the author of Sipvicious added svcrash.py to his set of
    scripts.

    Gordon

  • On Tue, Aug 31, 2010 at 8:30 AM, Gordon Henderson
    wrote:

    Amazon did something about it? I don’t remember seeing that, Gordon,
    it’s a new record. The average response has been zero.

    /r

  • Well… I don’t know if they actually did anything )-: However, after
    going through their automated submission widget the attack last time did
    stop… Some 12 hours later. I suspect the poor sod who’d EC2 got hacked
    ran out of credit or something…

    Their whole system is designed as a device to waste the time & effort of
    those trying to submit reports, etc. to them.

    Gordon

  • On Tue, Aug 31, 2010 at 7:09 PM, Gordon Henderson
    wrote:

    This is not the right list for the following comment, but vested
    interests always ruin life. Ego conflicts, often found in true
    open-source projects, are far less damaging. Email spammers got away
    for the longest time with doing that because the providers didn’t want
    to throw out paying customers. Eventually, it because clear that they
    would be forced to do something. I believe eventually, Amazon will be
    put in that position. Until then, they have done very little, and we
    have stopped using their cloud services as much as possible.

    /r

  • Hi guys,

    Interesting discussion – I learnt quite a bit. Thanks.

    That said, no one’s yet answered my two original questions. Anyone know?
    To repeat:

    1. When I used the line “dateformat=%F %T” in the general section of
    logger.conf, the format in /var/log/asterisk/full did change, but the
    round brackets around the date remained; the fail2ban/asterisk
    instructions I was following indicated that they should disappear when I
    do this. Is this an asterisk version issue (I’m using 1.4.21 – would 1.6
    behave in the way described?), a Debian issue (seems unlikely), or
    something else? Is there some other way to get the round brackets to
    disappear? This is necessary to get fail2ban to read that file;
    otherwise, I’ll have to log all asterisk NOTICEs through syslog.

    2. With alwaysauthreject=yes and using deny= and permit= in sip.conf,
    attempts from denied IP addresses to register an extension are responded
    to (denying them, obviously), but not logged as a NOTICE (or anything
    else, as far as I can tell). Is there a way to enable this logging – or,
    alternatively, to get asterisk to simply ignore these requests rather than
    responding?

    I fully appreciate the frustration others feel re ending up paying for
    other people’s attacks as part of their download limit, or even maxing out
    because of this. I’m lucky in that I haven’t reached that volume yet –
    I’m with Zen Internet, and have a 50GB monthly limit, which I’m not using
    anywhere near all of. (that said, I did use 30GB last month, and suspect
    that the lion’s share of that was from attacks, SIP reg or otherwise; this
    could certainly be an issue in the future.)

    Instead, my issue was with maxing out my *upload* bandwidth limit
    (currently only 448kbps), and hence having my whole connection screeching
    to a halt, with massive packet loss to other applications. At that point,
    not even (a sane amount of) money helps, as you can’t buy a higher upload
    rate (aside from regrading to ADSL2+, which I’m looking into now).

    Thanks in advance,

    Nikhil.