13.22.0 – HTTP Session Count Exceeded 100 Sessions – Instance Unusable
Hi all
I’m running an Asterisk 13.22.0 instance on CentOS 7 – I7-8700 12 core HT
with 16GB of RAM.
The server maintains a total active call count of approx 285 calls with 440
channels at any one time. The totals never go below 200 calls concurrently active.
For the kernel, top reports load levels at around 5.0 to 6.0 constantly.
I’m having issues where at random intervals, the CLI is filled with thousands of these lines
[Jul 1 14:32:24] WARNING[11530]: http.c:1942 httpd_helper_thread: HTTP
session count exceeded 100 sessions.
This appears to be some form of race condition, the Asterisk instance goes completely unresponsive, all RTP stops and all calls get hung up the moment these appear on the CLI. All registered SIP phones then deregister off the server withing one to two minutes.
At the time this takes place, kernel load spikes to about 9.0 and then falls off to under 1 while asterisk endlessly outputs the above warning in the Asterisk CLI.
You can still log into the CLI but no command is executed, you also cannot get the Asterisk to stop with HUP / kill -1 and I have to kill -9 the Asterisk PID at the end of the day.
I’m also, just prior the above taking place, getting tens of thousands of these warnings in the CLI:
[Jul 1 14:30:39] WARNING[16819][C-000004bf]: taskprocessor.c:895
taskprocessor_push: The ‘stasis-core-control’ task processor queue reached
500 scheduled tasks again.
I’m using AMI at several hundred lines of traffic per second to control the Asterisk instance, spawn new calls, get queue and agent statuses, etc. –
standard AMI stuff.
Where can I start to look for how to fix this “session count exceeded 100
sessions” error?
Sometimes, it doesn’t come up for weeks (loads remain much the same) and then on one day it will happen four to six times in quick succession, load broadly the same.
Any ideas?
Thanks!
Stefan
—
6 thoughts on - 13.22.0 – HTTP Session Count Exceeded 100 Sessions – Instance Unusable
This isn’t really a race condition. The message occurs when the limit for simultaneous HTTP sessions is exceeded. You haven’t explained what the HTTP
server in Asterisk is being used for, so I’d start with investigating that. Is ARI being used? WebRTC? If you do a packet capture on it do you see a flood of incoming HTTP connection attempts?
Hi Joshua
HTTP is used on in our setup on
127.0.0.1/mxml?
to send commands to the server, such as
http://127.0.0.1/mxml?action=login&username=myuser&secret=thesecret
to log in and then
http://127.0.0.1/mxml?ActionID=123&Action=BlindTransfer&Channel=Channel&Context=local&Exten=123&Priority=1
etc. to control transfers, for example.
ARI is not being used.
WebRTC is not being used.
I have in the meantime modified httpd.conf and halved the session_inactivity from 30 000 to 20 000 and session_keep_alive from 15 000 to 10 000, and increased the sessionlimit to 1000 (instead of 100) to see if shorter sessions expiring more frequently and just plain more available slots for sessions will help?
So far so good, done about 55 000 new calls that way the past 32 minutes.
Or am I barking up the wrong tree?
Thanks!
—
Stefan
—
It’s possible it will help – but that still doesn’t explain what exactly happened. Does your code have a back off strategy if HTTP requests fail? If not and they have an immediate retry with loop, then that may have effectively DDoSed things. It would also be good to explain why 100
connections were in use – did old connections not get closed? Was your load sufficient to just meet that amount?
Hi Joshua
No back-off, but I am caching the last 5000 result and and first hitting the cache to see if a recent command already provided the information I’m seeking for a particular request.
From: Joshua C. Colp
Sent: Wednesday, 01 July 2020 15:47
To: viljoens@verishare.co.za; Asterisk Users Mailing List – Non-Commercial Discussion
Subject: Re: [asterisk-users] 13.22.0 – HTTP session count exceeded 100 sessions – instance unusable
Hi Joshua
HTTP is used on in our setup on
http://127.0.0.1/mxml?
to send commands to the server, such as
http://127.0.0.1/mxml?action=login&username=myuser&secret=thesecret
to log in and then
http://127.0.0.1/mxml?ActionID=123&Action=BlindTransfer&Channel=Channel&Context=local&Exten=123&Priority=1
etc. to control transfers, for example.
ARI is not being used.
WebRTC is not being used.
I have in the meantime modified httpd.conf and halved the session_inactivity from 30 000 to 20 000 and session_keep_alive from 15 000 to 10 000, and increased the sessionlimit to 1000 (instead of 100) to see if shorter sessions expiring more frequently and just plain more available slots for sessions will help?
So far so good, done about 55 000 new calls that way the past 32 minutes.
Or am I barking up the wrong tree?
It’s possible it will help – but that still doesn’t explain what exactly happened. Does your code have a back off strategy if HTTP requests fail? If not and they have an immediate retry with loop, then that may have effectively DDoSed things. It would also be good to explain why 100 connections were in use – did old connections not get closed? Was your load sufficient to just meet that amount?
—
Joshua C. Colp Asterisk Technical Lead Sangoma Technologies Check us out at http://www.sangoma.com and http://www.asterisk.org
—
Hi Joshua
No back-off, but I am caching the last 5000 results and and first hitting the cache to see if a recent command already provided the information I’m seeking for a particular request.
I’ll see if I can do some simulation and see if I’m effectively DDOSing the local HTTP interface.
I’ll have to see if I maybe have a resource leak in my code that makes the HTTP request, which is set to “Connection: close” already (and not “Connection: keep-alive” at HTTP request header level) hang around and keep the HTTP connection open to Asterisk. Does the HTTP implementation in Asterisk heed this “Connection:” HTTP header?
—
Stefan
—
The code is there, I’ve never experimented with or used it.