Load Issues Using AGI
Hi all,
we have just upgraded from Asterisk 11 to Asterisk 16. After porting all the config to 16 we figured out some major load problems.
the majority running of our Asterisk instances is still having Asterisk 11
so we can compare load handling on both versions. On the same hardware configuration we see load differences that Asterisk 16
takes four times the load as Asterisk 11 (on 11 we see load 0.5, on 16 we see something around 2).
Our asterisk is only handling Calls, so there are no Subscription no Registration etc.
After some testing we figured out if we eliminate AGI Apps from Dialplan we reduce the load significantly. At the moment we have 6 AGI calls for one single call. If we eliminate 3 of them we reduce the load by half. I also tried to have the AGI replaced by some fake AGI which returns immediately to make sure it is not related to long running AGI Scripts but this made no difference. I tried to tweaked a bit manipulating the thread limitations in stasis.conf but it had no significant effect on load. Also I tried to “decline” all non AGI messages in stasis.conf. It seems it has very little effect.
Does anyone have similar issues or a solution?
Is there anyone who calls AGI several times during call establishment?
any hin and help would be very much appreciated!
I am happy to share more config and information if it helps to find a solution.
6 thoughts on - Load Issues Using AGI
Is this AGI or FastAGI? If AGI then moving to FastAGI will likely improve things. This is because when executing an AGI script the kernel has to fork the process which depending on memory consumption can take some time and drive up load. FreePBX ran into this problem[1] and it additionally caused audio issues in certain cases for them.
[1] https://www.freepbx.org/performance-improvements-in-freepbx/
—
Joshua C. Colp Digium – A Sangoma Company | Senior Software Developer
445 Jan Davis Drive NW – Huntsville, AL 35806 – US
Check us out at: http://www.digium.com & http://www.asterisk.org
—
Hello;
What programming language are you using? If you are using Perl, then I can suggest this.
(1) You are going to have to profile your scripts. This is a must. Without profiling, you don’t know what the problem is. In my opinion, Devel::NYTProf is the king of that hill. It could be as simple as optimizing a single subroutine. Then you will know exactly what the problem is.
(2) See which version of Perl are you running and see if upgrading it solves your problems. The easiest way is to download the newest Perl they support from ActiveState.com. It creates a completely independent installation in /opt which will not interfere with your system Perl. You can also compile and install the newest Perl from source completely separate from your system Perl. That’s what I do. I have a couple of scripts to automate that process. If you first get a list of your installed modules using ‘perlmod’, you can clean up the output a bit and pipe that to cpanm to make sure that you have all the modules you need.
If you are using PHP, then I’m sure that the above still applies, but PHP is not my area of expertise.
Regards;
John V.
From: asterisk-users On Behalf Of Jöran Vinzens Sent: Friday, September 20, 2019 12:47 PM
To: Asterisk Users Mailing List – Non-Commercial Discussion
Subject: [asterisk-users] Load issues using AGI
Hi all,
we have just upgraded from Asterisk 11 to Asterisk 16.
After porting all the config to 16 we figured out some major load problems.
the majority running of our Asterisk instances is still having Asterisk 11 so we can compare load handling on both versions.
On the same hardware configuration we see load differences that Asterisk 16 takes four times the load as Asterisk 11 (on 11 we see load 0.5, on 16 we see something around 2).
Our asterisk is only handling Calls, so there are no Subscription no Registration etc.
After some testing we figured out if we eliminate AGI Apps from Dialplan we reduce the load significantly.
At the moment we have 6 AGI calls for one single call. If we eliminate 3 of them we reduce the load by half.
I also tried to have the AGI replaced by some fake AGI which returns immediately to make sure it is not related to long running AGI Scripts but this made no difference.
I tried to tweaked a bit manipulating the thread limitations in stasis.conf but it had no significant effect on load.
Also I tried to “decline” all non AGI messages in stasis.conf. It seems it has very little effect.
Does anyone have similar issues or a solution?
Is there anyone who calls AGI several times during call establishment?
any hin and help would be very much appreciated!
I am happy to share more config and information if it helps to find a solution.
One other thing. I use a package called AGISpeedy. Its available for both Perl and PHP and I’ve used it for years without any problems. Its supposedly an order of magnitude faster than regular FastAGI scripts. The only downside is that it hasn’t been maintained a while, but the package is solid and really doesn’t need much maintaining. I’ll probably write the author and see if I can keep it updated for him.
Regards;
John
From: asterisk-users On Behalf Of Tech Support Sent: Friday, September 20, 2019 1:37 PM
To: ‘Asterisk Users Mailing List – Non-Commercial Discussion’
Subject: Re: [asterisk-users] Load issues using AGI
Hello;
What programming language are you using? If you are using Perl, then I can suggest this.
(1) You are going to have to profile your scripts. This is a must. Without profiling, you don’t know what the problem is. In my opinion, Devel::NYTProf is the king of that hill. It could be as simple as optimizing a single subroutine. Then you will know exactly what the problem is.
(2) See which version of Perl are you running and see if upgrading it solves your problems. The easiest way is to download the newest Perl they support from ActiveState.com. It creates a completely independent installation in /opt which will not interfere with your system Perl. You can also compile and install the newest Perl from source completely separate from your system Perl. That’s what I do. I have a couple of scripts to automate that process. If you first get a list of your installed modules using ‘perlmod’, you can clean up the output a bit and pipe that to cpanm to make sure that you have all the modules you need.
If you are using PHP, then I’m sure that the above still applies, but PHP is not my area of expertise.
Regards;
John V.
From: asterisk-users > On Behalf Of Jöran Vinzens Sent: Friday, September 20, 2019 12:47 PM >
To: Asterisk Users Mailing List – Non-Commercial Discussion
Subject: [asterisk-users] Load issues using AGI
Hi all,
we have just upgraded from Asterisk 11 to Asterisk 16.
After porting all the config to 16 we figured out some major load problems.
the majority running of our Asterisk instances is still having Asterisk 11 so we can compare load handling on both versions.
On the same hardware configuration we see load differences that Asterisk 16 takes four times the load as Asterisk 11 (on 11 we see load 0.5, on 16 we see something around 2).
Our asterisk is only handling Calls, so there are no Subscription no Registration etc.
After some testing we figured out if we eliminate AGI Apps from Dialplan we reduce the load significantly.
At the moment we have 6 AGI calls for one single call. If we eliminate 3 of them we reduce the load by half.
I also tried to have the AGI replaced by some fake AGI which returns immediately to make sure it is not related to long running AGI Scripts but this made no difference.
I tried to tweaked a bit manipulating the thread limitations in stasis.conf but it had no significant effect on load.
Also I tried to “decline” all non AGI messages in stasis.conf. It seems it has very little effect.
Does anyone have similar issues or a solution?
Is there anyone who calls AGI several times during call establishment?
any hin and help would be very much appreciated!
I am happy to share more config and information if it helps to find a solution.
Hi,
@josh, We are using AGI. It is a very simple perl script. If we need to move to FastAGI we most likely would port it to ARI instead. But I let you know.
@john, we using Perl. To see if it is a problem with the perl i had put an
“exit 0” just at the first lines so there is no logic done at the AGI. It’s only the start up and return from AGI what produces the most of the load. Nevertheless, we will try what you just posted.
thanks for your help. I will keep you updated!
BR
Jöran
Good point.
I will try that. We have just started the work to port our perl AGI to Java fastAGI. We will eliminate some of the AGI and see how performance improves.
In terms of the perl speed, i will try your suggestion.
Thanks Jöran
Tony Mountifield schrieb am Di., 24. Sep. 2019, 11:23:
Hello Jöran, how are you?
your issues got me very curious and concerned, because I will be working on something similar of what you are doing soon.
If you don’t mind, please keep us updated with our discoveries, especially that one related to the last suggestion of Tony.
PS: are you considering go to Fosdem next February?
Thanks.
Regards,
Marcelo H. Terres
https://www.mundoopensource.com.br https://twitter.com/mhterres https://linkedin.com/in/marceloterres