16 thoughts on - IOPS Required By Asterisk For Call Recording

• You’re assuming that asterisk will perform an fsync() after each write. If asterisk writes without an fsync after each write, then the OS will schedule writes intelligently based on RAM/disk IO available rather than scheduling each one as a separate write.

  Looking at the code for ast_writestream() there doesn’t appear to be an fsync() type call after each write, but someone more familiar with the internals of Asterisk would be better able to verify that.

• If you are running on Linux, don’t forget that Linux’s default behaviour is to cache all disk writes until the machine is rebooted or the RAM is needed for something else, and service read operations from the cache. In fact, it’s entirely possible for a temporary file to be written, read and deleted without ever going anywhere near a molecule of oxide.

  Solaris has the opposite default caching strategy — it assumes the worst about filesystem integrity, and write operations block until decaching and verifying have finished.

• Thanks for response. How do I derive the requirement? I need to size IO system to record multiple calls concurrently. I ran test with following configuration Quad Core Xeon with 4GB RAM
  250GB SATA disk (No RAID)
  Linux (CentOS 5.9)
  Asterisk 1.8.20

  I failed to record more than 80 calls.

  If I run test with simple IVR, I achieved 400+ calls with same server. So write seem to be an issue. Is there any way to tune / optimize / configure for better write performance?

  I am not sure if I need to post this query on developers list? Please guide…

  Regards Amit Patkar

• Thanks for response. How do I derive the requirement? I need to size IO system to record multiple calls concurrently. I ran test with following configuration Quad Core Xeon with 4GB RAM
  250GB SATA disk (No RAID)
  Linux (CentOS 5.9)
  Asterisk 1.8.20

  I failed to record more than 80 calls.

  If I run test with simple IVR, I achieved 400+ calls with same server. So write seem to be an issue. Is there any way to tune / optimize / configure for better write performance?

  I am not sure if I need to post this query on developers list? Please guide…

  Regards Amit Patkar

  Message: 1
  Date: Fri, 24 Jan 2014 11:46:39 -0400
  From: Mike
  To: Asterisk Users Mailing List – Non-Commercial Discussion
  
  Subject: Re: [asterisk-users] IOPS required by Asterisk for Call Recording Message-ID:<52E28ADF.8020409@gmail.com>
  Content-Type: text/plain; charset=”iso-8859-1″

• I suspect this might be your problem:

  Perhaps consider recording to a ramdisk first, then periodically write out completed files to HDD at your leisure (e.g. during slack periods)?

  Or, given the relatively low cost of 250GB SSDs these days, swap out the spinning disc for an SSD.

  Kind regards,

  Chris

• If you care enough to record the calls, you should care enough to get some fast and redundant storage. SSDs would be best, 15K SAS drives second choice. Even a good RAID10 of SATA drives would help a lot.

  A RAID card with battery backed cache would be helpful as well.

• I’m not aware of 400+ calls being recorded succesfully on an Asterisk box. If there is it probably has tons of RAM, enterprise grade SSDs or
  15K RPM FC/SAS drives in a battery backed RAID setup or a fast SAN
  saving the calls in native format (via a tmpfs) with the transcoding probably done on another box.

  Add more RAM and much much more if you are going to use tmpfs.

  Well you get the performance you pay for. CentOS comes with various utilities that allow you to analyze that.

  Imo CentOS 6.5 (x86_64) has better performance.

  In 9 months Asterisk 1.8 will only get security fixes. I would use Asterisk 11. It will get regular bug fixes for a much longer time.

  Hardly surprising.

  A simple IVR is not the same as call recording. The comparison makes as much sense as saying that copying to /dev/null is faster than to a disk.

  No, this is a user question and does not belong on the developer list.

  Since you seem to work for a call center business perhaps investigate a commercial solution like Orecx (I have no affiliation):

  http://www.orecx.com/OrecX-for-Asterisk.php

  HTH, Patrick

• I’d suggest testing your system while monitoring with top and iotop
  (which should be a “yum install” away).

  That should show you your bottlenecks.

  It looks to me like Asterisk doesn’t do compression until the call is ended, so recording to a compressed format would actually increase IO
  load (write, read and compress, write compressed data).

• Can you get a reading of the total number of I/Os during your test? Peak IOPS?
  That might tell you very quickly about the storage pattern that Asterisk uses.

  Can you configure a RAM drive to see if disk is really the bottleneck. May need to add some more RAM memory to your configuration.

  What is your network capacity? Usually one can write faster than the network can deliver – just to make sure that you are chasing the right bottleneck.

  What happens at 80 calls to tell you that you have run out of IOPS?

  Sorry for more questions than answers.

  Ron

• Dovetailing on this question, I’ll add one as well:

  Are you recording using MixMonitor, or Monitor?

  Depending on your answer to the “what happens at 80 calls”, you may get better results with MixMonitor over Monitor. MixMonitor offloads the recording of the media to a separate thread; Monitor attempts to record the audio on the thread servicing the channel(s).

  Matt

• Hi,

  MixMonitor takes a parameter of a system command to run when the recording finishes. Like Chris said, you can write to ramdisk, and run a script that will move the file into final position only when the call has done recording

  Here we use:
  Set(recordFile=${UNIQUEID}_${NUMBER}.gsm);

  Set(recordPath=/var/log/asterisk/recordings/${CALLERID(dnid)}/${STRFTIME(${EPOCH},GMT+0,%F)});

  MixMonitor(/ramdrive/${recordFile},,/usr/local/bin/mixmon “${recordFile}”
  “${recordPath}”);
  SIPAddHeader(X-REC-FILE:
  ${recordPath}/${recordFile});

  and /usr/local/bin/mixmon will move the file to $recordPath and whatever else needs done on that file…

• Thanks Ron. I will try to get these readings. About RAM disk, I will study on how to create RAM disk and conduct this test again. There is no bottleneck on network.

  After 80 calls, I see call drops, delay in responding, time out, re-transmission of SIP messages. If load is reduced, it settles again to normal.

  *Thanks & Regards,*
  Amit Patkar

• To create a ramdisk under Linux, assuming you have enough ram –

  # mkdir /ramdisk
  # mount -t tmpfs tmpfs /ramdisk

• I am using Monitor function. Let me try with MixMonitor and update. After 80 calls, I see retransmission of SIP messages, unanswered calls..

  *Thanks & Regards,*
  Amit Patkar

• Will check and update.

  *Thanks & Regards,*
  Amit Patkar

• I am surprised about the network. It should go before the disk if you have a lot of short transactions. There is a high percentage of overhead on streams of short messages. Make sure that you check at each point where messages are passing.

  Have you done any mathematical modeling of the disk and network traffic?

  Try changing your RAM to see if raising it (or lowering it if that is easier) affects the problem.

  What is your CPU utilization like at 80 calls?

  How many open files do you have at 80 calls? is this near the limit?

  Can you adjust the quality of the recordings to reduce the bits stored for each second of audio?
  What happens when you do this?

  Ron