Any Thoughts On Asterisk 16.17.0 Outputting FRACK Refcount Related Messages
We have an extremely busy/large customer. They run fine most of the time, but periodically asterisk will output FRACK refcount related messages. It doesn’t seem to be related to the volume, because it’s not breaking during their peak times.
When this happens, the system becomes unstable and they have to restart to get things resolved. To give an idea of the instability, we have seen INVITE/Trying responses in SIP messaging logs. We tell Asterisk to answer via AMI, but Asterisk never sends the OK (even 24 seconds later it hasn’t sent). Eventually the other send CANCEL of the call.
We’ve now captured 4 different days where something like the following occurs.
1) Is there a good way to tell if this may be fixed in Asterisk 16.20.0 (short of upgrading)?
2) Would this be something I should submit as an asterisk issue? Unfortunately, site is so busy capturing the debug will be very difficult (if not impossible) due to amount of data.
[09/23 14:43:45.095] ERROR[34763][C-00001d7f] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.095] ERROR[34763][C-00001d7f] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.095] ERROR[29830] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.095] ERROR[29832][C-00001920] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.095] ERROR[29830] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.095] ERROR[29832][C-00001920] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.095] ERROR[32973] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.095] ERROR[32973] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.095] ERROR[3248] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.095] ERROR[3248] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.095] WARNING[2798][C-00000093] channel.c: Exceptionally long queue length queuing to CBAnn/IS__8b9c6719-ca29-4c1b-ac87-75e8c6fe7074-00000062;1
[09/23 14:43:45.095] ERROR[12979] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.095] ERROR[12979] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.095] WARNING[21123][C-00001157] channel.c: Exceptionally long queue length queuing to CBAnn/IS__a652155f-b1fb-4c31-83e5-09ffa2107979-000010de;1
[09/23 14:43:45.096] ERROR[29830] : Got 8 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_bridge_channel_queue_frame+0x61) [0x5637ebf1cfe1]
# 3: /usr/lib/asterisk/modules/bridge_softmix.so(+0x40af) [0x7fc15362f0af]
# 4: /usr/lib/asterisk/modules/bridge_softmix.so(+0x560a) [0x7fc15363060a]
# 5: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.097] WARNING[36475][C-00000172] channel.c: Exceptionally long voice queue length queuing to CBAnn/IS__64bc075a-1ba4-4ad8-ba48-f0aea6ca6bab-00001ed3;1
[09/23 14:43:45.098] ERROR[12979] : Got 8 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_bridge_channel_queue_frame+0x61) [0x5637ebf1cfe1]
# 3: /usr/lib/asterisk/modules/bridge_softmix.so(+0x40af) [0x7fc15362f0af]
# 4: /usr/lib/asterisk/modules/bridge_softmix.so(+0x560a) [0x7fc15363060a]
# 5: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.098] ERROR[34763][C-00001d7f] : Got 19 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(+0xa9517) [0x5637ebf2e517]
# 3: /usr/sbin/asterisk(ast_unreal_write_stream+0x15a) [0x5637ebf6ce1a]
# 4: /usr/sbin/asterisk(ast_write_stream+0xd2e) [0x5637ebf4551e]
# 5: /usr/sbin/asterisk(bridge_channel_internal_join+0xe6d) [0x5637ebf2238d]
# 6: /usr/sbin/asterisk(ast_bridge_join+0x3d5) [0x5637ebf05ce5]
# 7: /usr/lib/asterisk/modules/app_confbridge.so(+0xff35) [0x7fc05c258f35]
# 8: /usr/sbin/asterisk(pbx_exec+0xd2) [0x5637ebfdcf92]
# 9: /usr/lib/asterisk/modules/res_agi.so(+0xbea5) [0x7fc059269ea5]
#10: /usr/lib/asterisk/modules/res_agi.so(+0xd29b) [0x7fc05926b29b]
#11: /usr/lib/asterisk/modules/res_agi.so(+0xf7b0) [0x7fc05926d7b0]
#12: /usr/sbin/asterisk(pbx_exec+0xd2) [0x5637ebfdcf92]
#13: /usr/sbin/asterisk(+0x149062) [0x5637ebfce062]
#14: /usr/sbin/asterisk(+0x14d143) [0x5637ebfd2143]
#15: /usr/sbin/asterisk(+0x14e9e9) [0x5637ebfd39e9]
#15: /usr/sbin/asterisk(+0x14e9e9) [0x5637ebfd39e9]
#16: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
#17: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
#18: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.098] ERROR[29832][C-00001920] : Got 10 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(+0xa9517) [0x5637ebf2e517]
# 3: /usr/sbin/asterisk(ast_unreal_write+0x15a) [0x5637ebf6ca0a]
# 4: /usr/sbin/asterisk(ast_write_stream+0xd2e) [0x5637ebf4551e]
# 5: /usr/sbin/asterisk(bridge_channel_internal_join+0xe6d) [0x5637ebf2238d]
# 6: /usr/sbin/asterisk(+0x806f3) [0x5637ebf056f3]
# 7: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 9: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.099] ERROR[3248] : Got 8 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_bridge_channel_queue_frame+0x61) [0x5637ebf1cfe1]
# 3: /usr/lib/asterisk/modules/bridge_softmix.so(+0x40af) [0x7fc15362f0af]
# 4: /usr/lib/asterisk/modules/bridge_softmix.so(+0x560a) [0x7fc15363060a]
# 5: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.099] ERROR[32973] : Got 8 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_bridge_channel_queue_frame+0x61) [0x5637ebf1cfe1]
# 3: /usr/lib/asterisk/modules/bridge_softmix.so(+0x40af) [0x7fc15362f0af]
# 4: /usr/lib/asterisk/modules/bridge_softmix.so(+0x560a) [0x7fc15363060a]
# 5: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.207] ERROR[36630] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.207] ERROR[36630] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.207] ERROR[34172][C-00001cfb] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.207] ERROR[34172][C-00001cfb] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.207] ERROR[28738] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.207] ERROR[28738] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.207] ERROR[36630] : Got 8 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_bridge_channel_queue_frame+0x61) [0x5637ebf1cfe1]
# 3: /usr/lib/asterisk/modules/bridge_softmix.so(+0x40af) [0x7fc15362f0af]
# 4: /usr/lib/asterisk/modules/bridge_softmix.so(+0x560a) [0x7fc15363060a]
# 5: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.208] WARNING[2798][C-00000093] channel.c: Exceptionally long queue length queuing to CBAnn/IS__8b9c6719-ca29-4c1b-ac87-75e8c6fe7074-00000062;1
[09/23 14:43:45.209] ERROR[34172][C-00001cfb] : Got 14 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_readframe+0x45) [0x5637ebf8c675]
# 3: /usr/lib/asterisk/modules/res_musiconhold.so(+0x4ca0) [0x7fc153b07ca0]
# 4: /usr/sbin/asterisk(+0xb5d6c) [0x5637ebf3ad6c]
# 5: /usr/sbin/asterisk(+0xc7a47) [0x5637ebf4ca47]
# 6: /usr/lib/asterisk/modules/res_agi.so(+0xeb9c) [0x7fc05926cb9c]
# 7: /usr/sbin/asterisk(pbx_exec+0xd2) [0x5637ebfdcf92]
# 8: /usr/sbin/asterisk(+0x149062) [0x5637ebfce062]
# 9: /usr/sbin/asterisk(+0x14d143) [0x5637ebfd2143]
#10: /usr/sbin/asterisk(+0x14e82b) [0x5637ebfd382b]
#11: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
#12: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
#13: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.210] ERROR[28738] : Got 8 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_bridge_channel_queue_frame+0x61) [0x5637ebf1cfe1]
# 3: /usr/lib/asterisk/modules/bridge_softmix.so(+0x40af) [0x7fc15362f0af]
# 4: /usr/lib/asterisk/modules/bridge_softmix.so(+0x560a) [0x7fc15363060a]
# 5: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
Dan
This email is intended only for the use of the party to which it is addressed and may contain information that is privileged, confidential, or protected by law. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this email or its contents is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer.
4 thoughts on - Any Thoughts On Asterisk 16.17.0 Outputting FRACK Refcount Related Messages
You could examine the changes between the two and see if any issues seem relevant.
Without a backtrace, or debug logs it’d be really hard to look at. The message just means that an object was referenced a lot. Why that is – who knows. It could be that the system can’t keep up for some reason, or there’s an off-nominal path, or something is deadlocked, or something else.
VGhhbmsgeW91IEpvc2h1YS4NCg0KSSBlbmFibGVkIGJldHRlciBkZWJ1Z2dpbmcgYW5kIHdl4oCZ
cmUgcnVubmluZyB0ZXN0cyBpbmhvdXNlIHRyeWluZyB0byByZXBsaWNhdGUgaXNzdWUuICBUaGlz IG1vcm5pbmcsIEkgc2VlIHRoZSBmb2xsb3dpbmcgaGFwcGVuZWQgZHVyaW5nIG91ciBpbnRlcm5h bCBsb2FkIHRlc3QuDQoNCldvdWxkIHRoaXMgaW5kaWNhdGUgc3lzdGVtIGlzIHRvbyBidXN5IG9y IGEgcG90ZW50aWFsIGlzc3VlPw0KDQoNClswOS8yOCAwOToyMzoxMC44MDVdIEVSUk9SWzMwODNd IGZyYW1lLmM6IEV4Y2Vzc2l2ZSByZWZjb3VudCAxMDAwMDAgcmVhY2hlZCBvbiBhbzIgb2JqZWN0
IDB4NTVkNDI5MzZlOGQ4DQpbMDkvMjggMDk6MjM6MTAuODAwXSBFUlJPUlszMDM1XSBmcmFtZS5j OiBFeGNlc3NpdmUgcmVmY291bnQgMTAwMDAwIHJlYWNoZWQgb24gYW8yIG9iamVjdCAweDU1ZDQy OTM2ZThkOA0KWzA5LzI4IDA5OjIzOjEwLjgxMF0gRVJST1JbMzA3M11bQy0wMDAwMDFjM10gZnJh bWUuYzogRXhjZXNzaXZlIHJlZmNvdW50IDEwMDAwMCByZWFjaGVkIG9uIGFvMiBvYmplY3QgMHg1
NWQ0MjkzNmU4ZDgNClswOS8yOCAwOToyMzoxMC44MTBdIEVSUk9SWzMxNTNdIGZyYW1lLmM6IEV4
Y2Vzc2l2ZSByZWZjb3VudCAxMDAwMDAgcmVhY2hlZCBvbiBhbzIgb2JqZWN0IDB4NTVkNDI5MzZl OGQ4DQpbMDkvMjggMDk6MjM6MTAuODE0XSBFUlJPUlszMDgzXSBmcmFtZS5jOiBGUkFDSyEsIEZh aWxlZCBhc3NlcnRpb24gRXhjZXNzaXZlIHJlZmNvdW50IDEwMDAwMCByZWFjaGVkIG9uIGFvMiBv YmplY3QgMHg1NWQ0MjkzNmU4ZDggKDApDQpbMDkvMjggMDk6MjM6MTAuODE4XSBFUlJPUlszMDM1
XSBmcmFtZS5jOiBGUkFDSyEsIEZhaWxlZCBhc3NlcnRpb24gRXhjZXNzaXZlIHJlZmNvdW50IDEw MDAwMCByZWFjaGVkIG9uIGFvMiBvYmplY3QgMHg1NWQ0MjkzNmU4ZDggKDApDQpbMDkvMjggMDk6
MjM6MTAuODE0XSBFUlJPUlsyODg0XVtDLTAwMDAwMThiXSBmcmFtZS5jOiBFeGNlc3NpdmUgcmVm Y291bnQgMTAwMDAwIHJlYWNoZWQgb24gYW8yIG9iamVjdCAweDU1ZDQyOTM2ZThkOA0KWzA5LzI4
IDA5OjIzOjEwLjgyMF0gRVJST1JbMzA3Nl1bQy0wMDAwMDFjMV0gZnJhbWUuYzogRXhjZXNzaXZl IHJlZmNvdW50IDEwMDAwMCByZWFjaGVkIG9uIGFvMiBvYmplY3QgMHg1NWQ0MjkzNmU4ZDgNClsw OS8yOCAwOToyMzoxMC44MTJdIEVSUk9SWzI4OTZdW0MtMDAwMDAxOTBdIGZyYW1lLmM6IEV4Y2Vz c2l2ZSByZWZjb3VudCAxMDAwMDAgcmVhY2hlZCBvbiBhbzIgb2JqZWN0IDB4NTVkNDI5MzZlOGQ4
DQpbMDkvMjggMDk6MjM6MTAuODE5XSBFUlJPUlszMDczXVtDLTAwMDAwMWMzXSBmcmFtZS5jOiBG
UkFDSyEsIEZhaWxlZCBhc3NlcnRpb24gRXhjZXNzaXZlIHJlZmNvdW50IDEwMDAwMCByZWFjaGVk IG9uIGFvMiBvYmplY3QgMHg1NWQ0MjkzNmU4ZDggKDApDQpbMDkvMjggMDk6MjM6MTAuODIxXSBF
UlJPUlszMTUzXSBmcmFtZS5jOiBGUkFDSyEsIEZhaWxlZCBhc3NlcnRpb24gRXhjZXNzaXZlIHJl ZmNvdW50IDEwMDAwMCByZWFjaGVkIG9uIGFvMiBvYmplY3QgMHg1NWQ0MjkzNmU4ZDggKDApDQpb MDkvMjggMDk6MjM6MTAuODMxXSBFUlJPUlsyODg0XVtDLTAwMDAwMThiXSBmcmFtZS5jOiBGUkFD
SyEsIEZhaWxlZCBhc3NlcnRpb24gRXhjZXNzaXZlIHJlZmNvdW50IDEwMDAwMCByZWFjaGVkIG9u IGFvMiBvYmplY3QgMHg1NWQ0MjkzNmU4ZDggKDApDQpbMDkvMjggMDk6MjM6MTAuODM0XSBFUlJP
UlszMDc2XVtDLTAwMDAwMWMxXSBmcmFtZS5jOiBGUkFDSyEsIEZhaWxlZCBhc3NlcnRpb24gRXhj ZXNzaXZlIHJlZmNvdW50IDEwMDAwMCByZWFjaGVkIG9uIGFvMiBvYmplY3QgMHg1NWQ0MjkzNmU4
ZDggKDApDQpbMDkvMjggMDk6MjM6MTAuODMwXSBFUlJPUlsyODk2XVtDLTAwMDAwMTkwXSBmcmFt ZS5jOiBGUkFDSyEsIEZhaWxlZCBhc3NlcnRpb24gRXhjZXNzaXZlIHJlZmNvdW50IDEwMDAwMCBy ZWFjaGVkIG9uIGFvMiBvYmplY3QgMHg1NWQ0MjkzNmU4ZDggKDApDQoNCuKApi4NCg0KWzA5LzI4
IDA5OjMxOjU1LjM3Ml0gRVJST1JbMzE4MV0gOiBHb3QgOSBiYWNrdHJhY2UgcmVjb3Jkcw0KIyAw OiBbMHg1NWQ0Mjg2NTIxNzddIGFzdGVyaXNrIHV0aWxzLmM6MjQ1NCBfX2FzdF9hc3NlcnRfZmFp bGVkKCkNCiMgMTogWzB4NTVkNDI4NGJiNzQyXSBhc3RlcmlzayBhc3RvYmoyLmM6NTg4IF9fYW8y X3JlZigpDQojIDI6IFsweDU1ZDQyODU2YzEwYl0gYXN0ZXJpc2sgZnJhbWUuYzozNDkgYXN0X2Zy ZHVwKCkNCiMgMzogWzB4NTVkNDI4NGU3ZWNkXSBhc3RlcmlzayBicmlkZ2VfY2hhbm5lbC5jOjEw NzQgYXN0X2JyaWRnZV9jaGFubmVsX3F1ZXVlX2ZyYW1lKCkNCiMgNDogWzB4N2ZhYTc5YWFiZjQw XSBicmlkZ2Vfc29mdG1peC5zbyBicmlkZ2Vfc29mdG1peC5jOjE5Mzkgc29mdG1peF9taXhpbmdf bG9vcCgpDQojIDU6IFsweDdmYWE3OWFhYzU0ZF0gYnJpZGdlX3NvZnRtaXguc28gYnJpZGdlX3Nv ZnRtaXguYzoyMDQ3IHNvZnRtaXhfbWl4aW5nX3RocmVhZCgpDQojIDY6IFsweDU1ZDQyODY0ZWY5
Yl0gYXN0ZXJpc2sgdXRpbHMuYzoxMjk5IGR1bW15X3N0YXJ0KCkNCiMgNzogWzB4N2ZhYjA2NGI5
NmRiXSBsaWJwdGhyZWFkLnNvLjAgcHRocmVhZF9jcmVhdGUuYzo0NjMgc3RhcnRfdGhyZWFkKCkN
CiMgODogWzB4N2ZhYjA1NmFiNzFmXSBsaWJjLnNvLjYgY2xvbmUuUzo5NyBjbG9uZSgpDQoNClsw OS8yOCAwOTozMTo1NS4zNzhdIFdBUk5JTkdbMjkxMV1bQy0wMDAwMDE5NF0gY2hhbm5lbC5jOiBF
eGNlcHRpb25hbGx5IGxvbmcgdm9pY2UgcXVldWUgbGVuZ3RoIHF1ZXVpbmcgdG8gQ0JBbm4vSVNf X2IwZTdlMGE0LWUwN2QtNDMxNS05NjkyLWQ2MjA4ZTJkN2FjOS0wMDAwMDAxODsxDQpbMDkvMjgg MDk6MzE6NTUuNDI3XSBXQVJOSU5HWzMzMzJdW0MtMDAwMDAxMmNdIGNoYW5uZWwuYzogRXhjZXB0
aW9uYWxseSBsb25nIHZvaWNlIHF1ZXVlIGxlbmd0aCBxdWV1aW5nIHRvIENCQW5uL0lTX18xNGI2
YWU5Ny03OTRiLTQ0ZWEtODNmZC02YTE3ZTAyYjQzZjEtMDAwMDAwOTU7MQ0KWzA5LzI4IDA5OjMx OjU1LjQ0MV0gV0FSTklOR1szMDk1XVtDLTAwMDAwMWNiXSBjaGFubmVsLmM6IEV4Y2VwdGlvbmFs bHkgbG9uZyB2b2ljZSBxdWV1ZSBsZW5ndGggcXVldWluZyB0byBDQkFubi9JU19fNmMyZTk1NzUt YWU2OS00MGU4LWI0NWUtYjc1NmVhOWM2YTBlLTAwMDAwMDU1OzENClswOS8yOCAwOTozMTo1NS41
MzZdIEVSUk9SWzMxODddIDogR290IDkgYmFja3RyYWNlIHJlY29yZHMNCiMgMDogWzB4NTVkNDI4
NjUyMTc3XSBhc3RlcmlzayB1dGlscy5jOjI0NTQgX19hc3RfYXNzZXJ0X2ZhaWxlZCgpDQojIDE6
IFsweDU1ZDQyODRiYjc0Ml0gYXN0ZXJpc2sgYXN0b2JqMi5jOjU4OCBfX2FvMl9yZWYoKQ0KIyAy OiBbMHg1NWQ0Mjg1NmMxMGJdIGFzdGVyaXNrIGZyYW1lLmM6MzQ5IGFzdF9mcmR1cCgpDQojIDM6
IFsweDU1ZDQyODRlN2VjZF0gYXN0ZXJpc2sgYnJpZGdlX2NoYW5uZWwuYzoxMDc0IGFzdF9icmlk Z2VfY2hhbm5lbF9xdWV1ZV9mcmFtZSgpDQojIDQ6IFsweDdmYWE3OWFhYmY0MF0gYnJpZGdlX3Nv ZnRtaXguc28gYnJpZGdlX3NvZnRtaXguYzoxOTM5IHNvZnRtaXhfbWl4aW5nX2xvb3AoKQ0KIyA1
OiBbMHg3ZmFhNzlhYWM1NGRdIGJyaWRnZV9zb2Z0bWl4LnNvIGJyaWRnZV9zb2Z0bWl4LmM6MjA0
NyBzb2Z0bWl4X21peGluZ190aHJlYWQoKQ0KIyA2OiBbMHg1NWQ0Mjg2NGVmOWJdIGFzdGVyaXNr IHV0aWxzLmM6MTI5OSBkdW1teV9zdGFydCgpDQojIDc6IFsweDdmYWIwNjRiOTZkYl0gbGlicHRo cmVhZC5zby4wIHB0aHJlYWRfY3JlYXRlLmM6NDYzIHN0YXJ0X3RocmVhZCgpDQojIDg6IFsweDdm YWIwNTZhYjcxZl0gbGliYy5zby42IGNsb25lLlM6OTcgY2xvbmUoKQ0KDQpbMDkvMjggMDk6MzE6
NTUuNTU3XSBXQVJOSU5HWzMxOTVdW0MtMDAwMDAxMzBdIGNoYW5uZWwuYzogRXhjZXB0aW9uYWxs eSBsb25nIHZvaWNlIHF1ZXVlIGxlbmd0aCBxdWV1aW5nIHRvIENCQW5uL0lTX185Y2IwMTRkYy1m MDIyLTRiMzYtOGFiYi04MWExODRhZDZlYTgtMDAwMDAwNzE7MQ0KWzA5LzI4IDA5OjMxOjU1LjU4
NV0gV0FSTklOR1szMzIzXVtDLTAwMDAwMTZjXSBjaGFubmVsLmM6IEV4Y2VwdGlvbmFsbHkgbG9u ZyB2b2ljZSBxdWV1ZSBsZW5ndGggcXVldWluZyB0byBDQkFubi9JU19fNjA1NTU0NmMtZDU2MC00
MDc0LWJiYzUtM2E4YjMzYjY3NTliLTAwMDAwMDkyOzENClswOS8yOCAwOTozMTo1NS42MTJdIFdB
Uk5JTkdbMzA3Nl1bQy0wMDAwMDFjMV0gY2hhbm5lbC5jOiBFeGNlcHRpb25hbGx5IGxvbmcgdm9p Y2UgcXVldWUgbGVuZ3RoIHF1ZXVpbmcgdG8gQ0JBbm4vSVNfXzQ2YTZlYWFkLTNmODUtNDk0MS1h YTQ1LTkyYWI4MTRiNmIwZi0wMDAwMDA0ZjsxDQpbMDkvMjggMDk6MzE6NTUuNjM3XSBFUlJPUlsy ODk4XSA6IEdvdCA4IGJhY2t0cmFjZSByZWNvcmRzDQojIDA6IFsweDU1ZDQyODY1MjE3N10gYXN0
ZXJpc2sgdXRpbHMuYzoyNDU0IF9fYXN0X2Fzc2VydF9mYWlsZWQoKQ0KIyAxOiBbMHg1NWQ0Mjg0
YmI3NDJdIGFzdGVyaXNrIGFzdG9iajIuYzo1ODggX19hbzJfcmVmKCkNCiMgMjogWzB4N2ZhYTc5
YWE3NWMwXSBicmlkZ2Vfc29mdG1peC5zbyBicmlkZ2Vfc29mdG1peC5jOjI1NyBzb2Z0bWl4X3By b2Nlc3Nfd3JpdGVfYXVkaW8oKQ0KIyAzOiBbMHg3ZmFhNzlhYWJlZjddIGJyaWRnZV9zb2Z0bWl4
LnNvIGJyaWRnZV9zb2Z0bWl4LmM6MTkzNCBzb2Z0bWl4X21peGluZ19sb29wKCkNCiMgNDogWzB4
N2ZhYTc5YWFjNTRkXSBicmlkZ2Vfc29mdG1peC5zbyBicmlkZ2Vfc29mdG1peC5jOjIwNDcgc29m dG1peF9taXhpbmdfdGhyZWFkKCkNCiMgNTogWzB4NTVkNDI4NjRlZjliXSBhc3RlcmlzayB1dGls cy5jOjEyOTkgZHVtbXlfc3RhcnQoKQ0KIyA2OiBbMHg3ZmFiMDY0Yjk2ZGJdIGxpYnB0aHJlYWQu c28uMCBwdGhyZWFkX2NyZWF0ZS5jOjQ2MyBzdGFydF90aHJlYWQoKQ0KIyA3OiBbMHg3ZmFiMDU2
YWI3MWZdIGxpYmMuc28uNiBjbG9uZS5TOjk3IGNsb25lKCkNCg0KWzA5LzI4IDA5OjMxOjU1LjYz OF0gV0FSTklOR1syODcyXVtDLTAwMDAwMTg5XSBjaGFubmVsLmM6IEV4Y2VwdGlvbmFsbHkgbG9u ZyB2b2ljZSBxdWV1ZSBsZW5ndGggcXVldWluZyB0byBDQkFubi9JU19fYjZiOTk3NTAtYzlhYS00
MjBjLTllOWYtZmE0MWU5YTk5ZmE5LTAwMDAwMDBkOzENClswOS8yOCAwOTozMTo1NS43MDZdIFdB
Uk5JTkdbMzA5Ml1bQy0wMDAwMDFjYV0gY2hhbm5lbC5jOiBFeGNlcHRpb25hbGx5IGxvbmcgdm9p Y2UgcXVldWUgbGVuZ3RoIHF1ZXVpbmcgdG8gQ0JBbm4vSVNfXzQzZjQ1MTJmLTQwMDktNDQ4Zi05
Y2NmLTRkM2FkYTk3NWY3NC0wMDAwMDA1NDsxDQpbMDkvMjggMDk6MzE6NTUuNzY0XSBXQVJOSU5H
WzMyMzFdW0MtMDAwMDAxNjFdIGNoYW5uZWwuYzogRXhjZXB0aW9uYWxseSBsb25nIHZvaWNlIHF1
ZXVlIGxlbmd0aCBxdWV1aW5nIHRvIENCQW5uL0lTX181N2YyZTNhNy02Yzg2LTQyM2YtYjM1Zi0y NTA0OGVmNjg1OTEtMDAwMDAwN2Q7MQ0KWzA5LzI4IDA5OjMxOjU1Ljc3MV0gRVJST1JbMzIwOF0g OiBHb3QgOCBiYWNrdHJhY2UgcmVjb3Jkcw0KIyAwOiBbMHg1NWQ0Mjg2NTIxNzddIGFzdGVyaXNr IHV0aWxzLmM6MjQ1NCBfX2FzdF9hc3NlcnRfZmFpbGVkKCkNCiMgMTogWzB4NTVkNDI4NGJiNzQy XSBhc3RlcmlzayBhc3RvYmoyLmM6NTg4IF9fYW8yX3JlZigpDQojIDI6IFsweDdmYWE3OWFhNzVj MF0gYnJpZGdlX3NvZnRtaXguc28gYnJpZGdlX3NvZnRtaXguYzoyNTcgc29mdG1peF9wcm9jZXNz X3dyaXRlX2F1ZGlvKCkNCiMgMzogWzB4N2ZhYTc5YWFiZWY3XSBicmlkZ2Vfc29mdG1peC5zbyBi cmlkZ2Vfc29mdG1peC5jOjE5MzQgc29mdG1peF9taXhpbmdfbG9vcCgpDQojIDQ6IFsweDdmYWE3
OWFhYzU0ZF0gYnJpZGdlX3NvZnRtaXguc28gYnJpZGdlX3NvZnRtaXguYzoyMDQ3IHNvZnRtaXhf bWl4aW5nX3RocmVhZCgpDQojIDU6IFsweDU1ZDQyODY0ZWY5Yl0gYXN0ZXJpc2sgdXRpbHMuYzox Mjk5IGR1bW15X3N0YXJ0KCkNCiMgNjogWzB4N2ZhYjA2NGI5NmRiXSBsaWJwdGhyZWFkLnNvLjAg cHRocmVhZF9jcmVhdGUuYzo0NjMgc3RhcnRfdGhyZWFkKCkNCiMgNzogWzB4N2ZhYjA1NmFiNzFm XSBsaWJjLnNvLjYgY2xvbmUuUzo5NyBjbG9uZSgpDQoNCg0KRnJvbTogYXN0ZXJpc2stdXNlcnMg PGFzdGVyaXNrLXVzZXJzLWJvdW5jZXNAbGlzdHMuZGlnaXVtLmNvbT4gT24gQmVoYWxmIE9mIEpv c2h1YSBDLiBDb2xwDQpTZW50OiBUaHVyc2RheSwgU2VwdGVtYmVyIDIzLCAyMDIxIDEyOjAzIFBN
DQpUbzogQXN0ZXJpc2sgVXNlcnMgTWFpbGluZyBMaXN0IC0gTm9uLUNvbW1lcmNpYWwgRGlzY3Vz c2lvbiA8YXN0ZXJpc2stdXNlcnNAbGlzdHMuZGlnaXVtLmNvbT4NClN1YmplY3Q6IFJlOiBbYXN0
ZXJpc2stdXNlcnNdIEFueSB0aG91Z2h0cyBvbiBBc3RlcmlzayAxNi4xNy4wIG91dHB1dHRpbmcg RlJBQ0sgcmVmY291bnQgcmVsYXRlZCBtZXNzYWdlcw0KDQpPbiBUaHUsIFNlcCAyMywgMjAyMSBh dCAxOjU5IFBNIERhbiBDcm9wcCA8ZGFuQGFtdGVsY28uY29tPG1haWx0bzpkYW5AYW10ZWxjby5j b20+PiB3cm90ZToNCldlIGhhdmUgYW4gZXh0cmVtZWx5IGJ1c3kvbGFyZ2UgY3VzdG9tZXIuICBU
aGV5IHJ1biBmaW5lIG1vc3Qgb2YgdGhlIHRpbWUsIGJ1dCBwZXJpb2RpY2FsbHkgYXN0ZXJpc2sg d2lsbCBvdXRwdXQgRlJBQ0sgcmVmY291bnQgcmVsYXRlZCBtZXNzYWdlcy4gIEl0IGRvZXNu4oCZ
dCBzZWVtIHRvIGJlIHJlbGF0ZWQgdG8gdGhlIHZvbHVtZSwgYmVjYXVzZSBpdOKAmXMgbm90IGJy ZWFraW5nIGR1cmluZyB0aGVpciBwZWFrIHRpbWVzLg0KDQpXaGVuIHRoaXMgaGFwcGVucywgdGhl IHN5c3RlbSBiZWNvbWVzIHVuc3RhYmxlIGFuZCB0aGV5IGhhdmUgdG8gcmVzdGFydCB0byBnZXQg dGhpbmdzIHJlc29sdmVkLg0KVG8gZ2l2ZSBhbiBpZGVhIG9mIHRoZSBpbnN0YWJpbGl0eSwgd2Ug aGF2ZSBzZWVuIElOVklURS9UcnlpbmcgcmVzcG9uc2VzIGluIFNJUCBtZXNzYWdpbmcgbG9ncy4N
CldlIHRlbGwgQXN0ZXJpc2sgdG8gYW5zd2VyIHZpYSBBTUksIGJ1dCBBc3RlcmlzayBuZXZlciBz ZW5kcyB0aGUgT0sgKGV2ZW4gMjQgc2Vjb25kcyBsYXRlciBpdCBoYXNu4oCZdCBzZW50KS4NCkV2
ZW50dWFsbHkgdGhlIG90aGVyIHNlbmQgQ0FOQ0VMIG9mIHRoZSBjYWxsLg0KDQoNCldl4oCZdmUg bm93IGNhcHR1cmVkIDQgZGlmZmVyZW50IGRheXMgd2hlcmUgc29tZXRoaW5nIGxpa2UgdGhlIGZv bGxvd2luZyBvY2N1cnMuDQoxKSBJcyB0aGVyZSBhIGdvb2Qgd2F5IHRvIHRlbGwgaWYgdGhpcyBt YXkgYmUgZml4ZWQgaW4gQXN0ZXJpc2sgMTYuMjAuMCAoc2hvcnQgb2YgdXBncmFkaW5nKT8NCg0K
WW91IGNvdWxkIGV4YW1pbmUgdGhlIGNoYW5nZXMgYmV0d2VlbiB0aGUgdHdvIGFuZCBzZWUgaWYg YW55IGlzc3VlcyBzZWVtIHJlbGV2YW50Lg0KDQoyKSBXb3VsZCB0aGlzIGJlIHNvbWV0aGluZyBJ
IHNob3VsZCBzdWJtaXQgYXMgYW4gYXN0ZXJpc2sgaXNzdWU/ICBVbmZvcnR1bmF0ZWx5LCBzaXRl IGlzIHNvIGJ1c3kgY2FwdHVyaW5nIHRoZSBkZWJ1ZyB3aWxsIGJlIHZlcnkgZGlmZmljdWx0IChp ZiBub3QgaW1wb3NzaWJsZSkgZHVlIHRvIGFtb3VudCBvZiBkYXRhLg0KDQpXaXRob3V0IGEgYmFj a3RyYWNlLCBvciBkZWJ1ZyBsb2dzIGl0J2QgYmUgcmVhbGx5IGhhcmQgdG8gbG9vayBhdC4gVGhl IG1lc3NhZ2UganVzdCBtZWFucyB0aGF0IGFuIG9iamVjdCB3YXMgcmVmZXJlbmNlZCBhIGxvdC4g V2h5IHRoYXQgaXMgLSB3aG8ga25vd3MuIEl0IGNvdWxkIGJlIHRoYXQgdGhlIHN5c3RlbSBjYW4n dCBrZWVwIHVwIGZvciBzb21lIHJlYXNvbiwgb3IgdGhlcmUncyBhbiBvZmYtbm9taW5hbCBwYXRo LCBvciBzb21ldGhpbmcgaXMgZGVhZGxvY2tlZCwgb3Igc29tZXRoaW5nIGVsc2UuDQoNCi0tDQpK
b3NodWEgQy4gQ29scA0KQXN0ZXJpc2sgVGVjaG5pY2FsIExlYWQNClNhbmdvbWEgVGVjaG5vbG9n aWVzDQpDaGVjayB1cyBvdXQgYXQgd3d3LnNhbmdvbWEuY29tPGh0dHA6Ly93d3cuc2FuZ29tYS5j b20+IGFuZCB3d3cuYXN0ZXJpc2sub3JnPGh0dHA6Ly93d3cuYXN0ZXJpc2sub3JnPg0KDQpUaGlz IGVtYWlsIGlzIGludGVuZGVkIG9ubHkgZm9yIHRoZSB1c2Ugb2YgdGhlIHBhcnR5IHRvIHdoaWNo IGl0IGlzIGFkZHJlc3NlZCBhbmQgbWF5IGNvbnRhaW4gaW5mb3JtYXRpb24gdGhhdCBpcyBwcml2
aWxlZ2VkLCBjb25maWRlbnRpYWwsIG9yIHByb3RlY3RlZCBieSBsYXcuIElmIHlvdSBhcmUgbm90
IHRoZSBpbnRlbmRlZCByZWNpcGllbnQgeW91IGFyZSBoZXJlYnkgbm90aWZpZWQgdGhhdCBhbnkg ZGlzc2VtaW5hdGlvbiwgY29weWluZyBvciBkaXN0cmlidXRpb24gb2YgdGhpcyBlbWFpbCBvciBp dHMgY29udGVudHMgaXMgc3RyaWN0bHkgcHJvaGliaXRlZC4gSWYgeW91IGhhdmUgcmVjZWl2ZWQg dGhpcyBtZXNzYWdlIGluIGVycm9yLCBwbGVhc2Ugbm90aWZ5IHVzIGltbWVkaWF0ZWx5IGJ5IHJl cGx5aW5nIHRvIHRoZSBtZXNzYWdlIGFuZCBkZWxldGluZyBpdCBmcm9tIHlvdXIgY29tcHV0ZXIu DQo
It’s possible, yes.
I don’t *think* it would purely volume related. We have 16.17 deployments with very large loads running without issue, and we also run 16.17 against load simulators without issue.
In each case you have to traceback to find the cause of the problem. For example, a bad SBC which does not fully adhere to the SIP protocol could be confusing PJSIP, causing timeouts, etc. Our engineers spent a few days tracing such a problem before we shipped a high volume system to a customer earlier this year. (I have NOT traced through your logs below, so I’m not saying that is your problem)
There are some helpful posts here to help you trace, in the archives of this list. (In fact I see one responding to a question you asked 2 months ago). Did that yield nothing? (The response was dead on in terms of how to diagnose). If you share the result of that diagnosis you might get some helpful answers.
Dave
From: asterisk-users [mailto:asterisk-users-bounces@lists.digium.com] On Behalf Of Dan Cropp Sent: Thursday, September 23, 2021 12:59 PM
To: ‘asterisk-users@lists.digium.com’
Subject: [asterisk-users] Any thoughts on Asterisk 16.17.0 outputting FRACK
refcount related messages
We have an extremely busy/large customer. They run fine most of the time, but periodically asterisk will output FRACK refcount related messages. It doesn’t seem to be related to the volume, because it’s not breaking during their peak times.
When this happens, the system becomes unstable and they have to restart to get things resolved.
To give an idea of the instability, we have seen INVITE/Trying responses in SIP messaging logs.
We tell Asterisk to answer via AMI, but Asterisk never sends the OK (even 24
seconds later it hasn’t sent).
Eventually the other send CANCEL of the call.
We’ve now captured 4 different days where something like the following occurs.
1) Is there a good way to tell if this may be fixed in Asterisk 16.20.0
(short of upgrading)?
2) Would this be something I should submit as an asterisk issue?
Unfortunately, site is so busy capturing the debug will be very difficult
(if not impossible) due to amount of data.
[09/23 14:43:45.095] ERROR[34763][C-00001d7f] frame.c: Excessive refcount
100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.095] ERROR[34763][C-00001d7f] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.095] ERROR[29830] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.095] ERROR[29832][C-00001920] frame.c: Excessive refcount
100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.095] ERROR[29830] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.095] ERROR[29832][C-00001920] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.095] ERROR[32973] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.095] ERROR[32973] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.095] ERROR[3248] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.095] ERROR[3248] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.095] WARNING[2798][C-00000093] channel.c: Exceptionally long queue length queuing to CBAnn/IS__8b9c6719-ca29-4c1b-ac87-75e8c6fe7074-00000062;1
[09/23 14:43:45.095] ERROR[12979] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.095] ERROR[12979] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.095] WARNING[21123][C-00001157] channel.c: Exceptionally long queue length queuing to CBAnn/IS__a652155f-b1fb-4c31-83e5-09ffa2107979-000010de;1
[09/23 14:43:45.096] ERROR[29830] : Got 8 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_bridge_channel_queue_frame+0x61)
[0x5637ebf1cfe1]
# 3: /usr/lib/asterisk/modules/bridge_softmix.so(+0x40af) [0x7fc15362f0af]
# 4: /usr/lib/asterisk/modules/bridge_softmix.so(+0x560a) [0x7fc15363060a]
# 5: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.097] WARNING[36475][C-00000172] channel.c: Exceptionally long voice queue length queuing to CBAnn/IS__64bc075a-1ba4-4ad8-ba48-f0aea6ca6bab-00001ed3;1
[09/23 14:43:45.098] ERROR[12979] : Got 8 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_bridge_channel_queue_frame+0x61)
[0x5637ebf1cfe1]
# 3: /usr/lib/asterisk/modules/bridge_softmix.so(+0x40af) [0x7fc15362f0af]
# 4: /usr/lib/asterisk/modules/bridge_softmix.so(+0x560a) [0x7fc15363060a]
# 5: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.098] ERROR[34763][C-00001d7f] : Got 19 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(+0xa9517) [0x5637ebf2e517]
# 3: /usr/sbin/asterisk(ast_unreal_write_stream+0x15a) [0x5637ebf6ce1a]
# 4: /usr/sbin/asterisk(ast_write_stream+0xd2e) [0x5637ebf4551e]
# 5: /usr/sbin/asterisk(bridge_channel_internal_join+0xe6d) [0x5637ebf2238d]
# 6: /usr/sbin/asterisk(ast_bridge_join+0x3d5) [0x5637ebf05ce5]
# 7: /usr/lib/asterisk/modules/app_confbridge.so(+0xff35) [0x7fc05c258f35]
# 8: /usr/sbin/asterisk(pbx_exec+0xd2) [0x5637ebfdcf92]
# 9: /usr/lib/asterisk/modules/res_agi.so(+0xbea5) [0x7fc059269ea5]
#10: /usr/lib/asterisk/modules/res_agi.so(+0xd29b) [0x7fc05926b29b]
#11: /usr/lib/asterisk/modules/res_agi.so(+0xf7b0) [0x7fc05926d7b0]
#12: /usr/sbin/asterisk(pbx_exec+0xd2) [0x5637ebfdcf92]
#13: /usr/sbin/asterisk(+0x149062) [0x5637ebfce062]
#14: /usr/sbin/asterisk(+0x14d143) [0x5637ebfd2143]
#15: /usr/sbin/asterisk(+0x14e9e9) [0x5637ebfd39e9]
#15: /usr/sbin/asterisk(+0x14e9e9) [0x5637ebfd39e9]
#16: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
#17: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
#18: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.098] ERROR[29832][C-00001920] : Got 10 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(+0xa9517) [0x5637ebf2e517]
# 3: /usr/sbin/asterisk(ast_unreal_write+0x15a) [0x5637ebf6ca0a]
# 4: /usr/sbin/asterisk(ast_write_stream+0xd2e) [0x5637ebf4551e]
# 5: /usr/sbin/asterisk(bridge_channel_internal_join+0xe6d) [0x5637ebf2238d]
# 6: /usr/sbin/asterisk(+0x806f3) [0x5637ebf056f3]
# 7: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 9: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.099] ERROR[3248] : Got 8 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_bridge_channel_queue_frame+0x61)
[0x5637ebf1cfe1]
# 3: /usr/lib/asterisk/modules/bridge_softmix.so(+0x40af) [0x7fc15362f0af]
# 4: /usr/lib/asterisk/modules/bridge_softmix.so(+0x560a) [0x7fc15363060a]
# 5: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.099] ERROR[32973] : Got 8 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_bridge_channel_queue_frame+0x61)
[0x5637ebf1cfe1]
# 3: /usr/lib/asterisk/modules/bridge_softmix.so(+0x40af) [0x7fc15362f0af]
# 4: /usr/lib/asterisk/modules/bridge_softmix.so(+0x560a) [0x7fc15363060a]
# 5: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.207] ERROR[36630] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.207] ERROR[36630] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.207] ERROR[34172][C-00001cfb] frame.c: Excessive refcount
100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.207] ERROR[34172][C-00001cfb] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.207] ERROR[28738] frame.c: Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8
[09/23 14:43:45.207] ERROR[28738] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x5637ed3742b8 (0)
[09/23 14:43:45.207] ERROR[36630] : Got 8 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_bridge_channel_queue_frame+0x61)
[0x5637ebf1cfe1]
# 3: /usr/lib/asterisk/modules/bridge_softmix.so(+0x40af) [0x7fc15362f0af]
# 4: /usr/lib/asterisk/modules/bridge_softmix.so(+0x560a) [0x7fc15363060a]
# 5: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.208] WARNING[2798][C-00000093] channel.c: Exceptionally long queue length queuing to CBAnn/IS__8b9c6719-ca29-4c1b-ac87-75e8c6fe7074-00000062;1
[09/23 14:43:45.209] ERROR[34172][C-00001cfb] : Got 14 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_readframe+0x45) [0x5637ebf8c675]
# 3: /usr/lib/asterisk/modules/res_musiconhold.so(+0x4ca0) [0x7fc153b07ca0]
# 4: /usr/sbin/asterisk(+0xb5d6c) [0x5637ebf3ad6c]
# 5: /usr/sbin/asterisk(+0xc7a47) [0x5637ebf4ca47]
# 6: /usr/lib/asterisk/modules/res_agi.so(+0xeb9c) [0x7fc05926cb9c]
# 7: /usr/sbin/asterisk(pbx_exec+0xd2) [0x5637ebfdcf92]
# 8: /usr/sbin/asterisk(+0x149062) [0x5637ebfce062]
# 9: /usr/sbin/asterisk(+0x14d143) [0x5637ebfd2143]
#10: /usr/sbin/asterisk(+0x14e82b) [0x5637ebfd382b]
#11: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
#12: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
#13: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
[09/23 14:43:45.210] ERROR[28738] : Got 8 backtrace records
# 0: /usr/sbin/asterisk(__ao2_ref+0x209) [0x5637ebef1519]
# 1: /usr/sbin/asterisk(ast_frdup+0x1e2) [0x5637ebf96612]
# 2: /usr/sbin/asterisk(ast_bridge_channel_queue_frame+0x61)
[0x5637ebf1cfe1]
# 3: /usr/lib/asterisk/modules/bridge_softmix.so(+0x40af) [0x7fc15362f0af]
# 4: /usr/lib/asterisk/modules/bridge_softmix.so(+0x560a) [0x7fc15363060a]
# 5: /usr/sbin/asterisk(+0x1db41f) [0x5637ec06041f]
# 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc1e9ebf6db]
# 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc1e93f971f]
Dan
This email is intended only for the use of the party to which it is addressed and may contain information that is privileged, confidential, or protected by law. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this email or its contents is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer.