Recent UnixODBC Issues
Kia ora,
As many of you are aware (and for those who aren’t) as part of Asterisk
13.8 changes went into the res_odbc module to more heavily leverage UnixODBC connection management and pooling capabilities. Previously we would use only a single connection to the database. Since this has been released many individuals have reported crashes and apparent deadlocks as a result of this change. We’ve been investigating and have some information we’d like to share so everyone can be aware of where things stand and where we can use some help.
These overall crashes have been a result of issues within both the database connectors themselves and UnixODBC.
For database connectors old versions seem to have issues with unicode support within UnixODBC and with the threading constraints now placed upon them by using multiple connections heavily. Using the latest versions has resolved these specific crashes for people.
For the UnixODBC crash the problem occurs at disconnect due to inadequate thread protection around critical environment data. This was fixed as of UnixODBC 2.3.2. Unfortunately distros may be shipping old versions causing the problem to be present for many individuals.
The apparent deadlock appears to be due to a regression introduced in UnixODBC 2.3.3 in configuration caching. This has resulted in the cache not being used and also in the cache growing out of control. For each connection request the configuration is cached again in a linked list. As the number of connections grow this list grows longer and longer taking longer to iterate. While the cache has a 20 second expiration at heavy connection use it’s still not enough to combat the growth. This issue has been reported upstream[1] and is on its way to being fixed in UnixODBC itself.
To help confirm some of these we’d like people to try (if your environment allows you) a combination of UnixODBC 2.3.2 and the latest version of their database connector. This will allow us to confirm some of the above information in more environments and to see if any other issues may occur.
Thank you for your help!
Cheers,
[1] http://mailman.unixodbc.org/pipermail/unixodbc-dev/2016-June/001890.html
4 thoughts on - Recent UnixODBC Issues
Happy Monday all,
Since I sent my previous email a lot has been learnt about our UnixODBC
problem and a path has emerged ensuring both better performance while making sure people are not required to upgrade their UnixODBC unless they want to.
So what’s this mean?
As of Asterisk 13.10.0 our own connection pool will be used in the res_odbc module. Under testing this has shown to be more performant than the UnixODBC implementation and also does not suffer the slowdown present in UnixODBC 2.3.3 and above. As well since our own connection pool has a fixed size we can restore behavior to that of previous versions so those using UnixODBC 2.3.1 will not experience the crashes that have been seen.
This is done by having the default be 1 connection, thus disabling connection pooling. To turn connection pooling on you will need to ensure you are using UnixODBC 2.3.2 or above with latest database connectors and configure a maximum limit on concurrent connections in res_odbc.conf. To facilitate figuring out the right limit for your environment I’ve made the current count and limit available using the
“odbc show” CLI command.
This change is up for review[1] and any feedback would be welcome, both from code review itself and testing.
Cheers,
[1] https://gerrit.asterisk.org/#/c/2943/
Dne 6.6.2016 v 17:42 Joshua Colp napsal(a):
whats the recommended settings in odbcinst.ini Pooling = no threading = 0
?
do you think it is possible backport this to 13.9 or are there some problematic dependencies?
Marek Červenka wrote:
The default settings are fine, including disabling pooling in UnixODBC. Due to the way the new connection pool works the pooling setting in UnixODBC won’t make a difference.
The change should apply cleanly to 13.9.
—
Joshua Colp Digium, Inc. | Senior Software Developer
445 Jan Davis Drive NW – Huntsville, AL 35806 – US
Check us out at: http://www.digium.com & http://www.asterisk.org
—
Happy Taco Tuesday All,
The change I previously mentioned[1] has been merged into all applicable branches and will be in the next release. From our own testing it has shown to solve all the problems mentioned and should return UnixODBC to a stable state. I hope that this won’t be the case but if you’re experiencing any issues using the 13 branch as of right now (or the next release) please create a new issue on the issue tracker[2] with details about your database connector version, UnixODBC version, version of Asterisk, and any other details you think will be useful and we will see what’s up.
We now return you to your regularly scheduled database usage!
Cheers,
[1] https://gerrit.asterisk.org/#/c/2943/
[2] https://issues.asterisk.org/jira