4 ms·
At first glibc's use of TSX/RTM for mutexes was behind a compile-time opt-in, and Intel was hyping up how great hardware transactional memory was, so most distr
by rrss 5y ago
At first glibc's use of TSX/RTM for mutexes was behind a compile-time opt-in, and Intel was hyping up how great hardware transactional memory was, so most distros flipped the switch to enable it. But glibc's use of TSX/RTM for rwlocks was not behind a compile-time guard, and was used whenever CPUID said it was enabled.
I think the first issue was that pthread locks using TSX behaved differently for incorrect usage, so that applications that had double-unlock bugs worked fine with the normal implementation but crashed with TSX glibc. These were app bugs, but a lot of people were unhappy that a glibc update caused their applications to crash, so some distros patched glibc to make time to fix the apps.
Then TSX was broken in Haswell and Intel disabled it, but it was a total mess, such that the bit in CPUID indicating support for TSX/RTM did not reliably indicate support for TSX/RTM (i.e. bit was set indicating support, but any TSX/RTM instruction resulted in SIGILL), so various versions of glibc ended up with various lists of CPU model numbers to determine when to avoid using TSX.
repeat for broadwell.
At this point more distros decided to turn off TSX in glibc entirely. But most did this by just removing the compile-time opt-in, which did not do anything for rwlocks. A couple distros noticed that rwlocks were still using TSX, and patched that too to remove lock elision entirely - but many did not, so rwlocks on haswell or early broadwell would still cause issues.
In my experience, a lot of the time RTM was not useful for performance because the transactions would be too large and abort. It was worse because in these cases glibc would try the transaction again several times before giving up, which could cause massive performance hits compared to just doing the normal lock to start with.
There were errata about TSX in skylake too, but I can't remember if Intel actually turned RTM off with microcode before now or just left it as is because no one was using it.
Eventually glibc was changed to always compile in support for TSX but require a runtime opt-in using an environment variable.
(this is from memory, the timeline is probably not quite right)
EDIT: I had a bit of deja-vu writing this, and it turns out I have whined about this on HN before lol: https://news.ycombinator.com/item?id=22694546 https://news.ycombinator.com/item?id=22694546
- aj3 5y agoThank you, this is very insightful.