8 ms·

The ABI status of ELF hash tables

74 points by chmaynard 4y ago

jwilk 4y ago
Related: https://news.ycombinator.com/item?id=32471624 https://news.ycombinator.com/item?id=32471624 ("Win32 Is the Only Stable ABI on Linux?", over 500 comments)
- ajross 4y ago
  FWIW: that whole frame is just wrong anyway. Microsoft breaks stuff constantly. Everyone out there (including anti-cheat vendors) is relying on hackery and trickery to make their stuff work on windows. And every windows release a ton of that stuff breaks and these companies drop everything to re-reverse-engineer whatever it is they need. What works compatibly[1] is the win32 C API implemented along with the specific set of DLLs that you built your application against[2]. MS curates the kernel/driver interface below that fairly well[3]. But the kind of trickery EAC is doing here is right in the middle of things that absolutely do not work on windows either. [1] In the sense of "do this and your app won't break". [2] Which is why windows binaries ship their own C libraries. FWIW this model exists in Linux too. It's basically the way we package Docker and Snap containers. Had EAC been written against a particular Snap environment, we wouldn't be having this problem either because it would be frozen on a particular glibc version. [3] And above that level, there are often other runtimes like .NET to which you can portably target.
YouWhy 4y ago
I strongly favor the Linux\Linus "don't break userspace" attitude than what has just happened for glibc. This is especially salient given that the DT_GNU_HASH format is not even documented. Leaving it as the only-option-by-default going forward is unconscionable.
- formerly_proven 4y ago
  > On typical distributions, none of the system libraries use DT_HASH, not even core libraries such as libgcc_s.so.1. Any tool that performs symbol lookups needs to support DT_GNU_HASH these days. It seems to me that EAC includes its own copy of a dynamic linker for the express purpose of validating that glibc functions are not obviously detoured in the current process.
  - fweimer 4y ago
    It's equally possible that they added DT_HASH lookup to achieve glibc 2.34 compatibility. The old way for getting the original dlsym function after interposing dlsym stopped working because the (internal, usually uninterposed, now unused) __libc_dlsym@@GLIBC_PRIVATE symbol went away in glibc 2.34. But a program can still obtain the dlsym function address by looking at the ELF data structures directly. This would explain why people are so annoyed: there's unplanned extra low-level work to enable glibc upgrades two years in a row.
- titzer 4y ago
  This is part of the reason Virgil doesn't rely on any userspace libraries. It uses the kernel directly, which is very stable on Linux.
dmitrygr 4y ago
There is the strict interpretation of the words "what is an abi?" and then there is the real world "what breaks things users would rather not be broken?" Maintainers of a package used by literally almost every executable really ought to use the latter thought process, IMHO. Hiding behind the "well, technically..." is kind of irresponsible. Hiding behind "It's been 16 years since we introduced $NEW_THING" is doubly so. Who the hell reads glibc's changelists for fun? You know the quote: "on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.`"
cratermoon 4y ago
Sympathy for the problems for EAC seems to be in short supply, and I can sympathize. It doesn't really excuse the abrupt removal of the compatible hash table format without at least a deprecation effort ahead of time. What I would expect to see would be something like step 1: provide complete and accurate documentation for the new format and communicate it widely and prominently to the developer community step 2: set a target date & version for removal, and get feedback from the community to ensure it gives sufficient time and information to complete a migration. Provide a way for developers to ask for and get support to make the necessary changes. Release sample code, if possible. step 3: one release before removal, loudly note the deprecation and the upcoming change. step 4: as the release that removes backward compatibility is being prepared, again, communicate the upcoming change and provide guidance for those who can't update on any ways to mitigate problems. step 5: finally, release the version with the breaking changes, and put the break boldly at the top of the release notes/changelog. Reiterate the previous guidance on mitigating problems and help. step 6: Go have a tasty beverage.
- cesarb 4y ago
  > It doesn't really excuse the abrupt removal of the compatible hash table format without at least a deprecation effort ahead of time. I strongly suspect that the reason for the "abrupt" removal was that it wasn't expected that anything would break: that table is used for dynamic linking, and the dynamic linker comes with glibc itself, so it can be assumed to understand the new table (and the dynamic linker which comes with other C libraries would be used to load these C libraries, and not glibc since you cannot load more than one C library in a single process). AFAIK, the code in the dynamic linker which understands files with only the old table wasn't removed, so from that point of view, compatibility was being kept. That is, they didn't expect anything to be using that table without going through dlopen() and dlsym().
  - cratermoon 4y ago
    Public vs. Published. https://martinfowler.com/ieeeSoftware/published.pdf https://martinfowler.com/ieeeSoftware/published.pdf
  - 4y ago
riskable 4y ago
Going through great pains to support EAC is a bit ridiculous. EAC itself is ridiculous from the get-go since it doesn't actually prevent cheaters it just pretends. It's not even meant to stop cheaters! Its entire purpose is to lock developers into Epic's proprietary system and prevent them from porting to non-approved platforms. The entire premise of client side anti-cheat tools is, "trust the client." I think pretty much everyone at Hacker News knows that's the exact opposite of how you make a secure system. Never trust the client.
- m-p-3 4y ago
  The issue with the server-client relationship in games is there a trade-off between latency and security required. Offload too many tasks to the server and you end up with a game where you encounter latency where it's critical, like hit detection (and likely many others). Those extra milliseconds does make or break the experience. Not saying it's an impossible problem to solve, just that multiplayer games are operating in a time-sensitive environment.
- brigade 4y ago
  I don't think anyone is proposing to eliminate support for --hash-style=both. Correct me if I'm wrong? So at this point, the maintenance burden on glibc is however much effort they want to spend justifying breaking backwards compatibility.
  - ajross 4y ago
    They didn't break backwards compatibility, only forwards. The core issue here is that there is a new format that won't work with old runtime linkers. But the only way to present that format to those old linkers[1] is to take a binary compiled with a glibc version from the last few weeks (!) and run it against on a system where the ld-linux.so is from pre-2006 glibcs. That direction of compatibility is not impossible, but it's not what one normally considers when discussing "breaking backwards compatibility". Fundamentally the only way to provide the kind of compatibility you're demanding, where any level of reverse engineering or parallel implementation must be supported in perpetuity, is to just Never Change Anything At All. [1] With the sole known exception of the "runtime linker" embedded in this EAC product under discussion.
miohtama 4y ago
Here is the crux of the issue: > Blaming the EAC developers for not keeping up with Linux ELF hash-table formats might not be entirely fair. The DT_HASH format is mandated by the System V ABI specification, the DT_GNU_HASH format is undocumented, and there has been no deprecation campaign to get users to move on. Most problems like this are communication and co-ordination problems.
- rwmj 4y ago
  > communication and co-ordination problems Which only exist in this case because the software isn't open source. This would have been spotted years ago in open source.
  - Spivak 4y ago
    Unless there’s bots out there scanning public code repos for esoteric stuff like this I kinda doubt it. If you mean it would have gotten caught by packagers then yeah but that presumes a world where `apt install eac` makes any sense.
    - rwmj 4y ago
      We do actually have a way to rebuild Fedora with particular changes (eg. when we change the baseline x86-64 version). I guess other distros can do that too. Of course it goes almost without saying that EAC would make about as much sense as an open source project as it does as a closed source project, which is to say none whatsoever because it's conceptually nonsense :-)
  - Narishma 4y ago
    An open source anti-cheat system doesn't make much sense.
    - akira2501 4y ago
      Neither does a closed source one, it is running on _my_ computer after all. Cynically.. keeping it closed just allows you to call it "anti-cheat" with a straight face, rather than what it actually is.. "mildly successful cheat mitigation and frustration techniques." Which is obvious once you notice there is an active "anti-anti-cheat" marketplace.
Arnavion 4y ago
If glibc updates its version number and EAC starts failing because it used too small a data type for that, does that mean glibc broke ABI compatibility and should be stuck on a lower version forever? Closed source software that wants to hold back open source software can go pound sand. Gamers can run their games with older glibc in a chroot / container. I say this as someone who does exactly this for games.
- mrguyorama 4y ago
  In the Windows world the answer to that question was emphatically "Yes, you should contort yourself to make sure users can still actually USE their computer", to the point that the official version number of windows was "3.95" because some apps looked at only the minor version number and would have broken otherwise.
  - Arnavion 4y ago
    Yes. And? What part of my comment applies to Windows? Certainly not the "open source software" part. BTW my comment lists two ways for people to continue "still actually USING their computers".
- Snetry 4y ago
  Other stuff like the game Shovel Knight and even an open source frame limiting library broke because of this change, there are probably more things but this has not reached the masses yet. I dont think the change was inherently bad but just ham fisting it into the release without widely announcing that the ABI will change is a massive failure on the glibc teams end.
ajross 4y ago
I don't know what people expect glibc to do here. EAC is a tool that reverse engineered library data that is only supposed to be used by the system runtime linker. People reverse engineer stuff all the time, and it's universally agreed that the onus is on the Weird Hack to bother with maintenance. EAC isn't using a supported API at all (the API for this is dlopen/dlsym et. al., not the hash format!). It's unfortunate, sure, but it's just not possible to prevent. And all the people screaming about "This Doesn't Happen On Windows" are just fooling themselves (or, to be blunt, being deliberately disingenous). This happens on windows every single release. Windows N+1 comes out and suddently a bunch of hacky drivers or tools or gadgets stop working until their authors manage to figure out what part of their trickery Microsoft broke.
- ronsor 4y ago
  Anticheat software like EAC (as well as commercial antiviruses and malicious rootkits) often reverse-engineer internal system data structures and call undocumented functions. You really can't blame system library developers for breaking compatibility with software that is doing things "the wrong way." Yes, I know Linus Torvalds says "don't break userspace," but that's for officially exposed kernel interfaces.
- dark-star 4y ago
  nothing was "reverse-engineered". Read the article. DT_HASH has been documented in SysV since ages. It's the DT_GNU_HASH that's (almost) undocumented and needs to be reverse-engineered from GNU ld's source code.
cosmiccatnap 4y ago
Seems like there is some handwaving here that "DT_HASH" should be supported forever because "The DT_HASH format is mandated by the System V ABI specification" It was replaced by DT_GNU_HASH in the gnu implementation of libc ....I think these people have forgotten what GNU stands for.
mhh__ 4y ago
Finding the most up to date ELF spec for your system is remarkably annoying.
- cesarb 4y ago
  Some time ago, https://refspecs.linuxfoundation.org/ https://refspecs.linuxfoundation.org/ was created to fix that problem. I don't know whether it's been kept up to date since then, however.

74 points by chmaynard 4y ago

Related: https://news.ycombinator.com/item?id=32471624 https://news.ycombinator.com/item?id=32471624 ("Win32 Is the Only Stable ABI on Linux?", over 500 comments)

ajross 4y ago
FWIW: that whole frame is just wrong anyway. Microsoft breaks stuff constantly. Everyone out there (including anti-cheat vendors) is relying on hackery and trickery to make their stuff work on windows. And every windows release a ton of that stuff breaks and these companies drop everything to re-reverse-engineer whatever it is they need. What works compatibly[1] is the win32 C API implemented along with the specific set of DLLs that you built your application against[2]. MS curates the kernel/driver interface below that fairly well[3]. But the kind of trickery EAC is doing here is right in the middle of things that absolutely do not work on windows either. [1] In the sense of "do this and your app won't break". [2] Which is why windows binaries ship their own C libraries. FWIW this model exists in Linux too. It's basically the way we package Docker and Snap containers. Had EAC been written against a particular Snap environment, we wouldn't be having this problem either because it would be frozen on a particular glibc version. [3] And above that level, there are often other runtimes like .NET to which you can portably target.

YouWhy 4y ago

I strongly favor the Linux\Linus "don't break userspace" attitude than what has just happened for glibc. This is especially salient given that the DT_GNU_HASH format is not even documented. Leaving it as the only-option-by-default going forward is unconscionable.

formerly_proven 4y ago
> On typical distributions, none of the system libraries use DT_HASH, not even core libraries such as libgcc_s.so.1. Any tool that performs symbol lookups needs to support DT_GNU_HASH these days. It seems to me that EAC includes its own copy of a dynamic linker for the express purpose of validating that glibc functions are not obviously detoured in the current process.
- fweimer 4y ago
  It's equally possible that they added DT_HASH lookup to achieve glibc 2.34 compatibility. The old way for getting the original dlsym function after interposing dlsym stopped working because the (internal, usually uninterposed, now unused) __libc_dlsym@@GLIBC_PRIVATE symbol went away in glibc 2.34. But a program can still obtain the dlsym function address by looking at the ELF data structures directly. This would explain why people are so annoyed: there's unplanned extra low-level work to enable glibc upgrades two years in a row.
titzer 4y ago
This is part of the reason Virgil doesn't rely on any userspace libraries. It uses the kernel directly, which is very stable on Linux.

dmitrygr 4y ago

There is the strict interpretation of the words "what is an abi?" and then there is the real world "what breaks things users would rather not be broken?" Maintainers of a package used by literally almost every executable really ought to use the latter thought process, IMHO. Hiding behind the "well, technically..." is kind of irresponsible. Hiding behind "It's been 16 years since we introduced $NEW_THING" is doubly so. Who the hell reads glibc's changelists for fun? You know the quote: "on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.`"

cratermoon 4y ago

Sympathy for the problems for EAC seems to be in short supply, and I can sympathize. It doesn't really excuse the abrupt removal of the compatible hash table format without at least a deprecation effort ahead of time. What I would expect to see would be something like step 1: provide complete and accurate documentation for the new format and communicate it widely and prominently to the developer community step 2: set a target date & version for removal, and get feedback from the community to ensure it gives sufficient time and information to complete a migration. Provide a way for developers to ask for and get support to make the necessary changes. Release sample code, if possible. step 3: one release before removal, loudly note the deprecation and the upcoming change. step 4: as the release that removes backward compatibility is being prepared, again, communicate the upcoming change and provide guidance for those who can't update on any ways to mitigate problems. step 5: finally, release the version with the breaking changes, and put the break boldly at the top of the release notes/changelog. Reiterate the previous guidance on mitigating problems and help. step 6: Go have a tasty beverage.

cesarb 4y ago
> It doesn't really excuse the abrupt removal of the compatible hash table format without at least a deprecation effort ahead of time. I strongly suspect that the reason for the "abrupt" removal was that it wasn't expected that anything would break: that table is used for dynamic linking, and the dynamic linker comes with glibc itself, so it can be assumed to understand the new table (and the dynamic linker which comes with other C libraries would be used to load these C libraries, and not glibc since you cannot load more than one C library in a single process). AFAIK, the code in the dynamic linker which understands files with only the old table wasn't removed, so from that point of view, compatibility was being kept. That is, they didn't expect anything to be using that table without going through dlopen() and dlsym().
- cratermoon 4y ago
  Public vs. Published. https://martinfowler.com/ieeeSoftware/published.pdf https://martinfowler.com/ieeeSoftware/published.pdf
- 4y ago

riskable 4y ago

Going through great pains to support EAC is a bit ridiculous. EAC itself is ridiculous from the get-go since it doesn't actually prevent cheaters it just pretends. It's not even meant to stop cheaters! Its entire purpose is to lock developers into Epic's proprietary system and prevent them from porting to non-approved platforms. The entire premise of client side anti-cheat tools is, "trust the client." I think pretty much everyone at Hacker News knows that's the exact opposite of how you make a secure system. Never trust the client.

m-p-3 4y ago
The issue with the server-client relationship in games is there a trade-off between latency and security required. Offload too many tasks to the server and you end up with a game where you encounter latency where it's critical, like hit detection (and likely many others). Those extra milliseconds does make or break the experience. Not saying it's an impossible problem to solve, just that multiplayer games are operating in a time-sensitive environment.
brigade 4y ago
I don't think anyone is proposing to eliminate support for --hash-style=both. Correct me if I'm wrong? So at this point, the maintenance burden on glibc is however much effort they want to spend justifying breaking backwards compatibility.
- ajross 4y ago
  They didn't break backwards compatibility, only forwards. The core issue here is that there is a new format that won't work with old runtime linkers. But the only way to present that format to those old linkers[1] is to take a binary compiled with a glibc version from the last few weeks (!) and run it against on a system where the ld-linux.so is from pre-2006 glibcs. That direction of compatibility is not impossible, but it's not what one normally considers when discussing "breaking backwards compatibility". Fundamentally the only way to provide the kind of compatibility you're demanding, where any level of reverse engineering or parallel implementation must be supported in perpetuity, is to just Never Change Anything At All. [1] With the sole known exception of the "runtime linker" embedded in this EAC product under discussion.

miohtama 4y ago

Here is the crux of the issue: > Blaming the EAC developers for not keeping up with Linux ELF hash-table formats might not be entirely fair. The DT_HASH format is mandated by the System V ABI specification, the DT_GNU_HASH format is undocumented, and there has been no deprecation campaign to get users to move on. Most problems like this are communication and co-ordination problems.

rwmj 4y ago
> communication and co-ordination problems Which only exist in this case because the software isn't open source. This would have been spotted years ago in open source.
- Spivak 4y ago
  Unless there’s bots out there scanning public code repos for esoteric stuff like this I kinda doubt it. If you mean it would have gotten caught by packagers then yeah but that presumes a world where `apt install eac` makes any sense.
  - rwmj 4y ago
    We do actually have a way to rebuild Fedora with particular changes (eg. when we change the baseline x86-64 version). I guess other distros can do that too. Of course it goes almost without saying that EAC would make about as much sense as an open source project as it does as a closed source project, which is to say none whatsoever because it's conceptually nonsense :-)
- Narishma 4y ago
  An open source anti-cheat system doesn't make much sense.
  - akira2501 4y ago
    Neither does a closed source one, it is running on _my_ computer after all. Cynically.. keeping it closed just allows you to call it "anti-cheat" with a straight face, rather than what it actually is.. "mildly successful cheat mitigation and frustration techniques." Which is obvious once you notice there is an active "anti-anti-cheat" marketplace.

Arnavion 4y ago

If glibc updates its version number and EAC starts failing because it used too small a data type for that, does that mean glibc broke ABI compatibility and should be stuck on a lower version forever? Closed source software that wants to hold back open source software can go pound sand. Gamers can run their games with older glibc in a chroot / container. I say this as someone who does exactly this for games.

mrguyorama 4y ago
In the Windows world the answer to that question was emphatically "Yes, you should contort yourself to make sure users can still actually USE their computer", to the point that the official version number of windows was "3.95" because some apps looked at only the minor version number and would have broken otherwise.
- Arnavion 4y ago
  Yes. And? What part of my comment applies to Windows? Certainly not the "open source software" part. BTW my comment lists two ways for people to continue "still actually USING their computers".
Snetry 4y ago
Other stuff like the game Shovel Knight and even an open source frame limiting library broke because of this change, there are probably more things but this has not reached the masses yet. I dont think the change was inherently bad but just ham fisting it into the release without widely announcing that the ABI will change is a massive failure on the glibc teams end.

ajross 4y ago

I don't know what people expect glibc to do here. EAC is a tool that reverse engineered library data that is only supposed to be used by the system runtime linker. People reverse engineer stuff all the time, and it's universally agreed that the onus is on the Weird Hack to bother with maintenance. EAC isn't using a supported API at all (the API for this is dlopen/dlsym et. al., not the hash format!). It's unfortunate, sure, but it's just not possible to prevent. And all the people screaming about "This Doesn't Happen On Windows" are just fooling themselves (or, to be blunt, being deliberately disingenous). This happens on windows every single release. Windows N+1 comes out and suddently a bunch of hacky drivers or tools or gadgets stop working until their authors manage to figure out what part of their trickery Microsoft broke.

ronsor 4y ago
Anticheat software like EAC (as well as commercial antiviruses and malicious rootkits) often reverse-engineer internal system data structures and call undocumented functions. You really can't blame system library developers for breaking compatibility with software that is doing things "the wrong way." Yes, I know Linus Torvalds says "don't break userspace," but that's for officially exposed kernel interfaces.
dark-star 4y ago
nothing was "reverse-engineered". Read the article. DT_HASH has been documented in SysV since ages. It's the DT_GNU_HASH that's (almost) undocumented and needs to be reverse-engineered from GNU ld's source code.

cosmiccatnap 4y ago

Seems like there is some handwaving here that "DT_HASH" should be supported forever because "The DT_HASH format is mandated by the System V ABI specification" It was replaced by DT_GNU_HASH in the gnu implementation of libc ....I think these people have forgotten what GNU stands for.

mhh__ 4y ago

Finding the most up to date ELF spec for your system is remarkably annoying.

cesarb 4y ago
Some time ago, https://refspecs.linuxfoundation.org/ https://refspecs.linuxfoundation.org/ was created to fix that problem. I don't know whether it's been kept up to date since then, however.