5 ms·

Thoroughly Understanding C++ ABI (2024)

83 points by rramadass 3mo ago

rramadass 3mo ago
As always, read wikipedia and follow links as needed - https://en.wikipedia.org/wiki/Application_binary_interface https://en.wikipedia.org/wiki/Application_binary_interface
rramadass 3mo ago
GNU libstdc++ "Dual ABI" issue - https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_a...
- jdw64 2mo ago
  I understand that Move Semantics led to better implementations and that this is why things changed. I know that in this case, problems arose when transitioning from C++98 to what's commonly called Modern C++ (C++11). However, I'm not very familiar with what the specific issues were with COW. If you happen to know of any documents that describe this problem in more detail, I'd appreciate it if you could let me know. I'm sorry for always asking questions. I've looked up a few documents, but since you clearly know much more about this than I do, I'm asking to study it myself as well. If it's too much trouble, feel free not to reply.
  - aw1621107 2mo ago
    The changes to std::string were made because "the [pre-C++11] definition of basic_string allows only very limited concurrent access to strings. Such limited concurrency will inhibit performance in multi-threaded applications." The proposed fix was to make "all iterator and element access operations safely concurrently executable." [0]. This involved limiting the circumstances in which iterators could be invalidated, which among other things meant that non-const operator[] could no longer invalidate iterators, which meant copy-on-write was no longer a viable implementation [1]. [0]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2668.htm https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n26... [1]: https://stackoverflow.com/q/12199710 https://stackoverflow.com/q/12199710
    - jdw64 2mo ago
      Thank you for the kind comment. !
  - rramadass 2mo ago
    Asking questions is always a good thing so don't worry about it. It makes us think and check our own understanding too, so we also learn something :-) A nice succinct post by Pavel Khaipov; Copy-on-Write in Modern C++: Why the Standard Dropped It - and Why Qt Still Uses It - https://www.linkedin.com/posts/khaipov_copy-on-write-in-modern-c-why-the-standard-activity-7405201870232989696-QuI8 https://www.linkedin.com/posts/khaipov_copy-on-write-in-mode... A detailed post by Andrii Nikishaiev; Why Copy-On-Write (COW) removed in C++? - https://www.linkedin.com/pulse/why-copy-on-write-cow-removed-c-andrii-nikishaiev-ua-9kmvf https://www.linkedin.com/pulse/why-copy-on-write-cow-removed... The std document on Concurrency Modifications to Basic String - https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2534.html https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n25... Finally, C++11 tightened the specification for std::string viz. contiguous memory, some methods must have O(1) access time, when references/iterators/pointers can/cannot be invalidated etc. which doomed COW.
    - jdw64 2mo ago
      I'm sorry for always bothering you, senior. I've learned quite a lot from the books you recommended. I've especially been studying low-level topics lately, and the books you suggested have been very helpful. Have a great day.
      - rramadass 2mo ago
        I am glad that you found my comments useful. I use HN as my "blog" and generally include references to books/papers/articles as a pointer to further information on a topic. You will find links to lots of books if you search my comments for key phrases (eg. embedded, book etc.). One thing i try to do is point people to lesser-known but very good and advanced books. I generally find that most HN book suggestions refer only to popular books most of which are just so-so while many great books remain unknown which is a shame. For example how many people know of Operating Systems in Depth by Thomas Doeppner? It has lots of helpful illustrations, compares concepts in both Linux/Windows implementations and is an all-round great book. I really understood the nuances of signal handling between kernel/user modes when i read it here.
Dwedit 2mo ago
C++ ABIs are why Win32 settled on things like COM to do cross-module passing of objects. Because nobody could agree on a standard, the COM standard enforced a very specific calling convention, and a very specific vtable layout for COM objects.
- pjmlp 2mo ago
  Not only, there is also XPC and IO/DriverKit on Apple, Android IPC, D-BUS, FIDL on Fucshia,.... Also C ABI also does not exist, people keep mistaking the ABI of their favourite C compiler with the OS ABI, which only overlap if the OS was written in C to start with. For example on mainframes and micros, naturally not written in C, it is either a bytecode based ABI like TIMI on IBM i, or language environments like on z/OS, ClearPath MCP, OS 2200 and so forth. And as a reminder, from a famous WG14 and WG21 member, and former Rust contributor, "To Save C, We Must Save ABI" https://thephd.dev/to-save-c-we-must-save-abi-fixing-c-function-abi https://thephd.dev/to-save-c-we-must-save-abi-fixing-c-funct...
  - bheadmaster 2mo ago
    System V ABI is pretty standardized.
    - pjmlp 2mo ago
      UNIX, the reason why C exists.
      - bheadmaster 2mo ago
        Yeah, I'm not interested in decoding what you meant by your aphorism. Either say what you mean or don't.
        pjmlp 2mo ago
        Great, so lets save us both time.
  - 2mo ago
jdw64 2mo ago
Does the C++ standard guarantee binary ABI? Isn't that something that compiler and library contributors handle separately? So I think the arguments you see online are more accurate—vendors are the ones maintaining it. In other words, the ABI we rely on today isn't really part of the C+ standard—it's more like the Itnaium C++ ABI or the MSVC C++ ABI. In the end, I think the ABI stays stable because of community conventions established by compiler vendors.
- andyg_blog 2mo ago
  It doesn't, and FWIW the article says that. A meta note: ABI stability is a holy war inside the C++ community. Those against it argue that it's holding back real language evolution. Herb Sutter is even working on his own C++ offshoot that shows what could be.
  - Mond_ 2mo ago
    Afaik Herb Sutter largely stopped working on Cppfront a few months ago, and was never quite supposed to be a real thing, just a personal playground (or so Sutter claims at this point, anyway). Afaik Carbon is at this point the only attempt at a successor language that's still going?
    - pjmlp 2mo ago
      Probably, but note that Google says they are they main customer anyway, and it is an experiment above anything else. Anyone that can reach out to Rust, Go, Java, C#, whatever, should do that preferably. There are also some efforts to tame existing C++ with profiles, and replacing UB with erroneous behaviour, and that's about it.
- jcranmer 2mo ago
  No, ABI is not part of the C++ standard, although there are parts of C++ that do go "this is an ABI thing" (e.g., [[no_unique_address]]). ABI is more of OS-level thing. Most systems these days follow the SysV ABI, which is largely defined by the hardware manufacturers via the processor-specific supplements (the x86-64 one is here: https://gitlab.com/x86-psABIs/x86-64-ABI https://gitlab.com/x86-psABIs/x86-64-ABI). These largely delegate C++-specific conventions to the Itanium C++ ABI (which they likely directly reference), although the ARM ecosystem uses a somewhat different layout for the exception handling tables. Microsoft uses a different ABI for both the underlying C ABI and for the C++ compatibility layer built on top of the C ABI. One of the issues that crops up is that vendors end up needing to add extensions to the ABI for various reasons, and these extensions tend to end up being incompatible, since they're added before they've had a time to be standardized. 16-bit floats is a particular historical bugbear, as is the C23 _BitInt stuff.
mohamedkoubaa 2mo ago
C++ famously has no ABI, it is the implementations of C++ that have ABIs. The reason the situation is so bad is that it's considered explicitly out of scope of the language (it targets an abstract machine).
- leni536 2mo ago
  Neither of the ISO C and ISO C++ standards define an ABI.
  - mohamedkoubaa 2mo ago
    Right, C doesn't specify an ABI either but (many) OSes do and they provide their ABIs in C so C became the defacto ABI of a given platform.
    - leni536 2mo ago
      Yes and no. The Linux kernel barely interacts with the C calling convention ABI (vDSO), and you can interact with the OS without going through a C library. I guess the layout of some structs are exposed through some binary protocols, but most of them are not C function calls.
jeffbee 2mo ago
One nice thing to remember is for projects you control, you can dictate the ABI. There isn't a law enforcement bureau that will arrest you for passing parameters in whatever register suits you. You just have to imagine life differently than they did in the 70s when they dreamed up dynamic libraries for reasons that are largely no longer valid.
- rramadass 2mo ago
  This technique is often used for security-sensitive projects involving binary protection, obfuscation etc. Program Obfuscation via ABI Debiasing (pdf) - https://www.google.com/goto?url=CAESYgHuR6pNsi8_4x_I2WBy7lmfVBPgEtf1-hDoHQEP2SByLHqYLEspUt89DkXa0-7nf6kMvVN1oHTNRFZkqEqSFlxhPooKLwwCsGkT2ZqwAFJXAwHhrjP3Bb_Kl8JRkjng4ZKf https://www.google.com/goto?url=CAESYgHuR6pNsi8_4x_I2WBy7lmf... PS: James Coplien in his excellent Advanced C++ Programming Styles and Idioms shows many techniques one of which is actually replacing vtable entries at runtime to mimic features from more dynamical runtime languages (eg. Smalltalk).

83 points by rramadass 3mo ago

As always, read wikipedia and follow links as needed - https://en.wikipedia.org/wiki/Application_binary_interface https://en.wikipedia.org/wiki/Application_binary_interface

rramadass 3mo ago

GNU libstdc++ "Dual ABI" issue - https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_a...

jdw64 2mo ago
I understand that Move Semantics led to better implementations and that this is why things changed. I know that in this case, problems arose when transitioning from C++98 to what's commonly called Modern C++ (C++11). However, I'm not very familiar with what the specific issues were with COW. If you happen to know of any documents that describe this problem in more detail, I'd appreciate it if you could let me know. I'm sorry for always asking questions. I've looked up a few documents, but since you clearly know much more about this than I do, I'm asking to study it myself as well. If it's too much trouble, feel free not to reply.
- aw1621107 2mo ago
  The changes to std::string were made because "the [pre-C++11] definition of basic_string allows only very limited concurrent access to strings. Such limited concurrency will inhibit performance in multi-threaded applications." The proposed fix was to make "all iterator and element access operations safely concurrently executable." [0]. This involved limiting the circumstances in which iterators could be invalidated, which among other things meant that non-const operator[] could no longer invalidate iterators, which meant copy-on-write was no longer a viable implementation [1]. [0]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2668.htm https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n26... [1]: https://stackoverflow.com/q/12199710 https://stackoverflow.com/q/12199710
  - jdw64 2mo ago
    Thank you for the kind comment. !
- rramadass 2mo ago
  Asking questions is always a good thing so don't worry about it. It makes us think and check our own understanding too, so we also learn something :-) A nice succinct post by Pavel Khaipov; Copy-on-Write in Modern C++: Why the Standard Dropped It - and Why Qt Still Uses It - https://www.linkedin.com/posts/khaipov_copy-on-write-in-modern-c-why-the-standard-activity-7405201870232989696-QuI8 https://www.linkedin.com/posts/khaipov_copy-on-write-in-mode... A detailed post by Andrii Nikishaiev; Why Copy-On-Write (COW) removed in C++? - https://www.linkedin.com/pulse/why-copy-on-write-cow-removed-c-andrii-nikishaiev-ua-9kmvf https://www.linkedin.com/pulse/why-copy-on-write-cow-removed... The std document on Concurrency Modifications to Basic String - https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2534.html https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n25... Finally, C++11 tightened the specification for std::string viz. contiguous memory, some methods must have O(1) access time, when references/iterators/pointers can/cannot be invalidated etc. which doomed COW.
  - jdw64 2mo ago
    I'm sorry for always bothering you, senior. I've learned quite a lot from the books you recommended. I've especially been studying low-level topics lately, and the books you suggested have been very helpful. Have a great day.
    - rramadass 2mo ago
      I am glad that you found my comments useful. I use HN as my "blog" and generally include references to books/papers/articles as a pointer to further information on a topic. You will find links to lots of books if you search my comments for key phrases (eg. embedded, book etc.). One thing i try to do is point people to lesser-known but very good and advanced books. I generally find that most HN book suggestions refer only to popular books most of which are just so-so while many great books remain unknown which is a shame. For example how many people know of Operating Systems in Depth by Thomas Doeppner? It has lots of helpful illustrations, compares concepts in both Linux/Windows implementations and is an all-round great book. I really understood the nuances of signal handling between kernel/user modes when i read it here.

Dwedit 2mo ago

C++ ABIs are why Win32 settled on things like COM to do cross-module passing of objects. Because nobody could agree on a standard, the COM standard enforced a very specific calling convention, and a very specific vtable layout for COM objects.

pjmlp 2mo ago
Not only, there is also XPC and IO/DriverKit on Apple, Android IPC, D-BUS, FIDL on Fucshia,.... Also C ABI also does not exist, people keep mistaking the ABI of their favourite C compiler with the OS ABI, which only overlap if the OS was written in C to start with. For example on mainframes and micros, naturally not written in C, it is either a bytecode based ABI like TIMI on IBM i, or language environments like on z/OS, ClearPath MCP, OS 2200 and so forth. And as a reminder, from a famous WG14 and WG21 member, and former Rust contributor, "To Save C, We Must Save ABI" https://thephd.dev/to-save-c-we-must-save-abi-fixing-c-function-abi https://thephd.dev/to-save-c-we-must-save-abi-fixing-c-funct...
- bheadmaster 2mo ago
  System V ABI is pretty standardized.
  - pjmlp 2mo ago
    UNIX, the reason why C exists.
    - bheadmaster 2mo ago
      Yeah, I'm not interested in decoding what you meant by your aphorism. Either say what you mean or don't.
      - pjmlp 2mo ago
        Great, so lets save us both time.
- 2mo ago

jdw64 2mo ago

Does the C++ standard guarantee binary ABI? Isn't that something that compiler and library contributors handle separately? So I think the arguments you see online are more accurate—vendors are the ones maintaining it. In other words, the ABI we rely on today isn't really part of the C+ standard—it's more like the Itnaium C++ ABI or the MSVC C++ ABI. In the end, I think the ABI stays stable because of community conventions established by compiler vendors.

andyg_blog 2mo ago
It doesn't, and FWIW the article says that. A meta note: ABI stability is a holy war inside the C++ community. Those against it argue that it's holding back real language evolution. Herb Sutter is even working on his own C++ offshoot that shows what could be.
- Mond_ 2mo ago
  Afaik Herb Sutter largely stopped working on Cppfront a few months ago, and was never quite supposed to be a real thing, just a personal playground (or so Sutter claims at this point, anyway). Afaik Carbon is at this point the only attempt at a successor language that's still going?
  - pjmlp 2mo ago
    Probably, but note that Google says they are they main customer anyway, and it is an experiment above anything else. Anyone that can reach out to Rust, Go, Java, C#, whatever, should do that preferably. There are also some efforts to tame existing C++ with profiles, and replacing UB with erroneous behaviour, and that's about it.
jcranmer 2mo ago
No, ABI is not part of the C++ standard, although there are parts of C++ that do go "this is an ABI thing" (e.g., [[no_unique_address]]). ABI is more of OS-level thing. Most systems these days follow the SysV ABI, which is largely defined by the hardware manufacturers via the processor-specific supplements (the x86-64 one is here: https://gitlab.com/x86-psABIs/x86-64-ABI https://gitlab.com/x86-psABIs/x86-64-ABI). These largely delegate C++-specific conventions to the Itanium C++ ABI (which they likely directly reference), although the ARM ecosystem uses a somewhat different layout for the exception handling tables. Microsoft uses a different ABI for both the underlying C ABI and for the C++ compatibility layer built on top of the C ABI. One of the issues that crops up is that vendors end up needing to add extensions to the ABI for various reasons, and these extensions tend to end up being incompatible, since they're added before they've had a time to be standardized. 16-bit floats is a particular historical bugbear, as is the C23 _BitInt stuff.

mohamedkoubaa 2mo ago

C++ famously has no ABI, it is the implementations of C++ that have ABIs. The reason the situation is so bad is that it's considered explicitly out of scope of the language (it targets an abstract machine).

leni536 2mo ago
Neither of the ISO C and ISO C++ standards define an ABI.
- mohamedkoubaa 2mo ago
  Right, C doesn't specify an ABI either but (many) OSes do and they provide their ABIs in C so C became the defacto ABI of a given platform.
  - leni536 2mo ago
    Yes and no. The Linux kernel barely interacts with the C calling convention ABI (vDSO), and you can interact with the OS without going through a C library. I guess the layout of some structs are exposed through some binary protocols, but most of them are not C function calls.

jeffbee 2mo ago

One nice thing to remember is for projects you control, you can dictate the ABI. There isn't a law enforcement bureau that will arrest you for passing parameters in whatever register suits you. You just have to imagine life differently than they did in the 70s when they dreamed up dynamic libraries for reasons that are largely no longer valid.

rramadass 2mo ago
This technique is often used for security-sensitive projects involving binary protection, obfuscation etc. Program Obfuscation via ABI Debiasing (pdf) - https://www.google.com/goto?url=CAESYgHuR6pNsi8_4x_I2WBy7lmfVBPgEtf1-hDoHQEP2SByLHqYLEspUt89DkXa0-7nf6kMvVN1oHTNRFZkqEqSFlxhPooKLwwCsGkT2ZqwAFJXAwHhrjP3Bb_Kl8JRkjng4ZKf https://www.google.com/goto?url=CAESYgHuR6pNsi8_4x_I2WBy7lmf... PS: James Coplien in his excellent Advanced C++ Programming Styles and Idioms shows many techniques one of which is actually replacing vtable entries at runtime to mimic features from more dynamical runtime languages (eg. Smalltalk).