10 ms·

A deep dive into Rust and C memory interoperability

154 points by hyperbrainer 1y ago

sesm 1y ago
Section named "The Interview Question That Started Everything" doesn't contain the interview question.
- hyperbrainer 1y ago
  That's the first thing on the page. > Interviewer: “What happens if you allocate memory with C’s malloc and try to free it with Rust’s dealloc, if you get a pointer to the memory from C?” > Me: “If we do it via FFI then there’s a possibility the program may continue working (because the underlying structs share the same memory layout? right? …right?)”
  - sesm 1y ago
    That's fair. Personally, I've skipped that entire pre-section thinking it's a long quote from some book.
- PoignardAzur 1y ago
  It is, but yeah, the entire article's formatting is pretty weird.
  - jeroenhd 1y ago
    The entire blog post feels formatted like AI output to me. Repeated checklists with restated points, tables and full blocks of code spread across the page in a very specific way. I don't know if the author used AI to write this, but if they didn't, this is the person AI agents decided to copy the writing style of. Edit: Reddit thread somewhere in the comments here to a post from the author pretty much confirmed my suspicions, this article is heavily AI generated and plain wrong in several cases. A good reminder not to use AI slop to learn new topics, because LLMs bullshit half the time and you need to know what you're doing to spot the lies.
phkahler 1y ago
Something I'd like to know for mixing Rust and C. I know it's possible to access a struct from both C and Rust code and have seen examples. But those all use accessor functions on the Rust side rather than accessing the members directly. Is it possible to define a structure in one of the languages and then via some wrapper or definitions be able to access it idiomatically in the other language? Can you point to some blog or documentation explaining how?
- Arnavion 1y ago
  I don't know what examples you've been seeing. The interop structs are just regular Rust structs with the `#[repr(C)]` attribute applied to them, to ensure that the Rust compiler lays the struct out exactly as the C compiler for that target ABI would. Rust code can access their fields just fine. There's no strict need for accessor functions.
  - stouset 1y ago
    And vice versa. Rust code and C code can both operate on each other’s structs natively. `#[repr(C)]` instructs the compiler to lay the struct out exactly according to C’s rules: order, alignment, padding, size, etc. Without this, the compiler is allowed a lot more freedom when laying out a struct.
- pmalynin 1y ago
  Like, repr(C)? https://doc.rust-lang.org/nomicon/other-reprs.html https://doc.rust-lang.org/nomicon/other-reprs.html
- GrantMoyer 1y ago
  Rust bindgen[1] will automatically generate native Rust stucts (and unions) from C headers where possible. Note that c_int, c_char, etc. are just aliases for the corresponding native Rust types. However, not all C constructs have idomatic Rust equivalents. For example, bitfields don't exist in Rust, and unlike Rust enums, C enums can have any value of the underlying type. And for ABI reasons, it's very commom in C APIs to use a pointer to an opaque type paired with what are effectively accessor function and methods, so mapping them to accessors and methods on a "Handle" type in Rust often is the most idomatic Rust representation of the C interface. [1]: https://github.com/rust-lang/rust-bindgen https://github.com/rust-lang/rust-bindgen
7e 1y ago
Allocating memory with C and freeing it with Rust is silly. If you want to free a C-allocated pointer in Rust, just have Rust call back in to C. Expecting that allocators work identically in both runtimes is unreasonable and borderline insane. Heck, I wouldn't expect allocators to work the same even across releases of libc from the same vendor (or across releases of Rust's std).
- rectang 1y ago
  I don't agree with your contemptuous framing. It's incorrect, and per the post's author, "dangerous" — but depending on your background it's not "silly" or "borderline insane". It's just naive, and writing a slab allocator as an exercise or making honest explorations like in this blog post will help cure the naivete.
  - 7e 1y ago
    It’s undefined behavior. It will never be stable. Investigating every permutation of zero-utility undefined behavior in the universe is borderline insane. Will the author next investigate exactly how a 2002 Fiat becomes inoperable after a head on collision with a 2025 Volkswagen? These are all deep dives into infinite chaos.
    - rectang 1y ago
      I agree that we shouldn't mix allocators, and so does the post's author. Can you put your technical arguments into terms which aren't so dismissive of people's honest efforts to learn? We ought to be all on the same page. You could affirm and refine the post's conclusions and bring us together, rather than ridicule someone for ever entertaining a notion you consider "insane". Here's what I got when I asked ChatGPT to rewrite your first comment to be as constructive as possible: "Totally agree — relying on C and Rust to interoperate at the allocator level is risky at best. Allocating in C and freeing in Rust (or vice versa) assumes a level of compatibility that just isn’t guaranteed. Even within a single ecosystem, allocator behavior can change across versions — whether it’s different versions of libc or updates to Rust’s std. So expecting consistent behavior across language boundaries is, at the very least, unreliable. If you need to free a C-allocated pointer, the safest and cleanest approach is to call back into C to do it." That's not a drop-in substitute for your original comment and "totally agree" is over-the-top cloying in the usual ChatGPT obsequious way, but I still think it's helpful in suggesting alternative framings.
eatonphil 1y ago
One of the areas I wonder about this a lot is when integrating Rust code into Postgres which has its own allocator system. Mostly right now when we need to have complex data structures (non-Postgres data structures) that must live outside of the lexical scope we put them somewhere global and return a handle to the C code to reference the object. But with the upcoming support for passing an allocator to any data structure (in the Rust standard library anyway) I think this gets a lot easier?
- steveklabnik 1y ago
  I’m not sure what those two things have to do with each other, though I did just wake up. The only thing the new allocator stuff would give you is the ability to allocate a standard library data structure with the Postgres allocator. Scoping and handles and such wouldn’t change, and using your own data structures wouldn’t change. It’s also very possible I’m missing something!
  - eatonphil 1y ago
    > The only thing the new allocator stuff would give you is the ability to allocate a standard library data structure with the Postgres allocator. Yeah no this is basically all I'm saying. I'm excited for this.
    - steveklabnik 1y ago
      Ah yeah, well it's gonna be a good feature for sure when it ships!
- Arnavion 1y ago
  >But with the upcoming support for passing an allocator to any data structure (in the Rust standard library anyway) I think this gets a lot easier? Yes and no. Even within libstd, some things require A=GlobalAlloc, eg `std::io::Read::read_to_end(&mut Vec<u8>)` will only accept Vec<u8, GlobalAlloc>. It cannot be changed to work with Vec<u8, A> because that change would make it not dyn-compatible (nee "object-safe"). And as you said it will cut you off from much of the third-party crates ecosystem that also assumes A=GlobalAlloc. But if the subset of libstd you need supports A=!GlobalAlloc then yes it's helpful.
ryanf 1y ago
This article looked interesting, but I bounced off it because the author appears to have made heavy use of an LLM to generate the text. How can I trust that the content is worth reading if a person didn't care enough to write it themselves?
- TechDebtDevin 1y ago
  Do you see Emojis in tables/code now and assume the person is using an llm? I dont really see it.
  - ryanf 1y ago
    Maybe I'm too paranoid! If it's not LLM then I don't think it's a very well-organized post though. In addition to the emoji, things that jumped out at me were the pervasive use of bullet lists with bold labels and some specific text choices like > Note: The bash scripts in tools/ dynamically generate Rust code for specialized analysis. This keeps the main codebase clean while allowing complex experiments. But I did just edit my post to walk it back slightly.
    - skydhash 1y ago
      Not TFA’s author As a non-native English speaker, 90% of my vocabulary come from technical books and SF and Fantasy novels. And due to an education done in French, I tend to prefer slightly complicated sentences forms. If someone uses LLM to give their posts clarity or for spellchecking, I would aplaud them. What I don’t agree with, LLM use or no, is meandering and inconsistency.
  - OmarAssadi 1y ago
    Personally, it is one of the flags, yeah. It's been a while since I've tried ChatGPT or some of the others, but the structure and particular usage felt a lot like what I'd have gotten out of deepseek. It's not a binary thing, of course, but it's definitely an LLM smell, IMO.
  - mvieira38 1y ago
    I mean, are we supposed not to? This doesn't read like a blog at all, it even has the dreaded "Key Takeaways" end section... The content is good and seems genuinely researched, but the text looks "AI enhanced", that's all
veber-alex 1y ago
The reason you are not seeing crashes when allocating with Rust and freeing with C (or vice versa) is that by default Rust also uses the libc allocator. https://stdrs.dev/nightly/x86_64-unknown-linux-gnu/src/std/sys/unix/alloc.rs.html#6-53 https://stdrs.dev/nightly/x86_64-unknown-linux-gnu/src/std/s...
- CupricTea 1y ago
  It's funny. When I first tried Rust in 2018 they were still statically linking jemalloc into every binary rustc compiled by default, and that alone very much put me off of the language for a while. Apparently they did away with jemalloc in favor of the system allocator that same year but nonetheless when I came back to it years later I was very happy to learn of its removal.
  - School-Cotton 1y ago
    > that alone very much put me off of the language for a while Why?
    - CupricTea 1y ago
      Jemalloc added over a megabyte to every project for only questionable gains, and it was awkward and unwieldy to remove it. While there are good reasons to use a different allocator depending on the project, Rust defaulting to this type of behavior failed a certain personal litmus test on what it wanted to be as a language in that it felt like it was fighting the system rather than integrating with it. It also does not give a good first impression at all to newcomers to see their hello world project built in release mode take up almost 2MiB of space. Today it's a much more (subjectively) tolerable 136kiB on Windows (considering that Rust std is statically linked).
      - deleted 1y ago
        [deleted]
      - School-Cotton 1y ago
tracker1 1y ago
Interesting read... and definitely good to know base of knowledge especially if you're working in transitional or mixed codebases.
potatogotato 1y ago
[flagged]
potatogotato 1y ago
[flagged]
jokoon 1y ago
Any insight on the quantity of paid rust job out there?
Tony_Delco 1y ago
Fantastic opening line (“Memory oppresses me.”). If this article was written by an AI, it’s the best AI I’ve seen in months. Seriously though: I already knew the “don’t mix allocators” rule, but I really enjoyed seeing such a careful and hands-on exploration of why it’s dangerous. Thanks for sharing it.
techlatest_net 1y ago
[dead]
mwkaufma 1y ago
Lots of detail, little substance, and misleading section headers. GPT-generated red flags.
- sjmulder 1y ago
  The interjected bullet point sections seem to be entirely LLM written and don't add anything, just meaningless interruption
commandersaki 1y ago
Me: “If we do it via FFI then there’s a possibility the program may continue working (because the underlying structs share the same memory layout? right? …right?)” I didn't understand what was being said here; was he suggesting that you call libc free using FFI; which would be fine? I understand the interviewer asked about using Rust dealloc though. I think the FFI bit is confusing me.

154 points by hyperbrainer 1y ago

sesm 1y ago

Section named "The Interview Question That Started Everything" doesn't contain the interview question.

hyperbrainer 1y ago
That's the first thing on the page. > Interviewer: “What happens if you allocate memory with C’s malloc and try to free it with Rust’s dealloc, if you get a pointer to the memory from C?” > Me: “If we do it via FFI then there’s a possibility the program may continue working (because the underlying structs share the same memory layout? right? …right?)”
- sesm 1y ago
  That's fair. Personally, I've skipped that entire pre-section thinking it's a long quote from some book.
PoignardAzur 1y ago
It is, but yeah, the entire article's formatting is pretty weird.
- jeroenhd 1y ago
  The entire blog post feels formatted like AI output to me. Repeated checklists with restated points, tables and full blocks of code spread across the page in a very specific way. I don't know if the author used AI to write this, but if they didn't, this is the person AI agents decided to copy the writing style of. Edit: Reddit thread somewhere in the comments here to a post from the author pretty much confirmed my suspicions, this article is heavily AI generated and plain wrong in several cases. A good reminder not to use AI slop to learn new topics, because LLMs bullshit half the time and you need to know what you're doing to spot the lies.

phkahler 1y ago

Something I'd like to know for mixing Rust and C. I know it's possible to access a struct from both C and Rust code and have seen examples. But those all use accessor functions on the Rust side rather than accessing the members directly. Is it possible to define a structure in one of the languages and then via some wrapper or definitions be able to access it idiomatically in the other language? Can you point to some blog or documentation explaining how?

Arnavion 1y ago
I don't know what examples you've been seeing. The interop structs are just regular Rust structs with the `#[repr(C)]` attribute applied to them, to ensure that the Rust compiler lays the struct out exactly as the C compiler for that target ABI would. Rust code can access their fields just fine. There's no strict need for accessor functions.
- stouset 1y ago
  And vice versa. Rust code and C code can both operate on each other’s structs natively. `#[repr(C)]` instructs the compiler to lay the struct out exactly according to C’s rules: order, alignment, padding, size, etc. Without this, the compiler is allowed a lot more freedom when laying out a struct.
pmalynin 1y ago
Like, repr(C)? https://doc.rust-lang.org/nomicon/other-reprs.html https://doc.rust-lang.org/nomicon/other-reprs.html
GrantMoyer 1y ago
Rust bindgen[1] will automatically generate native Rust stucts (and unions) from C headers where possible. Note that c_int, c_char, etc. are just aliases for the corresponding native Rust types. However, not all C constructs have idomatic Rust equivalents. For example, bitfields don't exist in Rust, and unlike Rust enums, C enums can have any value of the underlying type. And for ABI reasons, it's very commom in C APIs to use a pointer to an opaque type paired with what are effectively accessor function and methods, so mapping them to accessors and methods on a "Handle" type in Rust often is the most idomatic Rust representation of the C interface. [1]: https://github.com/rust-lang/rust-bindgen https://github.com/rust-lang/rust-bindgen

7e 1y ago

Allocating memory with C and freeing it with Rust is silly. If you want to free a C-allocated pointer in Rust, just have Rust call back in to C. Expecting that allocators work identically in both runtimes is unreasonable and borderline insane. Heck, I wouldn't expect allocators to work the same even across releases of libc from the same vendor (or across releases of Rust's std).

rectang 1y ago
I don't agree with your contemptuous framing. It's incorrect, and per the post's author, "dangerous" — but depending on your background it's not "silly" or "borderline insane". It's just naive, and writing a slab allocator as an exercise or making honest explorations like in this blog post will help cure the naivete.
- 7e 1y ago
  It’s undefined behavior. It will never be stable. Investigating every permutation of zero-utility undefined behavior in the universe is borderline insane. Will the author next investigate exactly how a 2002 Fiat becomes inoperable after a head on collision with a 2025 Volkswagen? These are all deep dives into infinite chaos.
  - rectang 1y ago
    I agree that we shouldn't mix allocators, and so does the post's author. Can you put your technical arguments into terms which aren't so dismissive of people's honest efforts to learn? We ought to be all on the same page. You could affirm and refine the post's conclusions and bring us together, rather than ridicule someone for ever entertaining a notion you consider "insane". Here's what I got when I asked ChatGPT to rewrite your first comment to be as constructive as possible: "Totally agree — relying on C and Rust to interoperate at the allocator level is risky at best. Allocating in C and freeing in Rust (or vice versa) assumes a level of compatibility that just isn’t guaranteed. Even within a single ecosystem, allocator behavior can change across versions — whether it’s different versions of libc or updates to Rust’s std. So expecting consistent behavior across language boundaries is, at the very least, unreliable. If you need to free a C-allocated pointer, the safest and cleanest approach is to call back into C to do it." That's not a drop-in substitute for your original comment and "totally agree" is over-the-top cloying in the usual ChatGPT obsequious way, but I still think it's helpful in suggesting alternative framings.

eatonphil 1y ago

One of the areas I wonder about this a lot is when integrating Rust code into Postgres which has its own allocator system. Mostly right now when we need to have complex data structures (non-Postgres data structures) that must live outside of the lexical scope we put them somewhere global and return a handle to the C code to reference the object. But with the upcoming support for passing an allocator to any data structure (in the Rust standard library anyway) I think this gets a lot easier?

steveklabnik 1y ago
I’m not sure what those two things have to do with each other, though I did just wake up. The only thing the new allocator stuff would give you is the ability to allocate a standard library data structure with the Postgres allocator. Scoping and handles and such wouldn’t change, and using your own data structures wouldn’t change. It’s also very possible I’m missing something!
- eatonphil 1y ago
  > The only thing the new allocator stuff would give you is the ability to allocate a standard library data structure with the Postgres allocator. Yeah no this is basically all I'm saying. I'm excited for this.
  - steveklabnik 1y ago
    Ah yeah, well it's gonna be a good feature for sure when it ships!
Arnavion 1y ago
>But with the upcoming support for passing an allocator to any data structure (in the Rust standard library anyway) I think this gets a lot easier? Yes and no. Even within libstd, some things require A=GlobalAlloc, eg `std::io::Read::read_to_end(&mut Vec<u8>)` will only accept Vec<u8, GlobalAlloc>. It cannot be changed to work with Vec<u8, A> because that change would make it not dyn-compatible (nee "object-safe"). And as you said it will cut you off from much of the third-party crates ecosystem that also assumes A=GlobalAlloc. But if the subset of libstd you need supports A=!GlobalAlloc then yes it's helpful.

ryanf 1y ago

This article looked interesting, but I bounced off it because the author appears to have made heavy use of an LLM to generate the text. How can I trust that the content is worth reading if a person didn't care enough to write it themselves?

TechDebtDevin 1y ago
Do you see Emojis in tables/code now and assume the person is using an llm? I dont really see it.
- ryanf 1y ago
  Maybe I'm too paranoid! If it's not LLM then I don't think it's a very well-organized post though. In addition to the emoji, things that jumped out at me were the pervasive use of bullet lists with bold labels and some specific text choices like > Note: The bash scripts in tools/ dynamically generate Rust code for specialized analysis. This keeps the main codebase clean while allowing complex experiments. But I did just edit my post to walk it back slightly.
  - skydhash 1y ago
    Not TFA’s author As a non-native English speaker, 90% of my vocabulary come from technical books and SF and Fantasy novels. And due to an education done in French, I tend to prefer slightly complicated sentences forms. If someone uses LLM to give their posts clarity or for spellchecking, I would aplaud them. What I don’t agree with, LLM use or no, is meandering and inconsistency.
- OmarAssadi 1y ago
  Personally, it is one of the flags, yeah. It's been a while since I've tried ChatGPT or some of the others, but the structure and particular usage felt a lot like what I'd have gotten out of deepseek. It's not a binary thing, of course, but it's definitely an LLM smell, IMO.
- mvieira38 1y ago
  I mean, are we supposed not to? This doesn't read like a blog at all, it even has the dreaded "Key Takeaways" end section... The content is good and seems genuinely researched, but the text looks "AI enhanced", that's all

veber-alex 1y ago

The reason you are not seeing crashes when allocating with Rust and freeing with C (or vice versa) is that by default Rust also uses the libc allocator. https://stdrs.dev/nightly/x86_64-unknown-linux-gnu/src/std/sys/unix/alloc.rs.html#6-53 https://stdrs.dev/nightly/x86_64-unknown-linux-gnu/src/std/s...

CupricTea 1y ago
It's funny. When I first tried Rust in 2018 they were still statically linking jemalloc into every binary rustc compiled by default, and that alone very much put me off of the language for a while. Apparently they did away with jemalloc in favor of the system allocator that same year but nonetheless when I came back to it years later I was very happy to learn of its removal.
- School-Cotton 1y ago
  > that alone very much put me off of the language for a while Why?
  - CupricTea 1y ago
    Jemalloc added over a megabyte to every project for only questionable gains, and it was awkward and unwieldy to remove it. While there are good reasons to use a different allocator depending on the project, Rust defaulting to this type of behavior failed a certain personal litmus test on what it wanted to be as a language in that it felt like it was fighting the system rather than integrating with it. It also does not give a good first impression at all to newcomers to see their hello world project built in release mode take up almost 2MiB of space. Today it's a much more (subjectively) tolerable 136kiB on Windows (considering that Rust std is statically linked).
    - deleted 1y ago
      [deleted]
    - School-Cotton 1y ago

tracker1 1y ago

Interesting read... and definitely good to know base of knowledge especially if you're working in transitional or mixed codebases.

potatogotato 1y ago

[flagged]

potatogotato 1y ago

[flagged]

jokoon 1y ago

Any insight on the quantity of paid rust job out there?

Tony_Delco 1y ago

Fantastic opening line (“Memory oppresses me.”). If this article was written by an AI, it’s the best AI I’ve seen in months. Seriously though: I already knew the “don’t mix allocators” rule, but I really enjoyed seeing such a careful and hands-on exploration of why it’s dangerous. Thanks for sharing it.

techlatest_net 1y ago

[dead]

mwkaufma 1y ago

Lots of detail, little substance, and misleading section headers. GPT-generated red flags.

sjmulder 1y ago
The interjected bullet point sections seem to be entirely LLM written and don't add anything, just meaningless interruption

commandersaki 1y ago

Me: “If we do it via FFI then there’s a possibility the program may continue working (because the underlying structs share the same memory layout? right? …right?)” I didn't understand what was being said here; was he suggesting that you call libc free using FFI; which would be fine? I understand the interviewer asked about using Rust dealloc though. I think the FFI bit is confusing me.