7 ms·

A great read! Might be worth pointing out: leptos, a Rust WASM UI library, uses the "store an index (or ID) into some central data structure" idea extensively.

A great read! Might be worth pointing out: leptos, a Rust WASM UI library, uses the "store an index (or ID) into some central data structure" idea extensively. Sycamore, another Rust WASM UI library, has been using arena allocation, but will also be transitioning into the leptos approach in coming versions. (I believe) this is partly because the borrow checker's enforcement of a tree-like data structures / ownership doesn't always work well with the JS event loop.

aatd86 3y ago
Interesting. So basically the index storage is a way to create a reference to an object that is not compiler checked?
- davidatbu 3y ago
  I would say that it's a way to trade compile-time checks of reference validity for runtime-checks of reference validity.
danielheath 3y ago
I feel like the "Store an index to get around the borrow checker" largely moves the problem - now you have A) array-bounds-checking bugs, and B) either "list was re-ordered, you now have the wrong item" or "can't compact the list, it just keeps growing even when old items are no longer used" bugs. Sure, you no longer have to satisfy the borrow checker - but it was a tool to help you catch bugs, and you've cleverly avoided getting that help.
- davidatbu 3y ago
  > array-bounds-checking bugs. Rust tries very very hard to avoid bounds-checking bugs, because it's central to the safety promise, maybe I'm misunderstanding you? > either "list was re-ordered, you now have the wrong item" ... "can't compact the list, it just keeps growing even when old items are no longer used" bugs. I have no direct experience implementing this pattern, but it's good to know that leptos (and others) tend to use one well-tested library that addresses these concerns[1]. > Sure, you no longer have to satisfy the borrow checker - but it was a tool to help you catch bugs, and you've cleverly avoided getting that help. My impression is that one can perhaps claim "one traded compile-time checks of safety for runtime _panics_ when unsafe things are about to happen", and not that using this pattern exposes one to the same issues that the borrow checker avoids. [1] https://docs.rs/slotmap/latest/slotmap/index.html#why-not-index-a-vec-or-use-slab-stable-vec-etc https://docs.rs/slotmap/latest/slotmap/index.html#why-not-in...
  - danielheath 3y ago
    Yes, the failure here is a panic, typically a crash (of at least the affected subsystem), not memory corruption. Slotmap avoids this via unsafe + implements its own handle system, which is probably the right approach. The borrow checker avoids many other issues in addition to memory safety.
OskarS 3y ago
I’ve always thought this is a pretty big blindspot in Rust. Having a raw pointer and having an index into an array are essentially equivalent, and doing this in Rust is a way to turn off the borrow checker for this particular object. The index is bounds-checked, sure, but lifetime issues like “double free” or “use after free” are every bit as present with indexes. For instance, if you have an index in one place, but it’s deleted and/or reinitialized to some other place, you are now holding on to an object in an undefined state. Using it will cause the same issues “use-after-free” does in C. Best case scenario in that cases is that it crashes, but it’s just as likely your program will just silently be wrong. I’m not saying not to do it, I’m just saying this is something Rust developers needs to be aware of: storing indexes like that removes a whole bunch of safety guarantees the borrow checker checks for you, without any code marked “unsafe”.
- hyperman1 3y ago
  I'd say this is a case of the 'Inner platform' effect (Using a tool to build a poor replica of the tool) . Pointers are nothing more than indexes in the memory array. It turns out keeping track of what index does what is so hard, rust has a borrow checker to help with that. But sometimes it's annoying, so we create a workaround, and if we go too far with that workaround, we forget why we had a borrow checker to begin with. Notice how the array technique has parallels to memory: You can have multiple arrays of different types a.k.a segmented memory. You can mark unused array objects, a.k.a. overwrite free'd memory with 0xDEADBEEF.
  - intelVISA 3y ago
    It's all memory in the end. Hide from it, fear it, run from it? It'll only cause you to develop maladative coping patterns like writing Go instead of beautiful C. Instead, attain Nirvana, open emacs and embrace Fearless Development.
  - zokier 3y ago
    > Pointers are nothing more than indexes in the memory array. Pointers in C point to objects and not memory locations afaik, which has subtle semantic implications. See the whole discussion around pointer provenance.
pavlov 3y ago
> “store an index (or ID) into some central data structure” Sounds like OpenGL object management. Shudder. This is not a step forward.

aatd86 3y ago

Interesting. So basically the index storage is a way to create a reference to an object that is not compiler checked?

davidatbu 3y ago
I would say that it's a way to trade compile-time checks of reference validity for runtime-checks of reference validity.

danielheath 3y ago

I feel like the "Store an index to get around the borrow checker" largely moves the problem - now you have A) array-bounds-checking bugs, and B) either "list was re-ordered, you now have the wrong item" or "can't compact the list, it just keeps growing even when old items are no longer used" bugs. Sure, you no longer have to satisfy the borrow checker - but it was a tool to help you catch bugs, and you've cleverly avoided getting that help.

davidatbu 3y ago
> array-bounds-checking bugs. Rust tries very very hard to avoid bounds-checking bugs, because it's central to the safety promise, maybe I'm misunderstanding you? > either "list was re-ordered, you now have the wrong item" ... "can't compact the list, it just keeps growing even when old items are no longer used" bugs. I have no direct experience implementing this pattern, but it's good to know that leptos (and others) tend to use one well-tested library that addresses these concerns[1]. > Sure, you no longer have to satisfy the borrow checker - but it was a tool to help you catch bugs, and you've cleverly avoided getting that help. My impression is that one can perhaps claim "one traded compile-time checks of safety for runtime _panics_ when unsafe things are about to happen", and not that using this pattern exposes one to the same issues that the borrow checker avoids. [1] https://docs.rs/slotmap/latest/slotmap/index.html#why-not-index-a-vec-or-use-slab-stable-vec-etc https://docs.rs/slotmap/latest/slotmap/index.html#why-not-in...
- danielheath 3y ago
  Yes, the failure here is a panic, typically a crash (of at least the affected subsystem), not memory corruption. Slotmap avoids this via unsafe + implements its own handle system, which is probably the right approach. The borrow checker avoids many other issues in addition to memory safety.

OskarS 3y ago

I’ve always thought this is a pretty big blindspot in Rust. Having a raw pointer and having an index into an array are essentially equivalent, and doing this in Rust is a way to turn off the borrow checker for this particular object. The index is bounds-checked, sure, but lifetime issues like “double free” or “use after free” are every bit as present with indexes. For instance, if you have an index in one place, but it’s deleted and/or reinitialized to some other place, you are now holding on to an object in an undefined state. Using it will cause the same issues “use-after-free” does in C. Best case scenario in that cases is that it crashes, but it’s just as likely your program will just silently be wrong. I’m not saying not to do it, I’m just saying this is something Rust developers needs to be aware of: storing indexes like that removes a whole bunch of safety guarantees the borrow checker checks for you, without any code marked “unsafe”.

hyperman1 3y ago
I'd say this is a case of the 'Inner platform' effect (Using a tool to build a poor replica of the tool) . Pointers are nothing more than indexes in the memory array. It turns out keeping track of what index does what is so hard, rust has a borrow checker to help with that. But sometimes it's annoying, so we create a workaround, and if we go too far with that workaround, we forget why we had a borrow checker to begin with. Notice how the array technique has parallels to memory: You can have multiple arrays of different types a.k.a segmented memory. You can mark unused array objects, a.k.a. overwrite free'd memory with 0xDEADBEEF.
- intelVISA 3y ago
  It's all memory in the end. Hide from it, fear it, run from it? It'll only cause you to develop maladative coping patterns like writing Go instead of beautiful C. Instead, attain Nirvana, open emacs and embrace Fearless Development.
- zokier 3y ago
  > Pointers are nothing more than indexes in the memory array. Pointers in C point to objects and not memory locations afaik, which has subtle semantic implications. See the whole discussion around pointer provenance.

pavlov 3y ago

> “store an index (or ID) into some central data structure” Sounds like OpenGL object management. Shudder. This is not a step forward.