5 ms·

I always love your take even if I don't agree, SpaceCurve was a phenomenal system, one of the most pragmatic, high performance, easy to use MPP database systems

I always love your take even if I don't agree, SpaceCurve was a phenomenal system, one of the most pragmatic, high performance, easy to use MPP database systems I have ever used. We never met btw, was just a user. But I think you are wrong about Rust not having the right machinery for making high performance dbs. Two examples are Noria and Materialize https://github.com/mit-pdos/noria https://github.com/mit-pdos/noria and it its 50k lines, in the immediate codebase, there are 40 uses of unsafe. In Materialize's 125k of Rust, there are 76 direct uses of unsafe. https://github.com/MaterializeInc/materialize https://github.com/MaterializeInc/materialize

jandrewrogers 5y ago
This kind of reinforces my point though: neither Materialize nor Noria are high-performance database kernels, and they don't need to implement the high-performance I/O structures database kernels have that give Rust problems. Rust works great for server software generally, database kernels are a very specific outlier. It is common in recent database kernel architectures to implement an entire virtual memory system in user space. This enables some great throughput optimizations. Almost all of your runtime objects are instantiated on top of this and, importantly, entities outside your process/code can write into your address space -- an invisible implicit reference. As a side effect, there are few memory references in the way Rust understands it, those outside entities don't understand or respect the object model, and some aspects of ownership, mutability, and lifetime can only be resolved at runtime and with some interesting edge cases. The model is elegant and safe, it just doesn't provide a coherent graph of classic memory references that Rust can latch onto at compile-time for safety analysis. All good wholesome fun.
- nerpderp82 5y ago
  Not sure proves your point, but maybe doesn't disprove your point strongly enough. I am not qualified to argue from experience about how Rust is ideally suited in the ways you think it is not. But from everything I have seen, it can do a whole lot of what C++ is also good at. Rust safety is not all or nothing and a codebase could definitely prioritize ergonomics over correctness. Two things that I saw in the last couple weeks that might start to sway you. https://github.com/sslab-gatech/Rudra#readme https://github.com/sslab-gatech/Rudra#readme GhostCell: Separating Permissions from Data in Rust https://www.youtube.com/watch?v=jIbubw86p0M https://www.youtube.com/watch?v=jIbubw86p0M Even unsafe Rust can be as ergonomic as C++. But that unsafety can be mediated, moderated and controlled.

jandrewrogers 5y ago

This kind of reinforces my point though: neither Materialize nor Noria are high-performance database kernels, and they don't need to implement the high-performance I/O structures database kernels have that give Rust problems. Rust works great for server software generally, database kernels are a very specific outlier. It is common in recent database kernel architectures to implement an entire virtual memory system in user space. This enables some great throughput optimizations. Almost all of your runtime objects are instantiated on top of this and, importantly, entities outside your process/code can write into your address space -- an invisible implicit reference. As a side effect, there are few memory references in the way Rust understands it, those outside entities don't understand or respect the object model, and some aspects of ownership, mutability, and lifetime can only be resolved at runtime and with some interesting edge cases. The model is elegant and safe, it just doesn't provide a coherent graph of classic memory references that Rust can latch onto at compile-time for safety analysis. All good wholesome fun.

nerpderp82 5y ago
Not sure proves your point, but maybe doesn't disprove your point strongly enough. I am not qualified to argue from experience about how Rust is ideally suited in the ways you think it is not. But from everything I have seen, it can do a whole lot of what C++ is also good at. Rust safety is not all or nothing and a codebase could definitely prioritize ergonomics over correctness. Two things that I saw in the last couple weeks that might start to sway you. https://github.com/sslab-gatech/Rudra#readme https://github.com/sslab-gatech/Rudra#readme GhostCell: Separating Permissions from Data in Rust https://www.youtube.com/watch?v=jIbubw86p0M https://www.youtube.com/watch?v=jIbubw86p0M Even unsafe Rust can be as ergonomic as C++. But that unsafety can be mediated, moderated and controlled.