3 ms·

All high-performance database engines do full kernel bypass -- Linux has standard APIs for this. Databases designed this way are much higher performance, which

All high-performance database engines do full kernel bypass -- Linux has standard APIs for this. Databases designed this way are much higher performance, which is people do it. It may not be obvious what this has to do with Rust. The downside of kernel bypass is that the kernel is no longer transparently hiding the relationship between storage, memory, and related low-level hardware mechanics. A consequence of this is that the mutability and lifetime of most objects cannot be evaluated at compile-time, which Rust relies on for its safety model. The hardware can hold mutable references to memory that are not represented in software and which don't understand your object model. How do you manage lifetimes when references to objects are not visible in the code? When an object does not have a fixed address? And so on. This is not trivial even in C++ but the language does allow you to write transparent abstractions that hide this behind something that behaves like a reference without complaining. There are proven models that guarantee safety under these constraints, widely used in database engines. Instead of ownership semantics, they are based on dynamic scheduling semantics. The execution scheduler can dynamically detect unsafe (or other) situations and rewrite the execution schedule to guarantee safety and forward progress without violating other invariants. This is why, as a simple example, some databases never seem to produce deadlocks -- deadlock situations still occur but they are dynamically detected before they occur and are resolved transparently by editing the lock and execution graph. Some major classes of database architecture optimization don't work in garbage collected languages. For this reason, you never see state-of-the-art database engines written in e.g. Java.

Rusky 5y ago
I've seen a similar pattern implemented in Rust already, presenting a safe interface around relatively little unsafe code. For example, the Bevy game engine's entity system works like a small in-memory database, and also replaces ownership (games also have lots of mutability and lifetimes that aren't clear at compile time) with careful dynamic scheduling. Users of this system just request a particular kind of access (mutable or immutable) to some set of entities, and the engine schedules them to avoid conflicts. I'm not that familiar with databases so maybe I'm just totally missing what you're talking about, but from what it sounds like this is not something that would require large portions of a Rust program to be unsafe. Overall I think the idea that Rust relies on knowing specific mutability and lifetimes at compile time is a bit of a misunderstanding of how the borrow checker works. It's rather a sort of glue that describes relationships between APIs, and when you are coming up with a custom approach to ensure safety it is still a useful language you can speak on the boundaries.
- jandrewrogers 5y ago
  A tacit assumption of Rust's model is that code is running on top of a transparent virtual memory system e.g. what Linux provides by default. Game engines, and in-memory databases, typically run on top of the OS virtual memory implementation. A database engine that bypasses or disables the OS virtual memory system, which many do, breaks this assumption. This is common architecture in any performance-sensitive application that is I/O intensive, but databases are the most extreme example. Almost every object in a database engine, whether transient or persistent, lives outside the virtual memory model provided by the OS. Because it is in user space, this is part of the database code and no longer transparent to the compiler. To a compiler, these objects have no fixed address, have an ambiguous number of mutable references at any point in time, and may have no address at all (e.g. if moved to storage). This is safe, and you can write abstractions in C++ that make this transparent to the developer, but the compiler can still see it and Rust does not like what it sees. Performance is not the only reason to do this. The OS implementation is tightly coupled to the silicon implementation of virtual memory. Very large storage volumes can exceed the limits of what the hardware can support, but user space software implementations have few practical limits on scale.
  - pkolaczk 5y ago
    > A tacit assumption of Rust's model is that code is running on top of a transparent virtual memory system e.g. what Linux provides by default. Where did you get that from? There is nothing more in Rust than there is in C++ that relies on virtual memory subsystem. Rust is perfectly able to compile for platforms with no OS at all. > This is safe, and you can write abstractions in C++ that make this transparent to the developer, but the compiler can still see it and Rust does not like what it sees. Rust can do the same abstractions as C++ can. This is pretty much a standard for low-level crates, which come with some unsafe code, wrapped in higher-level, easy and safe to use abstractions.
    - jandrewrogers 5y ago
      You seem to be missing a critical point. As a consequence of implementing some of these kernel functions in user space, it is not possible to determine object lifetimes or track mutable references at compile-time. Any object created in memory that bypasses the kernel mechanisms is unsafe as far as Rust is concerned. In a database, the vast majority of your objects are constructed on this type of memory. You can implement this in Rust, but essentially the entire database kernel will be "unsafe". At which point, why bother with Rust? C++ is significantly more expressive at writing this type of code, and it is not trivial even in C++.

Rusky 5y ago

I've seen a similar pattern implemented in Rust already, presenting a safe interface around relatively little unsafe code. For example, the Bevy game engine's entity system works like a small in-memory database, and also replaces ownership (games also have lots of mutability and lifetimes that aren't clear at compile time) with careful dynamic scheduling. Users of this system just request a particular kind of access (mutable or immutable) to some set of entities, and the engine schedules them to avoid conflicts. I'm not that familiar with databases so maybe I'm just totally missing what you're talking about, but from what it sounds like this is not something that would require large portions of a Rust program to be unsafe. Overall I think the idea that Rust relies on knowing specific mutability and lifetimes at compile time is a bit of a misunderstanding of how the borrow checker works. It's rather a sort of glue that describes relationships between APIs, and when you are coming up with a custom approach to ensure safety it is still a useful language you can speak on the boundaries.

jandrewrogers 5y ago
A tacit assumption of Rust's model is that code is running on top of a transparent virtual memory system e.g. what Linux provides by default. Game engines, and in-memory databases, typically run on top of the OS virtual memory implementation. A database engine that bypasses or disables the OS virtual memory system, which many do, breaks this assumption. This is common architecture in any performance-sensitive application that is I/O intensive, but databases are the most extreme example. Almost every object in a database engine, whether transient or persistent, lives outside the virtual memory model provided by the OS. Because it is in user space, this is part of the database code and no longer transparent to the compiler. To a compiler, these objects have no fixed address, have an ambiguous number of mutable references at any point in time, and may have no address at all (e.g. if moved to storage). This is safe, and you can write abstractions in C++ that make this transparent to the developer, but the compiler can still see it and Rust does not like what it sees. Performance is not the only reason to do this. The OS implementation is tightly coupled to the silicon implementation of virtual memory. Very large storage volumes can exceed the limits of what the hardware can support, but user space software implementations have few practical limits on scale.
- pkolaczk 5y ago
  > A tacit assumption of Rust's model is that code is running on top of a transparent virtual memory system e.g. what Linux provides by default. Where did you get that from? There is nothing more in Rust than there is in C++ that relies on virtual memory subsystem. Rust is perfectly able to compile for platforms with no OS at all. > This is safe, and you can write abstractions in C++ that make this transparent to the developer, but the compiler can still see it and Rust does not like what it sees. Rust can do the same abstractions as C++ can. This is pretty much a standard for low-level crates, which come with some unsafe code, wrapped in higher-level, easy and safe to use abstractions.
  - jandrewrogers 5y ago
    You seem to be missing a critical point. As a consequence of implementing some of these kernel functions in user space, it is not possible to determine object lifetimes or track mutable references at compile-time. Any object created in memory that bypasses the kernel mechanisms is unsafe as far as Rust is concerned. In a database, the vast majority of your objects are constructed on this type of memory. You can implement this in Rust, but essentially the entire database kernel will be "unsafe". At which point, why bother with Rust? C++ is significantly more expressive at writing this type of code, and it is not trivial even in C++.