4 ms·

I've seen a similar pattern implemented in Rust already, presenting a safe interface around relatively little unsafe code. For example, the Bevy game engine's

by Rusky 5y ago

I've seen a similar pattern implemented in Rust already, presenting a safe interface around relatively little unsafe code. For example, the Bevy game engine's entity system works like a small in-memory database, and also replaces ownership (games also have lots of mutability and lifetimes that aren't clear at compile time) with careful dynamic scheduling. Users of this system just request a particular kind of access (mutable or immutable) to some set of entities, and the engine schedules them to avoid conflicts. I'm not that familiar with databases so maybe I'm just totally missing what you're talking about, but from what it sounds like this is not something that would require large portions of a Rust program to be unsafe. Overall I think the idea that Rust relies on knowing specific mutability and lifetimes at compile time is a bit of a misunderstanding of how the borrow checker works. It's rather a sort of glue that describes relationships between APIs, and when you are coming up with a custom approach to ensure safety it is still a useful language you can speak on the boundaries.

jandrewrogers 5y ago
A tacit assumption of Rust's model is that code is running on top of a transparent virtual memory system e.g. what Linux provides by default. Game engines, and in-memory databases, typically run on top of the OS virtual memory implementation. A database engine that bypasses or disables the OS virtual memory system, which many do, breaks this assumption. This is common architecture in any performance-sensitive application that is I/O intensive, but databases are the most extreme example. Almost every object in a database engine, whether transient or persistent, lives outside the virtual memory model provided by the OS. Because it is in user space, this is part of the database code and no longer transparent to the compiler. To a compiler, these objects have no fixed address, have an ambiguous number of mutable references at any point in time, and may have no address at all (e.g. if moved to storage). This is safe, and you can write abstractions in C++ that make this transparent to the developer, but the compiler can still see it and Rust does not like what it sees. Performance is not the only reason to do this. The OS implementation is tightly coupled to the silicon implementation of virtual memory. Very large storage volumes can exceed the limits of what the hardware can support, but user space software implementations have few practical limits on scale.
- pkolaczk 5y ago
  > A tacit assumption of Rust's model is that code is running on top of a transparent virtual memory system e.g. what Linux provides by default. Where did you get that from? There is nothing more in Rust than there is in C++ that relies on virtual memory subsystem. Rust is perfectly able to compile for platforms with no OS at all. > This is safe, and you can write abstractions in C++ that make this transparent to the developer, but the compiler can still see it and Rust does not like what it sees. Rust can do the same abstractions as C++ can. This is pretty much a standard for low-level crates, which come with some unsafe code, wrapped in higher-level, easy and safe to use abstractions.
  - jandrewrogers 5y ago
    You seem to be missing a critical point. As a consequence of implementing some of these kernel functions in user space, it is not possible to determine object lifetimes or track mutable references at compile-time. Any object created in memory that bypasses the kernel mechanisms is unsafe as far as Rust is concerned. In a database, the vast majority of your objects are constructed on this type of memory. You can implement this in Rust, but essentially the entire database kernel will be "unsafe". At which point, why bother with Rust? C++ is significantly more expressive at writing this type of code, and it is not trivial even in C++.

by Rusky 5y ago

jandrewrogers 5y ago

A tacit assumption of Rust's model is that code is running on top of a transparent virtual memory system e.g. what Linux provides by default. Game engines, and in-memory databases, typically run on top of the OS virtual memory implementation. A database engine that bypasses or disables the OS virtual memory system, which many do, breaks this assumption. This is common architecture in any performance-sensitive application that is I/O intensive, but databases are the most extreme example. Almost every object in a database engine, whether transient or persistent, lives outside the virtual memory model provided by the OS. Because it is in user space, this is part of the database code and no longer transparent to the compiler. To a compiler, these objects have no fixed address, have an ambiguous number of mutable references at any point in time, and may have no address at all (e.g. if moved to storage). This is safe, and you can write abstractions in C++ that make this transparent to the developer, but the compiler can still see it and Rust does not like what it sees. Performance is not the only reason to do this. The OS implementation is tightly coupled to the silicon implementation of virtual memory. Very large storage volumes can exceed the limits of what the hardware can support, but user space software implementations have few practical limits on scale.

pkolaczk 5y ago
> A tacit assumption of Rust's model is that code is running on top of a transparent virtual memory system e.g. what Linux provides by default. Where did you get that from? There is nothing more in Rust than there is in C++ that relies on virtual memory subsystem. Rust is perfectly able to compile for platforms with no OS at all. > This is safe, and you can write abstractions in C++ that make this transparent to the developer, but the compiler can still see it and Rust does not like what it sees. Rust can do the same abstractions as C++ can. This is pretty much a standard for low-level crates, which come with some unsafe code, wrapped in higher-level, easy and safe to use abstractions.
- jandrewrogers 5y ago
  You seem to be missing a critical point. As a consequence of implementing some of these kernel functions in user space, it is not possible to determine object lifetimes or track mutable references at compile-time. Any object created in memory that bypasses the kernel mechanisms is unsafe as far as Rust is concerned. In a database, the vast majority of your objects are constructed on this type of memory. You can implement this in Rust, but essentially the entire database kernel will be "unsafe". At which point, why bother with Rust? C++ is significantly more expressive at writing this type of code, and it is not trivial even in C++.