5 ms·

These points strike me: Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those

by pm2222 1y ago

These points strike me: Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy. Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring. Rust needs to demonstrate that it can do the kinds of work that C does in SQLite without a significant speed penalty.

rstuart4133 1y ago
> Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy. This is annoying in Rust. To me array accesses aren't the most annoying, it's match{} branches that will never been invoked. There is unreachable!() for such situations, and you would hope that: if array_access_out_of_bounds { unreachable!(); } is recognised by the Rust tooling and just ignored. That's effectively the same as SQLite is doing now by not doing the check. But it isn't ignored by the tooling: unreachable!() is reported as a missed line. Then there is the test code coverage including the standard output by default, and you have to use regex's on path names to remove it.
- steveklabnik 1y ago
  A more direct translation of the sqlite strategy here is to use get_unchecked instead of [], and then you get the same behaviors. Your example does what [] does already, it’s just a more verbose way of writing the same thing. It’s not the same behavior as sqlite.
  - rstuart4133 1y ago
    > A more direct translation of the sqlite strategy here is to use get_unchecked instead of [], and then you get the same behaviors. Array access was just an example. My point was that Rust makes 100% code coverage for unit tests well neigh impossible. For those of us who like 100% test coverage, that is a major annoyance. For way I use Rust that could be fixed by the tooling simply not counting unreachable!() lines as unreached. For Sqlite, who does branch coverage testing on the compiled binary you would have to go a step further, and provide a compile time option that elides all paths that lead to unreachable!() from the binary. I recall Sqlite saying when they changed to 100% branch coverage, their bug reports dropped by a factor of 7. I hope I remember that correctly. If I do, I'm pretty sure they won't be looking at Rust until they can achieve the same outcome. They can't come close now. Replacing every array access with get_unchecked() and the consequent explosion of unsafe area's wouldn't fly with anyone I worked with. You did say to me in another thread unsafe is perfectly fine in Rust programs, but you are literally the only person holding that opinion I come across.
steveklabnik 1y ago
If the branch is never taken, and the optimizer can prove it, it will remove the check. Sometimes if it can’t actually prove it there’s ways to help it understand, or, in the almost extreme case, you do what I commented below.
- sedatk 1y ago
  Yeah I don't understand the argument. If you can't convince the compiler that that branch will never be taken, then I strongly suspect that it may be taken.
  - unclad5968 1y ago
    That's not the point. The point is that if it is never taken, you can't test it. They don't care that it inserts a conditional OP to check, they care that they can't test the conditional path.
    - sedatk 1y ago
      But, there is no conditional path when the type system can assure the compiler that there is nothing to be conditional about. Do they mean that it's impossible to be 100% sure about if there's a conditional path or not?
  - compiler-guy 1y ago
    A program can have many properties that the compiler cannot prove statically. To take a very basic case, the halting problem.
pella 1y ago
Turso: https://algora.io/challenges/turso https://algora.io/challenges/turso "Turso is rewriting SQLite in Rust ; Find a bug to win $1,000" ------ - Dec 10, 2024 : "Introducing Limbo: A complete rewrite of SQLite in Rust" https://turso.tech/blog/introducing-limbo-a-complete-rewrite-of-sqlite-in-rust https://turso.tech/blog/introducing-limbo-a-complete-rewrite... - Jan 21, 2025 - "We will rewrite SQLite. And we are going all-in" https://turso.tech/blog/we-will-rewrite-sqlite-and-we-are-going-all-in https://turso.tech/blog/we-will-rewrite-sqlite-and-we-are-go... - Project: https://github.com/tursodatabase/turso https://github.com/tursodatabase/turso Status: "Turso Database is currently under heavy development and is not ready for production use."
- a-dub 1y ago
  sqlite3 has one (apparently this is called "the amalgamation") c source file that is ~265 kloc (!) long with external dependencies on zlib, readline and ncurses. built binaries are libsqlite3.so at 4.8M and sqlite3 at 6.1M. turso has 341 rust source files spread across tens of directories and 514 (!) external dependencies that produce (in release mode) 16 libraries and 7 binaries with tursodb at 48M and libturso_sqlite3.so at 36M. looks roughly an order of magnitude larger to me. it would be interesting to understand the memory usage characteristics in real-world workloads. these numbers also sort of capture the character of the languages. for extreme portability and memory efficiency, probably hard to beat c and autotools though.
  - csande17 1y ago
    I don't think the SQLite authors actually edit the single giant source file directly. Their source control repository has the code split up into many separate files, which are combined into "the amalgamation" by a build script: https://github.com/sqlite/sqlite/tree/master/src https://github.com/sqlite/sqlite/tree/master/src
    - a-dub 1y ago
      yeah i saw that afterwards. they do it to squeeze more optimization out of the compiler by putting everything in one compilation unit. given the prominence of the library i have to wonder if this was an input to zig's behind-the-scenes single compilation unit design choice...
01HNNWZ0MV43FF 1y ago
But if you don't have the bounds checks in machine code, then you don't have bounds checks. I suppose SQLite might use a C linter tool that can prove the bounds checks happen at a higher layer, and then elide redundant ones in lower layers, but... C compilers won't do that by default, they'll just write memory-unsafe machine code. Right?
saghm 1y ago
The latter two of these points seem pretty subjective. I feel like they're written in a way that anyone could easily make an argument for or against Rust having fulfilled them today or literally at any point in the future. There's not really any way to refute an argument that something is "changing too fast", "not boring enough", or "hasn't demonstrated it can do the kind of work C does". I'd find them more compelling if they were expressed in a way it was actually possible to verify whether Rust reaches them or not at some arbitrary point in the future. Otherwise, people will inevitably just see them as a reflection of their existing opinion on whether it's wise or foolish for SQLite to use C rather than Rust.

by pm2222 1y ago

rstuart4133 1y ago

> Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy. This is annoying in Rust. To me array accesses aren't the most annoying, it's match{} branches that will never been invoked. There is unreachable!() for such situations, and you would hope that: if array_access_out_of_bounds { unreachable!(); } is recognised by the Rust tooling and just ignored. That's effectively the same as SQLite is doing now by not doing the check. But it isn't ignored by the tooling: unreachable!() is reported as a missed line. Then there is the test code coverage including the standard output by default, and you have to use regex's on path names to remove it.

steveklabnik 1y ago
A more direct translation of the sqlite strategy here is to use get_unchecked instead of [], and then you get the same behaviors. Your example does what [] does already, it’s just a more verbose way of writing the same thing. It’s not the same behavior as sqlite.
- rstuart4133 1y ago
  > A more direct translation of the sqlite strategy here is to use get_unchecked instead of [], and then you get the same behaviors. Array access was just an example. My point was that Rust makes 100% code coverage for unit tests well neigh impossible. For those of us who like 100% test coverage, that is a major annoyance. For way I use Rust that could be fixed by the tooling simply not counting unreachable!() lines as unreached. For Sqlite, who does branch coverage testing on the compiled binary you would have to go a step further, and provide a compile time option that elides all paths that lead to unreachable!() from the binary. I recall Sqlite saying when they changed to 100% branch coverage, their bug reports dropped by a factor of 7. I hope I remember that correctly. If I do, I'm pretty sure they won't be looking at Rust until they can achieve the same outcome. They can't come close now. Replacing every array access with get_unchecked() and the consequent explosion of unsafe area's wouldn't fly with anyone I worked with. You did say to me in another thread unsafe is perfectly fine in Rust programs, but you are literally the only person holding that opinion I come across.

steveklabnik 1y ago

If the branch is never taken, and the optimizer can prove it, it will remove the check. Sometimes if it can’t actually prove it there’s ways to help it understand, or, in the almost extreme case, you do what I commented below.

sedatk 1y ago
Yeah I don't understand the argument. If you can't convince the compiler that that branch will never be taken, then I strongly suspect that it may be taken.
- unclad5968 1y ago
  That's not the point. The point is that if it is never taken, you can't test it. They don't care that it inserts a conditional OP to check, they care that they can't test the conditional path.
  - sedatk 1y ago
    But, there is no conditional path when the type system can assure the compiler that there is nothing to be conditional about. Do they mean that it's impossible to be 100% sure about if there's a conditional path or not?
- compiler-guy 1y ago
  A program can have many properties that the compiler cannot prove statically. To take a very basic case, the halting problem.

pella 1y ago

Turso: https://algora.io/challenges/turso https://algora.io/challenges/turso "Turso is rewriting SQLite in Rust ; Find a bug to win $1,000" ------ - Dec 10, 2024 : "Introducing Limbo: A complete rewrite of SQLite in Rust" https://turso.tech/blog/introducing-limbo-a-complete-rewrite-of-sqlite-in-rust https://turso.tech/blog/introducing-limbo-a-complete-rewrite... - Jan 21, 2025 - "We will rewrite SQLite. And we are going all-in" https://turso.tech/blog/we-will-rewrite-sqlite-and-we-are-going-all-in https://turso.tech/blog/we-will-rewrite-sqlite-and-we-are-go... - Project: https://github.com/tursodatabase/turso https://github.com/tursodatabase/turso Status: "Turso Database is currently under heavy development and is not ready for production use."

a-dub 1y ago
sqlite3 has one (apparently this is called "the amalgamation") c source file that is ~265 kloc (!) long with external dependencies on zlib, readline and ncurses. built binaries are libsqlite3.so at 4.8M and sqlite3 at 6.1M. turso has 341 rust source files spread across tens of directories and 514 (!) external dependencies that produce (in release mode) 16 libraries and 7 binaries with tursodb at 48M and libturso_sqlite3.so at 36M. looks roughly an order of magnitude larger to me. it would be interesting to understand the memory usage characteristics in real-world workloads. these numbers also sort of capture the character of the languages. for extreme portability and memory efficiency, probably hard to beat c and autotools though.
- csande17 1y ago
  I don't think the SQLite authors actually edit the single giant source file directly. Their source control repository has the code split up into many separate files, which are combined into "the amalgamation" by a build script: https://github.com/sqlite/sqlite/tree/master/src https://github.com/sqlite/sqlite/tree/master/src
  - a-dub 1y ago
    yeah i saw that afterwards. they do it to squeeze more optimization out of the compiler by putting everything in one compilation unit. given the prominence of the library i have to wonder if this was an input to zig's behind-the-scenes single compilation unit design choice...

01HNNWZ0MV43FF 1y ago

But if you don't have the bounds checks in machine code, then you don't have bounds checks. I suppose SQLite might use a C linter tool that can prove the bounds checks happen at a higher layer, and then elide redundant ones in lower layers, but... C compilers won't do that by default, they'll just write memory-unsafe machine code. Right?

saghm 1y ago

The latter two of these points seem pretty subjective. I feel like they're written in a way that anyone could easily make an argument for or against Rust having fulfilled them today or literally at any point in the future. There's not really any way to refute an argument that something is "changing too fast", "not boring enough", or "hasn't demonstrated it can do the kind of work C does". I'd find them more compelling if they were expressed in a way it was actually possible to verify whether Rust reaches them or not at some arbitrary point in the future. Otherwise, people will inevitably just see them as a reflection of their existing opinion on whether it's wise or foolish for SQLite to use C rather than Rust.