6 ms·

Presumably people would want to rewrite sqlite into Rust. But it's still a database, a low level, high performance software system. Even if you write it in rust

by eloff 5y ago

Presumably people would want to rewrite sqlite into Rust. But it's still a database, a low level, high performance software system. Even if you write it in rust, some percentage of the code will be unsafe rust or rust that calls a library written in C. It will be safer, but still not a panacea. It would be more viable in my opinion to improve the safety of the C code that currently makes up sqlite.

tptacek 5y ago
I do not agree with this at all, for what it's worth.
- wk_end 5y ago
  That's not a useful or constructive comment. At best it is, indeed, worthless; at worst, you're using your stature on HN to bully another commenter by implying that their opinion can be dismissed out of hand, because you, tptacek, say so. I don't really agree with it either - it overstates the issues with occasionally using escape hatches in Rust by implying that they're at all comparable with the problems inherent with using C. But yours was still an unnecessary reply.
- eloff 5y ago
  Can you clarify why?
  - masklinn 5y ago
    The amount of code which would have to be `unsafe` in a library like sqlite would be absolutely minuscule if it would exist at all, and much easier to check for than literally all of the codebase?
    - eloff 5y ago
      Well, yeah, it's safer, there's no question about it. Is it sufficiently much safer to warrant the massive investment and the bugs (including security bugs) which will result from a rewrite? If I was the project manager I would not sign off on that. This is a volunteer project. Someone can of course do all that work free of charge, but they'd really just creating a fork of sqlite unless they also get most of the other major volunteers to agree.
  - tptacek 5y ago
    I don't think we understand enough about computer science or computer engineering (or something in the middle of those two things) to deliver large-scale C projects with rich, flexible interfaces safely. The last 20 years has just been a sequence of events where we've been surprised by new oversights, from buffer overflows to integer mishandling to uninitialized variables to UB. These problems compound; they're never solved, but rather beaten into development teams (at least the conscientious ones) and so even the decades-old problems recur, because it's not enough to know about the general pattern of a problem, you also have to viscerally understand all the combinatorics of those problems and all the new code that you write, all of which has the potential to create some new scenario that allows a well-known vulnerability to re-emerge. And even if you manage to clean the Augean Stables this way, you're still S.O.L. when the next new memory corruption bug class is discovered. A mug's game, a bad bet. You can use formal methods to sidestep this! But my argument would be that at that point, you're really writing C In Name Only. By all means, if the aesthetics of Rust are that painful to you (I get it!), write C against a formal verifier. :)
    - eloff 5y ago
      I agree with this for the most part. But just because you use Rust doesn't mean you have no security issues - there are still plenty of ways to go wrong, including integer overflow if you're using "as" conversions instead of checked conversions. But there's no doubt it eliminates a whole class of issues and makes others much less likely (and verifiably so.) For a greenfield project, you shouldn't use C or C++ today, unless you need some huge big C++ library (e.g. game engine). For sqlite which has 100% branch coverage and is quite well written, I don't think a rewrite makes sense. You'll surely introduce more bugs, including security bugs, than you fix. The cost seems too high. It'd be better to just fix security issues as they come up (and try to clean up bits of code that could be problematic.)
      - pdimitar 5y ago
        I love Rust but I agree with you. For sqlite in particular the argument seems heavily tilted against a rewrite.
vlovich123 5y ago
Let’s be generous and assume that 5% of the overall code remains unsafe. That’s 95% of the code that doesn’t need to be checked extra. Additionally, that 5% is likely to remain static. With C any “securing” effort is a snapshot effort that bitrots quickly as more code is added.
- tptacek 5y ago
  sqlite is renowned as a project that tests meticulously; they claim 100% branch test coverage, to the point where a reason they reject current Rust is that they can't achieve similar coverage against the branches the compiler itself inserts as safety checks (that is: they can't use Rust because they can't test safety checks that don't even exist in the language they use today). And the track record of that approach is, well, right there for you to see. The idea that what's needed for sqlite's C to be safe is more of the approach sqlite already uses seems pre-falsified, doesn't it?
  - infogulch 5y ago
    I think 100% machine code branch test coverage is legally required in some of the environments that SQLite targets. That is, there must be a test that exercises both sides of every machine code branch instruction emitted in the final binary. Basically, every branch must have a justification for why it exists. I feel like that's not such a bad target for rust to aim for.
  - vlovich123 5y ago
    > sqlite is renowned as a project that tests meticulously; they claim 100% branch test coverage You realize that there are lots of test coverage metrics and none of them tell you anything about the correctness or safety of the code or about how thoroughly the edge cases have been tested, right? 100% branch coverage is neat and may indicate that the testing is thorough, or it might just indicate the project is chasing it as a metric. I’ll bet on compile time lifetime and ownership analysis guaranteed by the language every time over metrics of how good the test suite is (a test suite is important but classes of bugs are just impossible in Rust and don’t need testing/coverage in the first place). > that is: they can't use Rust because they can't test safety checks that don't even exist in the language they use today Can you back up this claim? This feels like a very FUD statement as any high level language (including C) has a risk that the compiler inserts branches that aren’t present in the code. Regardless, AFAIK, coverage instrumentation happens at a level below where the distinction between C and Rust matters, so any coverage should be the same. I buy the argument that replicating the test suite may be time consuming bit, but the claim that there’s something inherent about Rust preventing branch coverage feels extraordinary to me. Also, I’m not even sure where the extra branches are being inserted. Are you referring to drop statements the compiler injects for lifetimes? > And the track record of that approach is, well, right there for you to see. Constant CVEs and a fundamental inability to handle maliciously crafted files? I’m not shitting on the SQLite team. The project is amazing and a marvel. I’m just saying that its C heritage has some inescapable realities. Also remember that studies tend to show that the bug rate is pretty constant across languages in terms of LOC. This you want the language and stdlib doing as much as possible for you if you’re prioritizing correctness and security.
CraigJPerry 5y ago
>> some percentage of the code will be unsafe rust What would be the significance of that? Unsafe blocks in rust must still satisfy the borrow checker and all the compiler’s safety rules. The only things unsafe gives you are: Dereference a raw pointer Call an unsafe function or method Access or modify a mutable static variable Implement an unsafe trait Access fields of unions From https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html The use of unsafe blocks does not automatically mean unsafe code?
- eloff 5y ago
  Those guarantees are checked by the programmer, not the compiler. Mistakes and bugs can happen, just like in C.

by eloff 5y ago

tptacek 5y ago

I do not agree with this at all, for what it's worth.

wk_end 5y ago
That's not a useful or constructive comment. At best it is, indeed, worthless; at worst, you're using your stature on HN to bully another commenter by implying that their opinion can be dismissed out of hand, because you, tptacek, say so. I don't really agree with it either - it overstates the issues with occasionally using escape hatches in Rust by implying that they're at all comparable with the problems inherent with using C. But yours was still an unnecessary reply.
eloff 5y ago
Can you clarify why?
- masklinn 5y ago
  The amount of code which would have to be `unsafe` in a library like sqlite would be absolutely minuscule if it would exist at all, and much easier to check for than literally all of the codebase?
  - eloff 5y ago
    Well, yeah, it's safer, there's no question about it. Is it sufficiently much safer to warrant the massive investment and the bugs (including security bugs) which will result from a rewrite? If I was the project manager I would not sign off on that. This is a volunteer project. Someone can of course do all that work free of charge, but they'd really just creating a fork of sqlite unless they also get most of the other major volunteers to agree.
- tptacek 5y ago
  I don't think we understand enough about computer science or computer engineering (or something in the middle of those two things) to deliver large-scale C projects with rich, flexible interfaces safely. The last 20 years has just been a sequence of events where we've been surprised by new oversights, from buffer overflows to integer mishandling to uninitialized variables to UB. These problems compound; they're never solved, but rather beaten into development teams (at least the conscientious ones) and so even the decades-old problems recur, because it's not enough to know about the general pattern of a problem, you also have to viscerally understand all the combinatorics of those problems and all the new code that you write, all of which has the potential to create some new scenario that allows a well-known vulnerability to re-emerge. And even if you manage to clean the Augean Stables this way, you're still S.O.L. when the next new memory corruption bug class is discovered. A mug's game, a bad bet. You can use formal methods to sidestep this! But my argument would be that at that point, you're really writing C In Name Only. By all means, if the aesthetics of Rust are that painful to you (I get it!), write C against a formal verifier. :)
  - eloff 5y ago
    I agree with this for the most part. But just because you use Rust doesn't mean you have no security issues - there are still plenty of ways to go wrong, including integer overflow if you're using "as" conversions instead of checked conversions. But there's no doubt it eliminates a whole class of issues and makes others much less likely (and verifiably so.) For a greenfield project, you shouldn't use C or C++ today, unless you need some huge big C++ library (e.g. game engine). For sqlite which has 100% branch coverage and is quite well written, I don't think a rewrite makes sense. You'll surely introduce more bugs, including security bugs, than you fix. The cost seems too high. It'd be better to just fix security issues as they come up (and try to clean up bits of code that could be problematic.)
    - pdimitar 5y ago
      I love Rust but I agree with you. For sqlite in particular the argument seems heavily tilted against a rewrite.

vlovich123 5y ago

Let’s be generous and assume that 5% of the overall code remains unsafe. That’s 95% of the code that doesn’t need to be checked extra. Additionally, that 5% is likely to remain static. With C any “securing” effort is a snapshot effort that bitrots quickly as more code is added.

tptacek 5y ago
sqlite is renowned as a project that tests meticulously; they claim 100% branch test coverage, to the point where a reason they reject current Rust is that they can't achieve similar coverage against the branches the compiler itself inserts as safety checks (that is: they can't use Rust because they can't test safety checks that don't even exist in the language they use today). And the track record of that approach is, well, right there for you to see. The idea that what's needed for sqlite's C to be safe is more of the approach sqlite already uses seems pre-falsified, doesn't it?
- infogulch 5y ago
  I think 100% machine code branch test coverage is legally required in some of the environments that SQLite targets. That is, there must be a test that exercises both sides of every machine code branch instruction emitted in the final binary. Basically, every branch must have a justification for why it exists. I feel like that's not such a bad target for rust to aim for.
- vlovich123 5y ago
  > sqlite is renowned as a project that tests meticulously; they claim 100% branch test coverage You realize that there are lots of test coverage metrics and none of them tell you anything about the correctness or safety of the code or about how thoroughly the edge cases have been tested, right? 100% branch coverage is neat and may indicate that the testing is thorough, or it might just indicate the project is chasing it as a metric. I’ll bet on compile time lifetime and ownership analysis guaranteed by the language every time over metrics of how good the test suite is (a test suite is important but classes of bugs are just impossible in Rust and don’t need testing/coverage in the first place). > that is: they can't use Rust because they can't test safety checks that don't even exist in the language they use today Can you back up this claim? This feels like a very FUD statement as any high level language (including C) has a risk that the compiler inserts branches that aren’t present in the code. Regardless, AFAIK, coverage instrumentation happens at a level below where the distinction between C and Rust matters, so any coverage should be the same. I buy the argument that replicating the test suite may be time consuming bit, but the claim that there’s something inherent about Rust preventing branch coverage feels extraordinary to me. Also, I’m not even sure where the extra branches are being inserted. Are you referring to drop statements the compiler injects for lifetimes? > And the track record of that approach is, well, right there for you to see. Constant CVEs and a fundamental inability to handle maliciously crafted files? I’m not shitting on the SQLite team. The project is amazing and a marvel. I’m just saying that its C heritage has some inescapable realities. Also remember that studies tend to show that the bug rate is pretty constant across languages in terms of LOC. This you want the language and stdlib doing as much as possible for you if you’re prioritizing correctness and security.

CraigJPerry 5y ago

>> some percentage of the code will be unsafe rust What would be the significance of that? Unsafe blocks in rust must still satisfy the borrow checker and all the compiler’s safety rules. The only things unsafe gives you are: Dereference a raw pointer Call an unsafe function or method Access or modify a mutable static variable Implement an unsafe trait Access fields of unions From https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html The use of unsafe blocks does not automatically mean unsafe code?

eloff 5y ago
Those guarantees are checked by the programmer, not the compiler. Mistakes and bugs can happen, just like in C.