5 ms·

Lack of multiple independent implementations means we can't ever switch to a different programing language or radically different system architecture. Nobody i

Lack of multiple independent implementations means we can't ever switch to a different programing language or radically different system architecture. Nobody is feasibly going to rewrite Sqlite in Rust. So yeah, it is important.

kristianp 6y ago
That's very interesting. The use case described is for quite a limited subset of sqlite. I imagine a limited query language and kv-store could suffice for the usage described in the article. (Probably could be done in a weekend! wink.)
GaelFG 6y ago
Nobody is feasibly going to rewrite Sqlite in Rust. Per (Genuine) curiosity why ? I know (good) database systems are hard to make but do sqlite have particulary 'hard' part which are near impossible to replicate ? Especially since we now have a (I assume) well documented base implementation and an extensive test suite/history of error to avoid ? Is making à file based database that hard,compared to creating new programming language for exemple ?
- fpoling 6y ago
  The problem is to write a reliable file database. Firefox got a lot of bugs related to history file corruption until it was replaced with SQLite. The same story was with Subversion where they replaced Berkeley key DB due to reliability issues. This reliability comes from a very deep knowledge of OS API and all their corner cases. Creating a programming language typically involves less corner cases and the bugs in the compiler/runtime are much simple to reproduce (again, typically).
- merijnv 6y ago
  > Is making à file based database that hard,compared to creating new programming language for exemple ? Yes. I can hack together a trivial compiler in a weekend. A serious (non-optimising) compiler for a complex language is a task you could do in a few months. If you gave me two year I'm not sure I could implement a filesystem API that correctly did something as simple as "atomically append to a file". See also Dan Luu's writing on the topic of filesystems: https://danluu.com/deconstruct-files/ https://danluu.com/deconstruct-files/
- colonwqbang 6y ago
  Maybe it's not so much that it's impossible to do so. Perhaps there is just no reason to do so. Sqlite is one of the most battle-tested pieces of software in the world. It's used "everywhere" and it's performant. It is not the case, as some seem to believe, that rewriting software in Rust will magically eliminate all bugs and give a free speed boost compared to well-tested and optimised C.
- ori_b 6y ago
  It's more than 200,000 lines of code. Lua and its libraries are about 45,000 lines of code. So, yes.
DougBTX 6y ago
The SQLite website is a little more positive than that, but as a "quite a bit in the future" option, rather than something that can happen today: > All that said, it is possible that SQLite might one day be recoded in Rust. Recoding SQLite in Go is unlikely since Go hates assert(). But Rust is a possibility. https://sqlite.org/whyc.html https://sqlite.org/whyc.html
- bluGill 6y ago
  Good read. It fails by calling C++ a object oriented language, which is false. C++ is a deterministic destruction language. But in the end it doesn't matter, sqlite has been in C long enough that all the hard things C++ gives you for free have been coded manually anyway so there is no real point. For new code C++ might be better, but that is debatable. Just like rust might be better, but that is debatable.
- quietbritishjim 6y ago
  Thanks for posting that link, it was a very interesting read. I'm a big fan of Rust, but that article really convinced me that C is the best language for SQLite. A couple of highlights IMO: > 2. Safe programming languages solve the easy problems: memory leaks, use-after-free errors, array overruns, etc. Safe languages provide no help beyond ordinary C code in solving the rather more difficult problem of computing a correct answer to an SQL statement. (My commentary) So SQLite development is at such a different level to normal application that the memory safety errors that are a concern for us mortals are the "easy" problems for SQLite developers! > 5. Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy. (My commentary) I don't really agree with this one: If a branch cannot possibly be taken then I don't think it counts towards your coverage percentage... just like, if you're careful never to dereference null pointers, you don't include dereferencing null pointers as a missing part of your test coverage. Still, I thought it was very interesting, especially since it's not quite the argument I was expecting (that those checks are wasted CPU time and binary bloat because they're not needed). (Edit: What's more, exactly this situation applies with SQLite's own ALWAYS() and NEVER() macros [1]) [1] https://sqlite.org/assert.html#different_behaviors_according_to_build_type https://sqlite.org/assert.html#different_behaviors_according...
typ 6y ago
Not sure if this is qualified: https://gitlab.com/cznic/sqlite https://gitlab.com/cznic/sqlite It seems to be a Sqlite (compatible) implementation written in Go.
- arp242 6y ago
  It's not exactly a reimplementation, but rather an automatic conversion of the C source code to Go.

kristianp 6y ago

That's very interesting. The use case described is for quite a limited subset of sqlite. I imagine a limited query language and kv-store could suffice for the usage described in the article. (Probably could be done in a weekend! wink.)

GaelFG 6y ago

Nobody is feasibly going to rewrite Sqlite in Rust. Per (Genuine) curiosity why ? I know (good) database systems are hard to make but do sqlite have particulary 'hard' part which are near impossible to replicate ? Especially since we now have a (I assume) well documented base implementation and an extensive test suite/history of error to avoid ? Is making à file based database that hard,compared to creating new programming language for exemple ?

fpoling 6y ago
The problem is to write a reliable file database. Firefox got a lot of bugs related to history file corruption until it was replaced with SQLite. The same story was with Subversion where they replaced Berkeley key DB due to reliability issues. This reliability comes from a very deep knowledge of OS API and all their corner cases. Creating a programming language typically involves less corner cases and the bugs in the compiler/runtime are much simple to reproduce (again, typically).
merijnv 6y ago
> Is making à file based database that hard,compared to creating new programming language for exemple ? Yes. I can hack together a trivial compiler in a weekend. A serious (non-optimising) compiler for a complex language is a task you could do in a few months. If you gave me two year I'm not sure I could implement a filesystem API that correctly did something as simple as "atomically append to a file". See also Dan Luu's writing on the topic of filesystems: https://danluu.com/deconstruct-files/ https://danluu.com/deconstruct-files/
colonwqbang 6y ago
Maybe it's not so much that it's impossible to do so. Perhaps there is just no reason to do so. Sqlite is one of the most battle-tested pieces of software in the world. It's used "everywhere" and it's performant. It is not the case, as some seem to believe, that rewriting software in Rust will magically eliminate all bugs and give a free speed boost compared to well-tested and optimised C.
ori_b 6y ago
It's more than 200,000 lines of code. Lua and its libraries are about 45,000 lines of code. So, yes.

DougBTX 6y ago

The SQLite website is a little more positive than that, but as a "quite a bit in the future" option, rather than something that can happen today: > All that said, it is possible that SQLite might one day be recoded in Rust. Recoding SQLite in Go is unlikely since Go hates assert(). But Rust is a possibility. https://sqlite.org/whyc.html https://sqlite.org/whyc.html

bluGill 6y ago
Good read. It fails by calling C++ a object oriented language, which is false. C++ is a deterministic destruction language. But in the end it doesn't matter, sqlite has been in C long enough that all the hard things C++ gives you for free have been coded manually anyway so there is no real point. For new code C++ might be better, but that is debatable. Just like rust might be better, but that is debatable.
quietbritishjim 6y ago
Thanks for posting that link, it was a very interesting read. I'm a big fan of Rust, but that article really convinced me that C is the best language for SQLite. A couple of highlights IMO: > 2. Safe programming languages solve the easy problems: memory leaks, use-after-free errors, array overruns, etc. Safe languages provide no help beyond ordinary C code in solving the rather more difficult problem of computing a correct answer to an SQL statement. (My commentary) So SQLite development is at such a different level to normal application that the memory safety errors that are a concern for us mortals are the "easy" problems for SQLite developers! > 5. Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy. (My commentary) I don't really agree with this one: If a branch cannot possibly be taken then I don't think it counts towards your coverage percentage... just like, if you're careful never to dereference null pointers, you don't include dereferencing null pointers as a missing part of your test coverage. Still, I thought it was very interesting, especially since it's not quite the argument I was expecting (that those checks are wasted CPU time and binary bloat because they're not needed). (Edit: What's more, exactly this situation applies with SQLite's own ALWAYS() and NEVER() macros [1]) [1] https://sqlite.org/assert.html#different_behaviors_according_to_build_type https://sqlite.org/assert.html#different_behaviors_according...

typ 6y ago

Not sure if this is qualified: https://gitlab.com/cznic/sqlite https://gitlab.com/cznic/sqlite It seems to be a Sqlite (compatible) implementation written in Go.

arp242 6y ago
It's not exactly a reimplementation, but rather an automatic conversion of the C source code to Go.