13 ms·
Limbo: A complete rewrite of SQLite in Rust
- toenail 2y agoDunno. Good luck to them, but I never saw a need to rewrite sqlite.
- kabirgoel 2y agoGuessing the shortcomings become starker if you’re spending lots of time in the codebase/building a company on top of it.
- cryptonector 2y agoI do see a need for multiple implementations of SQLite3. First there's the need for multiple implementations for the reasons given by the LibSQL folks, second there's the need for a memory-safe language implementation of SQLite3, and third there's the need for a native language implementation for languages whose runtimes really want not to have C involved (e.g., Go).
- xyc 2y agoThe fact that there's no alternative implementation of SQLite also seems to play a part in preventing standardization of WebSQL. https://www.w3.org/TR/webdatabase/ https://www.w3.org/TR/webdatabase/ "The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path."
- cryptonector 2y agoIndeed! This sort of thing is a problem. It's the same with Internet protocols: you need at least two implementations to get to Standard.
- glommer 2y agoI was completely unaware of that! How old is that document? I should reach out.
- bborud 2y agoMost of us don't see the need to do something until after it is done.
- xray2 2y agoI’ll take the faster c version anyday over the rust. How are those conditional if statements working for you all?
- AlotOfReading 2y agoThe benchmarks in the post indicate that Limbo is more performant than SQLite, not less.
- jsheard 2y agoRelated: https://old.reddit.com/r/rust/comments/1ha7uyi/memorysafe_png_decoders_now_vastly_outperform_c/ https://old.reddit.com/r/rust/comments/1ha7uyi/memorysafe_pn... C libraries aren't automatically the fastest option, there's a lot of C code which has stagnated on the performance front but is still widely used because it's battle tested and known to be robust by C standards.
- AlotOfReading 2y agoThere's still an element of truth in the idea that C is going to be faster by default. There's simply a much lower bar to writing fast (and unsafe) C. Fast Rust demands considerably more thoughtfulness from the programmer (at least for me).
- jsheard 2y ago> There's simply a much lower bar to writing fast (and unsafe) C. That's kind of my point, writing a faster PNG decoder in C may be easier for you but convincing anyone to actually use it instead of the slower but proven safe-ish libpng would be an uphill battle. Trust in C code is extremely hard-won compared to Rust which uses little if any unsafe. The 'png' crate that Chrome is considering to replace libpng has no unsafe whatsoever and is still faster.
- gmokki 2y agoFor very simple things C might be accidentally faster. But C lacks many modern data structures, which are critical for performance on modern hardware and bigger input data.
- edweis 2y agoI am not sure how feasible it is, but can't SQLite be partially rewritten step by step on the main branch instead of being forked? As the article mentioned, a complete rewrite will not be as stable as the original.
- cryptonector 2y ago> can't SQLite be partially rewritten step by step on the main branch Only by the SQLite team. They don't accept contributions of anything other than spelling fixes and such.
- alserio 2y agowhat do you mean with "on the main branch"? i doubt that migrating from c makes sense for their constraints and expertise, you would not want someone to come into your house and change your furniture. forking is the right political and technical approach for this team. also rust does not support a lot of sqlite target platforms
- bpicolo 2y agoSQLite is open-source but not open-contribution – they don't accept contributions of that sort. They follow "cathedral" style development and invented a whole alternative to git for that purpose https://fossil-scm.org/home/doc/43c3d95a/www/fossil-v-git.wiki https://fossil-scm.org/home/doc/43c3d95a/www/fossil-v-git.wi... https://www.sqlite.org/copyright.html https://www.sqlite.org/copyright.html > In order to keep SQLite completely free and unencumbered by copyright, the project does not accept patches. If you would like to suggest a change and you include a patch as a proof-of-concept, that would be great. However, please do not be offended if we rewrite your patch from scratch.
- avinassh 2y agoAs other commenters have already pointed out, SQLite does not take outside contributions. We already have a fork, called libSQL. However, the goals of Limbo are far more ambitious and we cannot rewrite some parts step by step. We want to have DST ( Deterministic Simulation Testing), a testing methodology pioneered by Foundation DB and TigerBeetle. It is not easy to do that in an existing codebase
- avinassh 2y agodisclosure: I work here. I am happy to answer any questions tl;dr We are rewriting SQLite in Rust. It uses Asynchronous I/O, considers WASM as first class, and has Deterministic Simulation Testing support from the beginning. source: https://github.com/tursodatabase/limbo https://github.com/tursodatabase/limbo
- eimrine 2y ago> It uses Asynchronous I/O Can it have more than 1 writer?
- avinassh 2y agoAs of now it has a single writer, same like SQLite. But we plan to add MVCC with multiple writers in the future. Pekka has experimented with MVCC earlier: https://github.com/penberg/tihku https://github.com/penberg/tihku
- paulryanrogers 2y agoSo a hard fork?
- neverartful 2y agoWhat is meant by a 'hard fork'? Are there different kinds of forks?
- klibertp 2y agoTo me, a "hard" fork is one where you plan not to maintain any compatibility with your upstream and not to share any future code in either way (neither from original to fork nor back). "Soft" forks often retain a degree of compatibility, and some future developments can be shared. (In this case, since it's a rewrite in Rust, it's not actually a fork at all, I think)
- 2y ago
- cryptonector 2y agoWhen the initial SQLite3->LibSQL fork was announced I was pretty negative about it because SQLite3 has a wonderful, 100% branch coverage test suite that is proprietary, and so without access to that any fork would be bound to fail. However, if there's a big product behind the fork, and even better, a rewrite to a memory-safe language, then the fork begins to make a lot of sense. So, hats off to y'all for pulling this off!
- cogman10 2y agoGood luck for sure, but browsing their compatibility matrix, it looks like they are a LONG way off. By the looks of it, they have mostly read compatibility with little write capabilities (no alter table, for example).
- cryptonector 2y agoAs long as they have the funding I'm sure they can get there.
- jey 2y agoThat's fully in line with what they're announcing here. It's the announcement of a new project that has passed the prototyping stage, but one that has not reached the 1.0 stage.
- glommer 2y agocorrect, this is just the project being moved from fun personal side project from the company's CTO to an experimental stage as a company project. There isn't a long term roadmap, or anything like that. I got pretty excited when I saw the results, though. It's less about the number of github stars - who the hell cares about those - but the contributors. Limbo already has a very nice list of contributors, which led me to believe there is something here!
- _ugfj 2y agoAnd it never will The first 90% is easy, it's the second 90% that is very hard.
- danieljanes 2y agoGiven the code quality and rigid testing, SQLite is probably the last project that should be rewritten. It'd be great to see all other C code rewritten first!
- cryptonector 2y agoThat was my take when LibSQL was announced. And it still is and would be my take if LibSQL remains C-coded. But a Rust-coded rewrite of SQLite3 or LibSQL is a different story. The SQLite3 business model is that SQLite3 is open source but the best test suite for it is proprietary, and they don't accept contributions to any of either. This incentivizes anyone who needs support and/or new features in SQLite3 to join the SQLite Consortium. It's a great business model -- I love it. But there are many users who want more of a say than even being a consortium member would grant them, and they want to contribute. For those users only a fork would make sense. But a fork would never gain much traction given that test suite being proprietary, and the SQLite3 team being so awesome. However, a memory-safe language re-implementation of SQLite3 is a very different story. The U.S. government wants everyone to abandon C/C++ -- how will they do this if they depend on SQLite3? Apart from that there's also just a general interest and need to use memory-safe languages. That said, you're right that there are many other projects that call for a rewrite in Rust way before SQLite3. The thing is: if you have the need and the funding, why wouldn't you rewrite the things you need first? And if SQLite3 is the first thing you need rewritten, why not?
- galangalalgol 2y agoAlso the very rigid testing makes the rewrite a lot easier to validate.
- nine_k 2y ago...as long as you can persuade the keepers of the proprietary test suite to agree to run it against your code.
- 2y ago
- ecmm 2y agoOT: for just a second I thought it was a rewrite of the Limbo programming language. Might be a fun side project! :)
- Retr0id 2y agoAre there any plans for the python bindings to support an async interface?
- simonw 2y agoThat would be a really cool feature! I've been running sqlite3 in async python for Datasette for six years now but it involves some pretty convoluted threading mechanisms, having native async would be fantastic.
- simonw 2y agoClearly still in very early days: uv run --with pylimbo --python 3.13 python Then: >>> import limbo >>> con = limbo.connect("/tmp/content.db") thread '<unnamed>' panicked at core/schema.rs:186:18: not yet implemented: Expected CREATE TABLE statement note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace With that environment variable: stack backtrace: 0: _rust_begin_unwind 1: core::panicking::panic_fmt 2: limbo_core::util::parse_schema_rows 3: _limbo::__pyfunction_connect 4: pyo3::impl_::trampoline::trampoline
- glommer 2y agoHey Simon! However early you think it is, I can guarantee it is even earlier =) If this is just a standard sqlite database that you are trying to open, though, I'd have expected it to work.
- simonw 2y agoIt's this database from here: https://datasette.io/content.db https://datasette.io/content.db - it uses the SQLite FTS extension though so it's not surprising there was something in there that caused problems!
- Koshkin 2y agoLimbo has been taken (as the name of a language), so this should be SQuaLor or something...
- wood_spirit 2y agoA side topic: is there a nice big extensive free test suite for sql, for people interested in making toy databases to use?
- School-Cotton 2y agohttps://github.com/gregrahn/sqllogictest https://github.com/gregrahn/sqllogictest
- odo1242 2y agoThe standard TPC-C and TPC-H benchmarks are available online at https://www.tpc.org/tpc_documents_current_versions/current_specifications5.asp https://www.tpc.org/tpc_documents_current_versions/current_s... They aren’t fully open source but are free to use, including use with open source software. They may be a bit on the complex side though.
- layer8 2y agoWhich SQL? Every database system implements their own variant, and (almost?) none are fully ANSI SQL compliant.
- plrtan 2y agoThe license is "Copyright 2024 the Limbo authors". How is that possible if Limbo is based on a rewrite? Do they claim a clean room implementation? It seems wise of SQLite to close down their test suite. That's a great idea I wish I had heard about earlier.
- teraflop 2y agoSQLite is in the public domain (i.e. not copyrighted), so a clean room implementation is unnecessary.
- elpocko 2y agoSQLite is in the public domain. It is perfectly legal to create a derivative works from a public domain project and license it however you want. It's not cool and kind of a dick move to put it under a more restrictive license, but it's legal.
- wongarsu 2y agoHowever that only really works for people who are satisfied with SQLite's public domain licensing. If you are in a jurisdiction that doesn't allow you to dedicate a work to the public domain and are worried about the SQLite developers suing you for infringement at some point, Limbo holds the exact same risk of SQLite suing you.
- vitiral 2y agoIt's not a dick move if you are making legitimate improvements -- especially if you still reference the origin. That's literally the idea behind public domain
- Animats 2y agoNice. They're just starting out, but it's a good idea.
- wgjordan 2y ago> For maximum performance, users have to choose WAL mode over journal mode, disable POSIX advisory locks, etc. Speaking of "wal" mode, is "wal2" mode [1] on your radar for this project to prevent wal files from growing indefinitely in a busy system? [1] https://sqlite.org/cgi/src/doc/wal2/doc/wal2.md https://sqlite.org/cgi/src/doc/wal2/doc/wal2.md
- ngrilly 2y agoWondering about this as well :) That would be a game changer.
- durandal1 2y ago20% faster for some operations now, but with only a small subset of SQL implemented. Sound like making it as fast as SQLite will be hard with full compatibility given that I expect a lot more branching etc? https://github.com/tursodatabase/limbo/blob/main/COMPAT.md https://github.com/tursodatabase/limbo/blob/main/COMPAT.md
- cozzyd 2y agoNot as fast as /dev/null
- 0cf8612b2e1e 2y agoIn case anyone else was wondering, SQLite is about 156k lines of code with 92,000k lines of test code.
- vitiral 2y agoSo three orders of magnitude more tests than code. Yikes, if that is what 100% branch coverage looks like then count me out!
- noname120 2y agoDo you have a source for that?
- cryptonector 2y agoWhy do you need a source? You can clone SQLite3 and count the lines yourself.
- delifue 2y agoThe test code of sqlite is not public.
- Aissen 2y agoYes and no. Part of it is public, just not the "best" part: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- noname120 2y agoThanks for the link. It looks like the public part is 27k lines of code (vs the 92,000k lines of code in the proprietary closed-source part).
- apitman 2y agoThe sqlite implementation that matters the most the next 10 years will be the one with the smallest WASM build.
- anitil 2y agoWhy do you believe that? I'm guessing you're thinking about in-browser or sandboxed lambdas?
- apitman 2y agoLocal-first primarily.
- anitil 2y agoOk I understand that, but why wasm size? Local-first works pretty well for a shared library, and it can already be compiled to wasm. But you're predicting based on the _size_ of the wasm bundle being the determining factor which is a really interesting opinion so are you able to explain that? I could understand if you said 'the fastest' or 'the safest' but 'the smallest' is what I'm hung up on
- apitman 2y agoI'm biased for sure, but the biggest thing keeping me from using sqlite or pglite in the browser is the size of the WASM payloads. They dwarf every other part of a well designed app, at least for the simple things I like to build.
- anitil 2y agoAh I see! If your use case was read-only you might be able to cut out some of the binary size, but it sounds like Web SQL would've been what you need unfortunately
- benoror 2y agoSuch a missed opportunity to call it "Lambo"
- glommer 2y agothen we'd have to ship with a PHP driver from day1.
- ledgerdev 2y ago> To complete the puzzle, we wanted to deterministically test the behavior of the database when interacting with the operating system and other components. To do that, we are partnering with Antithesis Are there any open source DST projects, even just getting started? I don't even know how/where to start if I would want to do the same on a small app, but can't afford nor want to depend long term on a commercial license.
- vitiral 2y ago> SQLite’s test suite is proprietary This is literally the first time I've ever heard of this, for any project anywhere. I suppose Android is built a bit in this way, but that's a whole other can of worms.
- heisenbit 2y agoWell, Java during its initial life was controlled to a degree through the control of the tests: https://en.wikipedia.org/wiki/Technology_Compatibility_Kit https://en.wikipedia.org/wiki/Technology_Compatibility_Kit .
- vitiral 2y agoHuh, I wasn't aware that Java was initially open source
- karussell 2y agoI think Java/JDK was closed source initially, then went open source in 2006/2007 (?), but without the TCK. The TCK was never open sourced but the JCK is now kind of "open": https://openjdk.org/groups/conformance/JckAccess/ https://openjdk.org/groups/conformance/JckAccess/
- galangalalgol 2y agoIt could be simply to prevent forks, but if it really is 100% branch coverage, why do they still have memory safety related CVE coming out? With asan turned on, and full static analysis, that should make such errors exceedingly rare. Part of the benefit of rust is that it makes coverage both easier to get due to its type system, and less necessary because of the guarantees it makes. But if they really went all the way to 100% branch coverage that should be almost as good if all the samitizers are running.
- int_19h 2y agoThey claim 100% on https://en.wikipedia.org/wiki/Modified_condition/decision_coverage https://en.wikipedia.org/wiki/Modified_condition/decision_co... However, unless you can guarantee that every branch tested has been covered for all possibly relevant application states, that does not preclude CVEs.
- minroot 2y agoIs there any software that let me make graphical user interface to a connected database, allows me to make data visualizations, all things automatic and interactive? Like a node editor or spreadsheet? It needs to be suitable for general public
- pizlonator 2y agoYou can have a memory safe SQLite today if you compile it with Fil-C. Only tiny changes required. I almost have it passing the test suite (only two test failures left, both of which look specious).
- DeathMetal3000 2y agoDid a little reading on Fil-C and… “Also, it's slow – about 1.5x-5x slower than legacy C.” So that’s dead on arrival.
- chasil 2y agoI am assuming that DO-178B certification for the Rust variant is not on the table. https://www.sqlite.org/hirely.html https://www.sqlite.org/hirely.html https://www.sqlite.org/qmplan.html https://www.sqlite.org/qmplan.html https://www.sqlite.org/th3.html https://www.sqlite.org/th3.html The name "Limbo" is also used by a post-C/UNIX language from AT&T for the Inferno operating system. https://en.wikipedia.org/wiki/Limbo_(programming_language) https://en.wikipedia.org/wiki/Limbo_(programming_language)
- ncruces 2y agoAll this talk of “SQLite is not open contribution” never seems to consider that a project being “open contribution” doesn't mean the maintainers will accept your contributions. They have a process for contributions to follow: you suggest a feature, they implement it. It's far from the only project to take such a stance. Just in the SQLite “ecosystem” see the contribution policies of Litestream and LiteFS. I don't see people brandishing the ”not open contribution” to Ben's projects. https://github.com/superfly/litefs?tab=readme-ov-file#contributing https://github.com/superfly/litefs?tab=readme-ov-file#contri... https://github.com/benbjohnson/litestream?tab=readme-ov-file#contribution-policy https://github.com/benbjohnson/litestream?tab=readme-ov-file...
- lovasoa 2y agoI would love to see it succeed! They mention testing that bytecode generation generates the exact same results as SQLite... Does this exclude writing new optimization passes that are not in sqlite?
- adamrezich 2y agoAs others have said, the “Performance” section is asinine, because they haven't fully implemented 100% of SQLite. Not disclaiming this obvious fact in the “Performance” section is incredibly misleading. I could trivially write “an SQLite clone” that could execute `SELECT * FROM users LIMIT 1` even faster than either this or SQLite—if that's the only string I accepted as input!
- sourcepluck 2y agoName clash, with this cool-looking emulator: https://virtualmachinery.weebly.com/ https://virtualmachinery.weebly.com/
- k_bx 2y agoOne killer feature I miss from SQLite is table compression. Especially important on various embedded devices where you collect data from sensors or logs.
- TheGoodBarn 2y agoSo many good things with incremental improvements in the space, but as a consumer it kinda stresses me out having to worry about libsql vs sqlite vs duckdb etc. I personally use SQLite and DuckDB daily, but recently adopted turso in lieu of litestream for a something. I appreciate that they all are relatively compatible but I'd love to just have a tool. Even then thats why I love the relationship between SQLite and DuckDB. I can backend my system with SQLite and run analytics and processing via DuckDB and they service specific purposes. The hard thing with this for me is being a split consumer and not having the bandwidth to split my attention between who is doing better innovation and just using a tool I can rely on to predictably get the job done for me. That being said, hats off this is awesome. I really appreciate turso.
- jmull 2y agoI'm not buying the rationale in the "async IO" section. First, there's no need to rewrite anything to add an async interface to sqlite if you want (many clients do, whether local or remote). The issue with sqlite's synchronous interface is leaving a thread idle while you wait for IO. But I wonder how much of an issue that really is. sqlite is designed to run very locally to the storage, and can make use of native file caching, etc, which makes IO blocking very short if not zero. You wonder if applications have enough idling sqlite threads to justify the switching. (It's not free and would be at quite a fine-grained level.) The section does mention remote storage, but in that case you're much better off with an async client talking to compute running sqlite, sync interface and all, that is very local to the storage. AKA, a client/server database. Also, in the WASM section, we're still talking about something that would best be implemented as a sqlite client/wrapper, with no need at all to rewrite it.
- phiresky 2y ago> The issue with sqlite's synchronous interface is leaving a thread idle while you wait for IO That's not the only issue. waiting for the result of every read to be able to queue the next read is also an issue, particularily for a VFS that exists on a network (which is a target of theirs, they explicitly mention S3). I'm not sure if they also are doing work on improving this, but I'm sure that theoretically many reads and writes that SQLite does do not depend on all previous reads and writes, which means you could queue many of them earlier. If your latency to storage is large, this can be a huge performance difference.
- Retr0id 2y agoYou can get more total IO throughput (at the cost of latency) by queueing up multiple reads and writes concurrently. You can do this with threads, but io_uring should theoretically go faster (but don't take my word for it, let's wait for benchmarks). I'm personally interested in the potential for async bindings for Python. Making fast async wrappers for blocking APIs in Python-land is painful (although it might improve in the future with nogil).
- jmull 2y ago
- gautamcgoel 2y agoThis is slightly OT, but I wanted to mention that Limbo is also the name of a programming language: https://en.wikipedia.org/wiki/Limbo_(programming_language) https://en.wikipedia.org/wiki/Limbo_(programming_language)
- guilhas 2y agoIs there any big open soure, long term, community contributed, in rust?
- guilhas 2y agoEdit: Is there any big open source project, long term, community contributed, in rust?
- synergy20 2y agosqlite3 is 1.6MB while limbo is 6MB, size matters for many low-end but huge-volume embedded boards.