16 ms·

SQLite: QEMU All over Again?

279 points by eatonphil 4y ago

js8 4y ago
Just a nitpick: Virtualization was pioneered by IBM S/370 mainframes, CP/CMS and the like. Author is talking about virtualization on desktop/x86 specifically. Unfortunately, for historical reasons (microcomputer revolution), these are separate worlds.
eatonphil 4y ago
Whether or not this succeeds, I think it's a great effort. SQLite not taking any outside contributions is of course their prerogative. But it would also be cool to see what could happen with a more open development model. And their (libsql) plans around io_uring and Rust for future code both sound like a good start. The way they're going about this fork (described in the repo [0] readme) seems healthy enough for both projects as well. Maybe the biggest challenge though is recreating SQLite's private/proprietary test suite [1]. [0] https://github.com/libsql/libsql https://github.com/libsql/libsql [1] https://www.sqlite.org/th3.html https://www.sqlite.org/th3.html
- fluidcruft 4y ago
  Yeah, the test suite seems pretty key. It also probably contains information about the proprietary extensions and contract work from clients and can likely never be released. Proprietary test suite is a pretty interesting strategy actually. Very difficult for others to just steal your code and support effort and run if they have to start from zero to build a massive test suite. I wonder how difficult it would be for some sort of tool to assume a particular original sqlite.c has full coverage and then suggest where additional tests are needed for patches?
  - chasil 4y ago
    Dr. Hipp has expressed interest in licensing TH3. Perhaps a startup could/should pursue this to truly maintain a reliable fork.
- cryptonector 4y ago
  SQLite's proprietary test suite is its secret sauce that has kept it closed to contributions and kept forks from happening. It is the thing that made the SQLite Consortium a going business proposition. It is the thing that makes it possible to fund an open source infrastructure project with a small, cohesive team. The public believes that that test suite exists and has 100% branch coverage, and that it is much larger and more complete than the public test suite. Of course, there's no at the private test suite exists, but we the public believe it does and we have plenty of reason to believe that it does. Any fork will instantly lose the benefits of that private test suite. This is what keeps the SQLite team able to reject external contributions, and what keeps forks from taking hold. Any hard fork will have a struggle with this. I believe a Rust re-write would have much less trouble w.r.t. the private test suite, owing to Rust's memory safety. But any fork that remains C-coded will have trouble getting public acceptance, and will have to have amazing features -or a new public test suite- to get acceptance. Meanwhile the SQLite team could respond by making SQLite3 a bit more modular and able to be used in distributed database constructions w/o alteration to any of the SQLite3 code. That would take the wind out of the sails of any fork. This possibility means that any forks need to be properly funded to be able to compete, but the SQLite Consortium is almost certainly well-funded, so it will take a serious commitment -maybe even by some of the consortium's members- to see a fork succeed.
  - chasil 4y ago
    SQLite is 100% committed to avionics with DO-178B compliance. Increasing modularity means the interfaces must comply. Rewriting in a memory-safe language means that the garbage collector must comply. (How) Can a networking interface comply? That is really asking quite a lot.
    - remram 4y ago
      Rust doesn't have a garbage collector.
  - Agingcoder 4y ago
    The private suite is about (among others) MCDC ie extremely stringent test coverage constraints at the binary level - what does this have to do with rust vs c?
- justin66 4y ago
  > But it would also be cool to see what could happen with a more open development model. It seems a little disingenuous to act like we don't know what would happen, at least in broad strokes. Just compare and contrast the nature of SQLite with "more open" projects and you can get the gist. More features, more bugs, abandonment of excellent testing standards, poor handling of what are dismissed by the developers (but not the current, extremely wide base of users) as niche concerns.
  - eatonphil 4y ago
    It would be quite an uphill battle for this project to succeed in the longterm. But I don't understand what's disingenuous about hoping for innovation by a change in process.
dleslie 4y ago
The headline is needlessly deceiving. The author is making the case for, and subsequently announcing, a hard fork of SQLite for the purposes of meeting the needs of edge computing.
- tgv 4y ago
  It seems OP wants to "shame" (*) the sqlite devs into supporting this fork. "You're going to miss the boat! History repeats itself!" (*) the modern word for guilt-tripping.
  - MichaelCollins 4y ago
    That's the way it looks to me. All of the SQLite code is already given away for free. What more is there to give? The trademark (owned by Hwaci according to the SQLite Consortium Agreement.) He wants to use politically charged shame tactics to get his hands on not just the SQLite code but the SQLite brand as well. Why does he want the trademark? Because he knows that his complaints are fringe and few people will give a damn about his fork, unless he has the trademark and can call his fork the official SQLite.
    - glommer 4y ago
      I don't want the SQLite trademark.
      - MichaelCollins 4y ago
        Your manifesto and accompanying blog post are lay out your desire for your fork to be the official SQLite. That is contrary to the inclinations of the SQLite developers, who don't accept outside contributions, so you are trying to coerce a change to their policy using public shaming and call-out tactics. So you claim you don't want the trademark signed over to you, but you do want access to the trademark opened up to your fork.
        cryptonector 4y ago
tptacek 4y ago
Just a note that LiteFS isn't a distributed filesystem; despite the name, it's not a filesystem at all, but rather just a filesystem proxy. It does essentially the same thing that the VFS layer in SQLite itself does, but it does it with a FUSE filesystem, so you don't have to change SQLite's configuration in your application. As for the "distributed" part of it: LiteFS has a single-writer multi-reader architecture; it's the same strategy you'd use to scale out Postgres. It's a little ironic to see LiteFS brought up in relation to a SQLite fork, since the premise of LiteFS is not changing the SQLite code you link into your application. Much of the work in LiteFS is about cooperating with standard SQLite. At any rate: it seems somewhat unlikely that a hard fork of SQLite is going to succeed, in that part of the reason so many teams are building SQLite tooling is the trust they have in the current SQLite team.
- mmastrac 4y ago
  Has sqlite been forked before? This is the first true fork and re-license attempt that's caught my eye. The others I've seen are "ports" and "modifications", but always pointing people back upstream. It's possible that the appetite for a SQLite fork is there, but nobody has provided it.
  - gaius_baltar 4y ago
    > Has sqlite been forked before? There is a fork called SQLCipher with native encryption. Probably lots of others too, but this one is the one that I remember.
  - vhanda 4y ago
    I do remember the author of LMDB [0] porting some parts of sqlite to use lmdb instead, and then talking about the results. A quick googling doesn't seem to give me a result though. [0] - http://www.lmdb.tech/doc/ http://www.lmdb.tech/doc/
    - acrispino 4y ago
      https://github.com/LMDB/sqlightning https://github.com/LMDB/sqlightning https://github.com/LumoSQL/LumoSQL https://github.com/LumoSQL/LumoSQL
wyldfire 4y ago
> Fabrice was, without any exaggeration, a true genius. He's still living. So is he no longer a genius?
- speedgoose 4y ago
  He is doing well: https://bellard.org/ https://bellard.org/
- csdvrx 4y ago
  Step 1 for a commercial fork: pretend the author is dead :)
- blastonico 4y ago
  You have to renew your genius certificate every 2 years before expiration.
  - gw99 4y ago
    My last one expired 20 years ago.
    - Karupan 4y ago
      You can issue yourself a self signed “Genius” certificate on LinkedIn. Just make sure your tagline has lots of buzzwords.
adamgordonbell 4y ago
I don't like how the 'manifesto' presents SQLite's code of ethics as nefarious: We take our code of conduct seriously, and unlike SQLite, we do not substitute it with an unclear alternative. We strive to foster a community that values diversity, equity, and inclusion. We encourage others to speak up if they feel uncomfortable. It seems in bad faith to imply that SQLite is some sort of toxic, anti-diversity space. It's not. it's just a closed-to-contributions open source project. Richard is pretty friendly guy and I'm not certain if efforts have been made to reach out to him here. https://github.com/libsql/libsql/blob/main/README.md https://github.com/libsql/libsql/blob/main/README.md
- incomingpain 4y ago
  - HideousKojima 4y ago
    "This document was originally called a "Code of Conduct" and was created for the purpose of filling in a box on "supplier registration" forms submitted to the SQLite developers by some clients." From the very first line of the link.
    - incomingpain 4y ago
      Good catch, i glossed over history. Apparently HN wants open source projects to push morality. I am quite downvoted.
      - eitland 4y ago
        Half of HN detest Code of Conduct and that among other things is why we love the what the SQLite team has done here.
        incomingpain 4y ago
        I detest pretty much all(including sqlite's) codes of conduct. None of them are about getting rid of toxicity but rather exercising power over people. Alas, I have even been flagged by the admins. I must have been really off base with my comment in a way I don't understand.
0xbadcafebee 4y ago
SQLite only works as a concept because it is not networked. Nobody truly understands the vast and unsolveable problem that is random shit going wrong within the communication of an application over vast distances. SQLite works great because it rejects the dogma that having one piece of software deal with all of that shit is in any way a good idea. Back your dinky microservice with SQLite, run multiple copies, have them talk to each other and fumble about trying to get consensus over the data they contain in a very loose way. That will be much, much less difficult than managing a distributed decentralized database (I speak from experience). It's good enough for 90% of cases. Remember P2P applications? That was basically the same thing. A single process running on thousands of computers with their own independent storage, shuffling around information about other nodes and advertising searches until two nodes "found each other" and shared their data (aw, love at first byte!). It's not great, but it works, and is a lot less trouble than a real distributed database.
- kortex 4y ago
  Amen. The first rule of distributed objects is "don't distribute your objects". It's much easier to reason about a bunch of different actors, each with their own copies of objects, trying to converge on consensus, than it is to actually have a distributed database that obeys ACID and does all the databasey things. A subtle distinction, but an important one.
karmicthreat 4y ago
SQLite is probably not compatible with community contributions. They seem to be very optimized to the stability and reliability of SQLite rather than features. Which is why people use it over other solutions.
johnboiles 4y ago
SITUATION: there are too many competing SQLite forks...
- discodave 4y ago
  My thoughts exactly! Why does it even need to be forked? The whole point of using SQLite as a foundation for distributed, edge, or other stuff is that it's a solid foundation that can be trusted to not change too much. Why the need to open SQLite to contributions? Build your fancy distributed, or edge database on top of SQLite and you won't need to modify SQLite at all.
  - sgbeal 4y ago
    > The whole point of using SQLite as a foundation for distributed, edge, or other stuff is that it's a solid foundation that can be trusted to not change too much. Just to clarify: sqlite3 changes almost literally every day[^1]. However, the project has always placed a premium on backwards compatibility and the developers go way out of their way not to break in-the-wild applications. Given how many databases there are ("billions and billions"), even the slightest backwards incompatibility is likely to affect _someone_, and even 1% of "billions and billions" is a significant number of databases. [^1]: https://sqlite.org/src/timeline https://sqlite.org/src/timeline
- cestith 4y ago
  Good ol' #927.
- Vt71fcAqt7 4y ago
  They aren't competing. One has trillions of installs, the other has a new code of conduct.
  - zackees 4y ago
afavour 4y ago
From a spectator's perspective it's going to be interesting to see how this all works out. SQLite's dedication to doing one thing, doing it well and staying incredibly small means it's basically my go-to example for phenomenal software in 2022. But at the same time maybe a more open model means we'll end up with an embeddable DB that's capable of even more wonderful things. I look forward to finding out.
bob1029 4y ago
I don't want to distract too much from the intended point of this, but something to keep in mind per QEMU and virtual machines - SQLite has its own VM too. In fact, by binding application-defined functions to your SQLite connection, you can realize your own virtualized application execution environment. https://www.sqlite.org/opcode.html https://www.sqlite.org/opcode.html https://www.sqlite.org/appfunc.html https://www.sqlite.org/appfunc.html
Beltalowda 4y ago
Things like rqlite are great and all, but I'm not sure if that's something that should be included in SQLite no matter which development model it uses. Looking at the libSQL page, it seems they want SQLite to go in quite a different direction in general. That's all fine, but at that point you're working on other problems than what SQLite is intended to solve. People want software to solve every possible problem under the sun, but SQLite is a project with a fairly specific and narrow scope, which is intentional. The "trick" is to enable things to be extended and patched easily, if need be, so people can build extensions and derivate projects if they want. I don't know to what degree SQLite allows this – I'm not familiar with the SQLite source code. At any rate, for this to succeed you need at least one person actually writing code for it. Thus far, all I see in the repo are some non-code changes (add license, coc, Makefile twaeks, etc.) I don't know what the exact plans are, but you need more than a GitHub repo with "we accept contributions" and wait for people to submit them, because most people/companies won't. Almost all projects live or die by their core contributors, not the community.
- justinclift 4y ago
  SQLite allows people to write extensions (C based), and load them into the running database. There are some popular ones around like Libspatialite, which provides GIS functions roughly equivalent to PostGIS: https://www.gaia-gis.it/fossil/libspatialite/index https://www.gaia-gis.it/fossil/libspatialite/index There are a bunch of others too. And yeah, I agree with you that if things could be done via extensions rather than forking SQLite... that would be better. :)
tenebrisalietum 4y ago
> Which means that the data that hits a single SQLite instance now needs to be replicated to all the others. My initial reaction: Yeah because ... SQLite is not a database server - it's an awesome API on top of a flat file? Do you want a replicated database? Then use a replicated database. Then looking at the list of solutions (rqlite, BedrockdB, dqlite, ChiselStore, LiteFS) ... I guess people really do want to make it replicated. Why not use a SQL server? Also: I was using QEMU on an old 32-bit Dell server that was before Intel's VM efforts. It worked decently. :)
- otoolep 4y ago
  As for rqlite.... see this FAQ entry: https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-would-i-use-this-versus-some-other-distributed-database https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-...
_jal 4y ago
> We encourage others to speak up if they feel uncomfortable. OK - your weird swipe at Sqlite tentatively tells me you're founding with a grudge, which (a) doesn't tend to last very long as a motivation and (b) has nothing to do with me, so I'm not interested at all. So I don't have a dog in this fight, I'll check back in a few years and see how it went. Happy forking!
_tom_ 4y ago
SQLite's maintainers have a choice. They can spend their time coding, or they can spend their time explaining, over and over again, why they don't want to do a distributed database. I can see why they don't take contributions.
- cryptonector 4y ago
  They have changed their minds on features before. E.g., FULL OUTER JOIN. They should and almost certainly do feel free to change their minds on features relating to distributed DBs.
nothanks789 4y ago
By all means, fork the code base for your specific use case, but I'll trust drh and his team over some rando any day. sqlite isn't the most widely used database by accident - it's installed on practically every mobile phone on earth. It's the result of careful and deliberate design with millions of tests and fuzzing. The sqlite team is very responsive to its users' needs, but some features just don't make the cut. That's a good thing - to keep the library small, fast and reliable.
- rwmj 4y ago
  Glauber is definitely not "some rando". He's a very skilled programmer who has worked on many things (including QEMU).
  - glommer 4y ago
    rwmj! Now here's a name I don't hear in forever! Hope you're doing great, man! And I won't take offense, I'm pretty rando, though =p
  - Kwpolska 4y ago
    Has he produced any code for the libsql project yet, or only FUD blog posts?
  - chasil 4y ago
    He will have to be extremely skilled to develop and pass a DO-178B test suite. SQLite does have a "moat" of sorts - it must always be avionics-reliable code, until 2050 or beyond. Any fork of popularity will be constantly rebasing to the head.
jacobn 4y ago
The approach to edge-ifying SQLite taken in [1] looks quite promising - using FoundationDb as the storage handles a lot of hairiness. 1: https://github.com/losfair/mvsqlite https://github.com/losfair/mvsqlite
elteto 4y ago
There is something distasteful about this announcement but I can’t quite pinpoint it. Maybe it feels like a bait and switch announcing their own fork after the whole qemu commentary, or the wording about the code of conduct. I don’t know. One of the greatest things about SQLite is how easy it is to embed in random targets/build systems/languages: a .c and .h and you are all set. Moving away from that model will turn off many people away so I hope they retain that model.
- ColonelPhantom 4y ago
  The README has "Use Rust for new features" in it, so I doubt it will retain the same simplicity. As much as I like Rust, and despite mixing Rust and C++ being the clear path forward for Mozilla, I'm not so sure it's the winning approach here. Part of the beauty of SQLite is the single .c/.h thing. That said, I can see that maybe they're trying to expand the use cases of libsql compared to SQLite. That seems to be the whole idea with adding support for e.g. distributed databases, which is something SQLite just doesn't bother with at all, and would introduce a ton of external interfacing regarding networking. SQLite uses only the standard C library, and even then barely scratches its surface. Also, SQLite's VFS API can do a lot there already. For example, I remember seeing SQLite compiled to WASM, using a VFS that downloads a remote database on S3 using HTTP Range requests. (I don't think it supported writing to the database, but it was still a really cool way of allowing complex client-side querying of a dataset that is static but too big to transfer).
  - sgbeal 4y ago
    > Part of the beauty of SQLite is the single .c/.h thing. Noting that it's not _developed_ that way. Its many files are compounded together by the build process to produce the amalgamation build. In a mixed-language project (Rust for some, C for others) a single-file distribution literally won't be possible and it will require as many toolchains to build as there are languages involved.
    - bpye 4y ago
      I wonder if mrustc [0] would be sufficient to retain the amalgamated build even if Rust were adopted. The regular Rust tool chain would be needed for development still, but if simply depending on the library the Rust components could be transpiled to C… [0] - https://github.com/thepowersgang/mrustc https://github.com/thepowersgang/mrustc
logicchains 4y ago
I suspect there's a strong correlation in this case between "Closed to outside contribution" and SQLite being one of if not the most robust pieces of software out there.
chrismorgan 4y ago
> Rejoin core SQLite if its policy changes > We are strong believers in open source that is also open to community contributions. If and when SQLite changes its policy to accept contributions, we will gladly merge our work back into the core product and continue in that space. If libSQL is going Apache-2.0¹ rather than public domain, that seems extremely unlikely. The public domain nature of SQLite is rather important for its deployments. And in fact licensing is a rather important part of why SQLite is closed to contributions (though with some administrative overhead it could be opened to contributions from people from some countries). The fact that this announcement and project documentation seems to make absolutely no mention of the licensing situation perplexes me. —⁂— ¹ As they state in the text; but https://github.com/libsql/libsql/commit/f7c54b8f792aa502f02508f521e17cee74325998 https://github.com/libsql/libsql/commit/f7c54b8f792aa502f025... is entirely insufficient, not constituting application of the license. I find this a bad sign, in stark contrast to the meticulous care SQLite has taken to copyright matters.
- cryptonector 4y ago
  Choosing a license that is not as good as public domain is a way to preclude "rejoining core SQLite".
mritun 4y ago
A lot of words, manifesto decrying sqlite somehow striffling innovation and a battle-cry for all to join … but backed with very little code. SQLite is public domain. Anyone is free to claim it as their work and do anything they wish. No need to disparage. You never were prevented from doing what you wanted to do. Take the code, improve it and if it’s any good it may get recognition. The fact is sqlite is good enough for 99.9% case. Everything else is fighting for the niche and the poster seems to be simply pissed that sqlite has the mindshare even without having their favorite features.
gw99 4y ago
This smells like "I don't like how SQLite because I can't get in to the party so I'm making my own party" which is fine but don't denigrate the product or the contributors who have been doing an amazing job for a very long time.
diffxx 4y ago
> The few core developers they have do not work with modern tools like git and collaboration tools like Github, and don’t accept contributions, although they may or may not accept your suggestion for a new feature request. The funny thing about this comment is that SQLite is as close to the gold standard of software quality that we have in the open source world. SQLite is the only program that I've ever used that reliably gets better with every release and never regresses. Could it be that this is precisely because they don't use "modern tools" and accept outside contributions?
- Ar-Curunir 4y ago
  You're conflating two different arguments: not using modern tools, and not accepting outside contributions. It's certainly possible that limiting contributions to a set of trusted contributors helps things move smoothly. However it's not clear at all that using old tools has the same effect.
  - notRobot 4y ago
    They don't use old tools, they use different tools. Git development started in 2005, Fossil development started in 2006, they'ré equally modern, and largely do the same things, just in different ways.
    - diffxx 4y ago
      Also, Fossil is essentially implemented _in_ SQLite. Fossil is used to develop SQLite which is used to implement Fossil. It's a virtuous cycle. For the SQLite project, using Fossil is obviously superior to git. This doesn't mean that arbitrary projects should use Fossil over git.
      - ThunderSizzle 4y ago
        The opposite can be said though, but git arbitrarily won over a bunch of other DVCS systems (another e.g. hg), mainly because of bandwagon and marketing.
resoluteteeth 4y ago
All of the projects listed are clearly outside the scope of what would potentially be merged into sqlite even if it accepted contributions. That said there is probably room for someone to maintain a sqlite extension distribution with standard build settings for use across different languages or something.
outworlder 4y ago
Ew. If you want to say something, say it. "I don't like that my contributions aren't accepted, so I'm forking the codebase". That's going to generate some discussion, but it's fine. Public domain and all. However, "look what happened to qemu, same thing will happen to sqlite (and I'm contributing to the problem and forking it)". One can't say "no contributions led to fragmentation" while _at the same time_ contributing to fragmentation by making a hard fork! Also, > "However, edge computing also means that your code will be running in many geographical locations, as close as possible to the user for the better possible latency. Which means that the data that hits a single SQLite instance now needs to be replicated to all the others." Then replicate away. Leave that stuff out of SQLite. If that's a really important use-case, go use couchdb or something similar.
- chasil 4y ago
  It's wonderful that there is a fork of SQLite. It's good to see new ideas. Is Airbus going to use it in the A350? No. Why not? It's not compliant with DO-178B, because it has not been confirmed correct with "Test Harness 3" (TH3).
  - dathinab 4y ago
    people are not annoyed by anyone forking SQLite they are annoyed by people misrepresenting facts in a subtle manipulative way to make their fork and the reasons around it look like something it isn't what the article intentionally or unintentionally conveys in its tone and formulation is: "sqlite is badly outdate beyond saving and needs to be replaced" what actually is the case: "we want something like SQLite but with some core changes incompatible with SQLites core principles which should be API compatible enough to allow reusing a lot of DB tooling".
nowtern 4y ago
Just looking through the libsql repository on GitHub, turns out they've changed literally zero actual code. All they've done is added a README, some long-winded 'code of conduct' document, and a configuration file for CI builds. That's it. What is even the point of announcing this? There's nothing new, nothing different. Just empty promises and hot air.
- glommer 4y ago
  yes, we haven't changed any code yet. There are essentially two approaches here: one of them is to present a finished thing, and the other is to announce your intentions as soon as possible and build every single thing in public. We chose the latter in this case, as we think community is the most important aspect of this. but you are right to be suspicious! recommend checking back later
  - throwaway0x7E6 4y ago
    in other words, the only thing you had to contribute was the coc.
  - Kwpolska 4y ago
    No code changes and inflammatory/defamatory blog posts aren’t a good way to build a community. If you had shown some code that solves a real problem SQLite has, then people would take you seriously. But if all your “contributions” consist of ideas that 99% of SQLite users won’t need, and a few text files, your project doesn’t look too serious to me.
    - chasil 4y ago
      I hope both projects are successful. I'm going to reserve judgment; perhaps something profound will emerge.
  - DoneWithAllThat 4y ago
    What does “build everything in public” mean? You’re literally describing open source software which SQLite already is.
gray_-_wolf 4y ago
> they have do not work with modern tools like git Isn't fossil younger than git?
- sgbeal 4y ago
  > Isn't fossil younger than git? They were initially developed in parallel. To the very best of my recollection, Richard had no knowledge of git when he started fossil. Fossil's initial development dates back to 2005, IIRC, (edit: maybe 2006) initially as a TCL prototype. It wasn't developed in earnest in C until 2007 and was self-hosting by the end of 2007 (i discovered it around Christmas that year).
gjvc 4y ago
This will be a flop.
- cryptonector 4y ago
  Any fork of SQLite will need serious funding to replace SQLite itself.
as4hretret 4y ago
tarasglek 4y ago
SQLite has a lot of rough edges. As amazing as it is, it would be better if the core devs were more open to outside world. I contributed a somewhat deep performance fix/workaround, it was one of the more terrible open source experiences. Hugely excited for libsql as an sqlite successor with a more modern architecture.
- glommer 4y ago
  Would love to see your contributions on libSQL!
xet7 4y ago
> The few core developers they have do not work with modern tools like git and collaboration tools like Github, and don’t accept contributions, although they may or may not accept your suggestion for a new feature request. Git is not modern. It is many executeables, and Git repos are a many files. GitHub is not modern. It is closed source huge RoR application. Fossil SCM is features of Git and GitHub much improved, and in small portable fast executeable, storing repos in much smaller space in one SQLite file. Apache is not https://copyfree.org https://copyfree.org license. I prefer copyfree licenses.
- Liquid_Fire 4y ago
  > Git is not modern. It is many executeables, and Git repos are a many files. What makes many executables and many repo files "not modern"?
jpgvm 4y ago
One one hand, SQLite would probably be a more expansive project if it allowed contributions in the same way PostgreSQL does. On the other hand it would likely make it both a larger and more complex project. I think a better tradeoff would be making the case for why SQLite should have the hooks needed to implement the edge computing bits desired without forking the project itself or attempting to replace what has made it so great for so many years. This approach would allow the extra behaviors to be managed out of tree and allow more free experimentation as a result without prematurely landing on "blessed" approaches to these very complex and frankly not yet well solved problems.
- chasil 4y ago
  Microsoft paid SQLite for Windows extensions to the database. Symbian paid SQLite for the original mobile functionality. AOL was the original customer for extending SQLite. This does not appear to be an unresponsive project. It just has to be their way, if it's their code.
Joel_Mckay 4y ago
If someone cares about their project, and chooses to share it with the community... no one should feel entitled to pressure them into another development model... I find some of SQLite's library could be cleaner, but appreciate the philosophy of keeping it consistent, understandable, and simple... It is nice to have something reliable completed in an afternoon. =)
wg0 4y ago
What they are missing here is that with QEMU only emulating and not virtualising in a way, was a far bigger limitation counterpart of which does not exist in SQLite. SQLite delivers for the process embedded database promise and delivers it well. Sure, can have column oriented flavour like DuckDB etc but they don't call themselves SQLite or its fork. Lastly, like many pointed out, SQLite is not even licensed (Apache, MIT etc) it is just in public domain so anything is possible.
losfair 4y ago
This project looks really exciting! I'm working on mvsqlite [1], a distributed SQLite based on FoundationDB. When doing the VFS integration I have always wanted to patch SQLite itself, but didn't because of uncertainty around correctness of the patched version... A few features on my wishlist: 1. Asynchronous I/O. mvsqlite is currently doing its own prefetch prediction that is not very accurate. I assume higher layers in SQLite have more information that can help with better prediction. 2. Custom page allocator. SQLite internally uses a linked list to manage database pages - this causes contention on any two transactions that both allocate or free pages. 3. Random ROWID, without the `max(int64)` row trick. Sequentially increasing ROWIDs is a primary source of contention, and causes significant INSERT slowdown in my benchmark [2]. [1] https://github.com/losfair/mvsqlite https://github.com/losfair/mvsqlite [2] https://univalence.me/posts/mvsqlite-bench-20220930 https://univalence.me/posts/mvsqlite-bench-20220930
- chasil 4y ago
  My guess is that, with the oncoming changes for concurrent writers, these will be implemented in time. Still, Rome was not built in a day, and functionality must take a backseat to reliability and legacy compatibility. https://www.sqlite.org/cgi/src/doc/begin-concurrent/doc/begin_concurrent.md https://www.sqlite.org/cgi/src/doc/begin-concurrent/doc/begi...
justin66 4y ago
I wonder if they'll attempt to replicate the portion of SQLite's test harnesses [0] that are proprietary but nevertheless beneficial to all SQLite users, tests comprising over a million lines of C code. Those tests are possibly among what the blog author does not consider to be "modern tools," which I don't even... [0] https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- cryptonector 4y ago
  That would be very expensive, and would require extensive funding, and would take years to complete.
kazinator 4y ago
Re: > * We would like to use io_uring and asynchronous interfaces > * We would like to provide replication hooks for distributed systems > * We would like to allow for WASM-based user-defined functions This shit isn't needed in SqLite any more than QEMU needs to have anything whatsoever to do with virtualization.
ENOTTY 4y ago
Frankly, as someone more interested in emulation than virtualization, I have been occasionally unhappy that most of the energy behind QEMU goes to virtualization. Some of the things QEMU has done to better adapt to virtualization do not translate well to the emulation use case.
- bonzini 4y ago
  This is not true, for example a lot of the work to enable concurrent emulation of multiple CPUs started on the virtualization side. It is also mentioned in the article that both emulation and virtualization benefited from the innovation.
mrtweetyhack 4y ago
Am4TIfIsER0ppos 4y ago
> do not work with [...] tools like Github feature > with a clear code of conduct bug > Contributor Covenant Code of Conduct terminal infection
edfletcher_t137 4y ago
This whole thing reads like a slighted, uppity, smarter-than-thou* child taking their ball and going home. It's off-putting. *what's the point of pointing out other "geniuses": to encourage the reader to think "oh well then our author must be in the same group too!"? Transparently manipulative.
thekozmo 4y ago
Listening to SQLite creator podcast (https://corecursive.com/066-sqlite-with-richard-hipp/# https://corecursive.com/066-sqlite-with-richard-hipp/# ), it does feel that Glauber is right about weird collaboration standards. The guy is against gmail, git, etc. Fossil may be better for SQlite today, w/o many contributors, that's the problem Glauber is trying to solve
- deleted 4y ago
  [deleted]
jasonhansel 4y ago
SQLite and QEMU have been successful in large part because of their limited scope. Further expanding the scope of a project puts it at risk of becoming unfocused and unmaintainable. It makes a lot more sense to start a new project to support this use case than to try to reshape SQLite.
throwawaylinux 4y ago
Paravirtualization was not pioneered by Xen in 2004, that was probably done by IBM with mainframe virtualization in the early 70s, if not before.
dathinab 4y ago
> do not work with modern tools like git That is VERY misleading and intentionally so, doesn't put a good light on the author. Sure SQLite isn't open for contribution but their tolling (VCS, issue tracking) is in no way "less modern" then git+github, it is just focused on a different approach. Putting a widely used open source but not open contribution project on GitHub is a nightmare. Similar if your are having no open contribution and only a small trusted team then you can use change flows not viable otherwise and their VCS is designed for such design flows for which git has quite a bunch of unnecessary overhead and food guns. Lastly for many project choosing no open contribution is a very sane approach properly maintaining contributions for a widely used project is a lot of work basically forcing you to delegate a lot of work to people you hardly know. If you don't have time for this or don't feel good with trusting people you hardly know then open contribution can easily become a night mare. The fact that you AFIK have to be very disciplined to write C doesn't make this better. Putting this aside another thing I found of putting in the article is that the author fails to mention that basically all of the things he wants to have in SQLit are in direct conflict with core design principles of SQLite. So they need a fork anyway independent of weather or not SQLite accepts contributions, their changes wouldn't be accepted anyway... Through without question for their use cases a new database is needed which is similar to SQLite but also in subtle but very fundamental ways very different. Starting it with a fork doesn't seem a bad idea. But they really should not say/imply it's SQLite it's not sqlite, it's just very similar and once was forked of SQLite.
kmavm 4y ago
The idea that "virtualization" began with Zen in 2004 is rather difficult to read as an early VMware employee. Before QEMU independently discovered it, VMware was JIT'ing unrestricted x86 to a safe x86 subset from 1999 on[1]. Hardware support for trap-and-emulate virtualization came to the market in the early 'aughts after VMware had proven the market demand for it. [1] https://www.vmware.com/pdf/asplos235_adams.pdf https://www.vmware.com/pdf/asplos235_adams.pdf
zackees 4y ago
draw_down 4y ago
larsnystrom 4y ago
I believe the reason this post rubs so many the wrong way is because the author keeps praising his “opponents”, trying to sound positive while being inherently negative. It’s an ugly way of argumenting as it forces the other side to be the negative one, when in fact the author is the one who’s trying to bring something down. In the end the author just sounds dishonest, like a politician with a smile plastered over their face all the time. I believe there are two ways this post could have been written which would have given a better impression of the author and his intentions. Either openly criticize SQLite, with well-formulated arguments, or don’t criticize SQLite and just announce that you’re doing something new (building on top of SQLite is not at all uncommon after all).
huino 4y ago
There's an interesting set of comments by keithmadams on Twitter https://twitter.com/keithmadams/status/1577483553281310721 https://twitter.com/keithmadams/status/1577483553281310721, regarding the accuracy of this: > This article's history of "virtualization" begins with paravirt in Zen in 2004. Then, after hardware support emerged, v12n started working with with unmodified OSes. No part of this is true. VMware was running binary OSes without hardware support in 1999. > We did this by JIT'ing unrestricted x86 to a safe x86 subset. User-level code, with few exceptions, was directly executed, but kernel-level code ran through a little JIT. We described it in https://www.vmware.com/pdf/asplos235_adams.pdf https://www.vmware.com/pdf/asplos235_adams.pdf > It performs pretty well, as described above. The only problem is that tiny JITs that run in kernel mode and use x86 segmentation to hide themselves in the top 4MB of linear memory are ... hard to write. And VMware was making a lot of money. > Zen invented paravirt; AMD invented SVM; and Intel invented VT in parallel all around 2003-2004, not to make virtualization on x86 practical, but to make it easy enough to disrupt VMware's emerging monopoly. > As a VC focused on companies exploiting a tech advantage, VMware remains my favorite case study. For a time, they could do something that was undeniably valuable, that nobody else could do. And, moats like that are always time-limited; their value invites their destruction.
Dowwie 4y ago
Can any of the contributors to ChiselStore [1] , mentioned in the article, elaborate on the comment made about it? What's the big deal about not fitting well with SQLx? This isn't a requirement for such a library to be successful. What other problems is it allegedly suffering from? It could be more of a problem of perspective. [1] https://github.com/chiselstrike/chiselstore https://github.com/chiselstrike/chiselstore
zvr 4y ago
From the README in https://github.com/libsql/libsql https://github.com/libsql/libsql: > Could SQLite be embedded in the Linux kernel? Choosing to license the fork under Apache-2.0 ensures that this will not happen (via this fork, at least), as the license is considered incompatible with the kernel's GPL-2.0-only license.
yencabulator 4y ago
> LiteFS: [...] I personally think that if distributed filesystems were easy, we’d have a good one by now. The part that makes that "easy" for LiteFS is that it's a single-writer system. That's a pretty rare trade-off in general purpose distributed filesystems.

279 points by eatonphil 4y ago

js8 4y ago

Just a nitpick: Virtualization was pioneered by IBM S/370 mainframes, CP/CMS and the like. Author is talking about virtualization on desktop/x86 specifically. Unfortunately, for historical reasons (microcomputer revolution), these are separate worlds.

eatonphil 4y ago

Whether or not this succeeds, I think it's a great effort. SQLite not taking any outside contributions is of course their prerogative. But it would also be cool to see what could happen with a more open development model. And their (libsql) plans around io_uring and Rust for future code both sound like a good start. The way they're going about this fork (described in the repo [0] readme) seems healthy enough for both projects as well. Maybe the biggest challenge though is recreating SQLite's private/proprietary test suite [1]. [0] https://github.com/libsql/libsql https://github.com/libsql/libsql [1] https://www.sqlite.org/th3.html https://www.sqlite.org/th3.html

fluidcruft 4y ago
Yeah, the test suite seems pretty key. It also probably contains information about the proprietary extensions and contract work from clients and can likely never be released. Proprietary test suite is a pretty interesting strategy actually. Very difficult for others to just steal your code and support effort and run if they have to start from zero to build a massive test suite. I wonder how difficult it would be for some sort of tool to assume a particular original sqlite.c has full coverage and then suggest where additional tests are needed for patches?
- chasil 4y ago
  Dr. Hipp has expressed interest in licensing TH3. Perhaps a startup could/should pursue this to truly maintain a reliable fork.
cryptonector 4y ago
SQLite's proprietary test suite is its secret sauce that has kept it closed to contributions and kept forks from happening. It is the thing that made the SQLite Consortium a going business proposition. It is the thing that makes it possible to fund an open source infrastructure project with a small, cohesive team. The public believes that that test suite exists and has 100% branch coverage, and that it is much larger and more complete than the public test suite. Of course, there's no at the private test suite exists, but we the public believe it does and we have plenty of reason to believe that it does. Any fork will instantly lose the benefits of that private test suite. This is what keeps the SQLite team able to reject external contributions, and what keeps forks from taking hold. Any hard fork will have a struggle with this. I believe a Rust re-write would have much less trouble w.r.t. the private test suite, owing to Rust's memory safety. But any fork that remains C-coded will have trouble getting public acceptance, and will have to have amazing features -or a new public test suite- to get acceptance. Meanwhile the SQLite team could respond by making SQLite3 a bit more modular and able to be used in distributed database constructions w/o alteration to any of the SQLite3 code. That would take the wind out of the sails of any fork. This possibility means that any forks need to be properly funded to be able to compete, but the SQLite Consortium is almost certainly well-funded, so it will take a serious commitment -maybe even by some of the consortium's members- to see a fork succeed.
- chasil 4y ago
  SQLite is 100% committed to avionics with DO-178B compliance. Increasing modularity means the interfaces must comply. Rewriting in a memory-safe language means that the garbage collector must comply. (How) Can a networking interface comply? That is really asking quite a lot.
  - remram 4y ago
    Rust doesn't have a garbage collector.
- Agingcoder 4y ago
  The private suite is about (among others) MCDC ie extremely stringent test coverage constraints at the binary level - what does this have to do with rust vs c?
justin66 4y ago
> But it would also be cool to see what could happen with a more open development model. It seems a little disingenuous to act like we don't know what would happen, at least in broad strokes. Just compare and contrast the nature of SQLite with "more open" projects and you can get the gist. More features, more bugs, abandonment of excellent testing standards, poor handling of what are dismissed by the developers (but not the current, extremely wide base of users) as niche concerns.
- eatonphil 4y ago
  It would be quite an uphill battle for this project to succeed in the longterm. But I don't understand what's disingenuous about hoping for innovation by a change in process.

dleslie 4y ago

The headline is needlessly deceiving. The author is making the case for, and subsequently announcing, a hard fork of SQLite for the purposes of meeting the needs of edge computing.

tgv 4y ago
It seems OP wants to "shame" (*) the sqlite devs into supporting this fork. "You're going to miss the boat! History repeats itself!" (*) the modern word for guilt-tripping.
- MichaelCollins 4y ago
  That's the way it looks to me. All of the SQLite code is already given away for free. What more is there to give? The trademark (owned by Hwaci according to the SQLite Consortium Agreement.) He wants to use politically charged shame tactics to get his hands on not just the SQLite code but the SQLite brand as well. Why does he want the trademark? Because he knows that his complaints are fringe and few people will give a damn about his fork, unless he has the trademark and can call his fork the official SQLite.
  - glommer 4y ago
    I don't want the SQLite trademark.
    - MichaelCollins 4y ago
      Your manifesto and accompanying blog post are lay out your desire for your fork to be the official SQLite. That is contrary to the inclinations of the SQLite developers, who don't accept outside contributions, so you are trying to coerce a change to their policy using public shaming and call-out tactics. So you claim you don't want the trademark signed over to you, but you do want access to the trademark opened up to your fork.
      - cryptonector 4y ago

tptacek 4y ago

Just a note that LiteFS isn't a distributed filesystem; despite the name, it's not a filesystem at all, but rather just a filesystem proxy. It does essentially the same thing that the VFS layer in SQLite itself does, but it does it with a FUSE filesystem, so you don't have to change SQLite's configuration in your application. As for the "distributed" part of it: LiteFS has a single-writer multi-reader architecture; it's the same strategy you'd use to scale out Postgres. It's a little ironic to see LiteFS brought up in relation to a SQLite fork, since the premise of LiteFS is not changing the SQLite code you link into your application. Much of the work in LiteFS is about cooperating with standard SQLite. At any rate: it seems somewhat unlikely that a hard fork of SQLite is going to succeed, in that part of the reason so many teams are building SQLite tooling is the trust they have in the current SQLite team.

mmastrac 4y ago
Has sqlite been forked before? This is the first true fork and re-license attempt that's caught my eye. The others I've seen are "ports" and "modifications", but always pointing people back upstream. It's possible that the appetite for a SQLite fork is there, but nobody has provided it.
- gaius_baltar 4y ago
  > Has sqlite been forked before? There is a fork called SQLCipher with native encryption. Probably lots of others too, but this one is the one that I remember.
- vhanda 4y ago
  I do remember the author of LMDB [0] porting some parts of sqlite to use lmdb instead, and then talking about the results. A quick googling doesn't seem to give me a result though. [0] - http://www.lmdb.tech/doc/ http://www.lmdb.tech/doc/
  - acrispino 4y ago
    https://github.com/LMDB/sqlightning https://github.com/LMDB/sqlightning https://github.com/LumoSQL/LumoSQL https://github.com/LumoSQL/LumoSQL

wyldfire 4y ago

> Fabrice was, without any exaggeration, a true genius. He's still living. So is he no longer a genius?

speedgoose 4y ago
He is doing well: https://bellard.org/ https://bellard.org/
csdvrx 4y ago
Step 1 for a commercial fork: pretend the author is dead :)
blastonico 4y ago
You have to renew your genius certificate every 2 years before expiration.
- gw99 4y ago
  My last one expired 20 years ago.
  - Karupan 4y ago
    You can issue yourself a self signed “Genius” certificate on LinkedIn. Just make sure your tagline has lots of buzzwords.

adamgordonbell 4y ago

I don't like how the 'manifesto' presents SQLite's code of ethics as nefarious: We take our code of conduct seriously, and unlike SQLite, we do not substitute it with an unclear alternative. We strive to foster a community that values diversity, equity, and inclusion. We encourage others to speak up if they feel uncomfortable. It seems in bad faith to imply that SQLite is some sort of toxic, anti-diversity space. It's not. it's just a closed-to-contributions open source project. Richard is pretty friendly guy and I'm not certain if efforts have been made to reach out to him here. https://github.com/libsql/libsql/blob/main/README.md https://github.com/libsql/libsql/blob/main/README.md

incomingpain 4y ago
- HideousKojima 4y ago
  "This document was originally called a "Code of Conduct" and was created for the purpose of filling in a box on "supplier registration" forms submitted to the SQLite developers by some clients." From the very first line of the link.
  - incomingpain 4y ago
    Good catch, i glossed over history. Apparently HN wants open source projects to push morality. I am quite downvoted.
    - eitland 4y ago
      Half of HN detest Code of Conduct and that among other things is why we love the what the SQLite team has done here.
      - incomingpain 4y ago
        I detest pretty much all(including sqlite's) codes of conduct. None of them are about getting rid of toxicity but rather exercising power over people. Alas, I have even been flagged by the admins. I must have been really off base with my comment in a way I don't understand.

0xbadcafebee 4y ago

SQLite only works as a concept because it is not networked. Nobody truly understands the vast and unsolveable problem that is random shit going wrong within the communication of an application over vast distances. SQLite works great because it rejects the dogma that having one piece of software deal with all of that shit is in any way a good idea. Back your dinky microservice with SQLite, run multiple copies, have them talk to each other and fumble about trying to get consensus over the data they contain in a very loose way. That will be much, much less difficult than managing a distributed decentralized database (I speak from experience). It's good enough for 90% of cases. Remember P2P applications? That was basically the same thing. A single process running on thousands of computers with their own independent storage, shuffling around information about other nodes and advertising searches until two nodes "found each other" and shared their data (aw, love at first byte!). It's not great, but it works, and is a lot less trouble than a real distributed database.

kortex 4y ago
Amen. The first rule of distributed objects is "don't distribute your objects". It's much easier to reason about a bunch of different actors, each with their own copies of objects, trying to converge on consensus, than it is to actually have a distributed database that obeys ACID and does all the databasey things. A subtle distinction, but an important one.

karmicthreat 4y ago

SQLite is probably not compatible with community contributions. They seem to be very optimized to the stability and reliability of SQLite rather than features. Which is why people use it over other solutions.

johnboiles 4y ago

SITUATION: there are too many competing SQLite forks...

discodave 4y ago
My thoughts exactly! Why does it even need to be forked? The whole point of using SQLite as a foundation for distributed, edge, or other stuff is that it's a solid foundation that can be trusted to not change too much. Why the need to open SQLite to contributions? Build your fancy distributed, or edge database on top of SQLite and you won't need to modify SQLite at all.
- sgbeal 4y ago
  > The whole point of using SQLite as a foundation for distributed, edge, or other stuff is that it's a solid foundation that can be trusted to not change too much. Just to clarify: sqlite3 changes almost literally every day[^1]. However, the project has always placed a premium on backwards compatibility and the developers go way out of their way not to break in-the-wild applications. Given how many databases there are ("billions and billions"), even the slightest backwards incompatibility is likely to affect _someone_, and even 1% of "billions and billions" is a significant number of databases. [^1]: https://sqlite.org/src/timeline https://sqlite.org/src/timeline
cestith 4y ago
Good ol' #927.
Vt71fcAqt7 4y ago
They aren't competing. One has trillions of installs, the other has a new code of conduct.
- zackees 4y ago

afavour 4y ago

From a spectator's perspective it's going to be interesting to see how this all works out. SQLite's dedication to doing one thing, doing it well and staying incredibly small means it's basically my go-to example for phenomenal software in 2022. But at the same time maybe a more open model means we'll end up with an embeddable DB that's capable of even more wonderful things. I look forward to finding out.

bob1029 4y ago

I don't want to distract too much from the intended point of this, but something to keep in mind per QEMU and virtual machines - SQLite has its own VM too. In fact, by binding application-defined functions to your SQLite connection, you can realize your own virtualized application execution environment. https://www.sqlite.org/opcode.html https://www.sqlite.org/opcode.html https://www.sqlite.org/appfunc.html https://www.sqlite.org/appfunc.html

Beltalowda 4y ago

Things like rqlite are great and all, but I'm not sure if that's something that should be included in SQLite no matter which development model it uses. Looking at the libSQL page, it seems they want SQLite to go in quite a different direction in general. That's all fine, but at that point you're working on other problems than what SQLite is intended to solve. People want software to solve every possible problem under the sun, but SQLite is a project with a fairly specific and narrow scope, which is intentional. The "trick" is to enable things to be extended and patched easily, if need be, so people can build extensions and derivate projects if they want. I don't know to what degree SQLite allows this – I'm not familiar with the SQLite source code. At any rate, for this to succeed you need at least one person actually writing code for it. Thus far, all I see in the repo are some non-code changes (add license, coc, Makefile twaeks, etc.) I don't know what the exact plans are, but you need more than a GitHub repo with "we accept contributions" and wait for people to submit them, because most people/companies won't. Almost all projects live or die by their core contributors, not the community.

justinclift 4y ago
SQLite allows people to write extensions (C based), and load them into the running database. There are some popular ones around like Libspatialite, which provides GIS functions roughly equivalent to PostGIS: https://www.gaia-gis.it/fossil/libspatialite/index https://www.gaia-gis.it/fossil/libspatialite/index There are a bunch of others too. And yeah, I agree with you that if things could be done via extensions rather than forking SQLite... that would be better. :)

tenebrisalietum 4y ago

> Which means that the data that hits a single SQLite instance now needs to be replicated to all the others. My initial reaction: Yeah because ... SQLite is not a database server - it's an awesome API on top of a flat file? Do you want a replicated database? Then use a replicated database. Then looking at the list of solutions (rqlite, BedrockdB, dqlite, ChiselStore, LiteFS) ... I guess people really do want to make it replicated. Why not use a SQL server? Also: I was using QEMU on an old 32-bit Dell server that was before Intel's VM efforts. It worked decently. :)

otoolep 4y ago
As for rqlite.... see this FAQ entry: https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-would-i-use-this-versus-some-other-distributed-database https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-...

_jal 4y ago

> We encourage others to speak up if they feel uncomfortable. OK - your weird swipe at Sqlite tentatively tells me you're founding with a grudge, which (a) doesn't tend to last very long as a motivation and (b) has nothing to do with me, so I'm not interested at all. So I don't have a dog in this fight, I'll check back in a few years and see how it went. Happy forking!

_tom_ 4y ago

SQLite's maintainers have a choice. They can spend their time coding, or they can spend their time explaining, over and over again, why they don't want to do a distributed database. I can see why they don't take contributions.

cryptonector 4y ago
They have changed their minds on features before. E.g., FULL OUTER JOIN. They should and almost certainly do feel free to change their minds on features relating to distributed DBs.

nothanks789 4y ago

By all means, fork the code base for your specific use case, but I'll trust drh and his team over some rando any day. sqlite isn't the most widely used database by accident - it's installed on practically every mobile phone on earth. It's the result of careful and deliberate design with millions of tests and fuzzing. The sqlite team is very responsive to its users' needs, but some features just don't make the cut. That's a good thing - to keep the library small, fast and reliable.

rwmj 4y ago
Glauber is definitely not "some rando". He's a very skilled programmer who has worked on many things (including QEMU).
- glommer 4y ago
  rwmj! Now here's a name I don't hear in forever! Hope you're doing great, man! And I won't take offense, I'm pretty rando, though =p
- Kwpolska 4y ago
  Has he produced any code for the libsql project yet, or only FUD blog posts?
- chasil 4y ago
  He will have to be extremely skilled to develop and pass a DO-178B test suite. SQLite does have a "moat" of sorts - it must always be avionics-reliable code, until 2050 or beyond. Any fork of popularity will be constantly rebasing to the head.

jacobn 4y ago

The approach to edge-ifying SQLite taken in [1] looks quite promising - using FoundationDb as the storage handles a lot of hairiness. 1: https://github.com/losfair/mvsqlite https://github.com/losfair/mvsqlite

elteto 4y ago

There is something distasteful about this announcement but I can’t quite pinpoint it. Maybe it feels like a bait and switch announcing their own fork after the whole qemu commentary, or the wording about the code of conduct. I don’t know. One of the greatest things about SQLite is how easy it is to embed in random targets/build systems/languages: a .c and .h and you are all set. Moving away from that model will turn off many people away so I hope they retain that model.

ColonelPhantom 4y ago
The README has "Use Rust for new features" in it, so I doubt it will retain the same simplicity. As much as I like Rust, and despite mixing Rust and C++ being the clear path forward for Mozilla, I'm not so sure it's the winning approach here. Part of the beauty of SQLite is the single .c/.h thing. That said, I can see that maybe they're trying to expand the use cases of libsql compared to SQLite. That seems to be the whole idea with adding support for e.g. distributed databases, which is something SQLite just doesn't bother with at all, and would introduce a ton of external interfacing regarding networking. SQLite uses only the standard C library, and even then barely scratches its surface. Also, SQLite's VFS API can do a lot there already. For example, I remember seeing SQLite compiled to WASM, using a VFS that downloads a remote database on S3 using HTTP Range requests. (I don't think it supported writing to the database, but it was still a really cool way of allowing complex client-side querying of a dataset that is static but too big to transfer).
- sgbeal 4y ago
  > Part of the beauty of SQLite is the single .c/.h thing. Noting that it's not _developed_ that way. Its many files are compounded together by the build process to produce the amalgamation build. In a mixed-language project (Rust for some, C for others) a single-file distribution literally won't be possible and it will require as many toolchains to build as there are languages involved.
  - bpye 4y ago
    I wonder if mrustc [0] would be sufficient to retain the amalgamated build even if Rust were adopted. The regular Rust tool chain would be needed for development still, but if simply depending on the library the Rust components could be transpiled to C… [0] - https://github.com/thepowersgang/mrustc https://github.com/thepowersgang/mrustc

logicchains 4y ago

I suspect there's a strong correlation in this case between "Closed to outside contribution" and SQLite being one of if not the most robust pieces of software out there.

chrismorgan 4y ago

> Rejoin core SQLite if its policy changes > We are strong believers in open source that is also open to community contributions. If and when SQLite changes its policy to accept contributions, we will gladly merge our work back into the core product and continue in that space. If libSQL is going Apache-2.0¹ rather than public domain, that seems extremely unlikely. The public domain nature of SQLite is rather important for its deployments. And in fact licensing is a rather important part of why SQLite is closed to contributions (though with some administrative overhead it could be opened to contributions from people from some countries). The fact that this announcement and project documentation seems to make absolutely no mention of the licensing situation perplexes me. —⁂— ¹ As they state in the text; but https://github.com/libsql/libsql/commit/f7c54b8f792aa502f02508f521e17cee74325998 https://github.com/libsql/libsql/commit/f7c54b8f792aa502f025... is entirely insufficient, not constituting application of the license. I find this a bad sign, in stark contrast to the meticulous care SQLite has taken to copyright matters.

cryptonector 4y ago
Choosing a license that is not as good as public domain is a way to preclude "rejoining core SQLite".

mritun 4y ago

A lot of words, manifesto decrying sqlite somehow striffling innovation and a battle-cry for all to join … but backed with very little code. SQLite is public domain. Anyone is free to claim it as their work and do anything they wish. No need to disparage. You never were prevented from doing what you wanted to do. Take the code, improve it and if it’s any good it may get recognition. The fact is sqlite is good enough for 99.9% case. Everything else is fighting for the niche and the poster seems to be simply pissed that sqlite has the mindshare even without having their favorite features.

gw99 4y ago

This smells like "I don't like how SQLite because I can't get in to the party so I'm making my own party" which is fine but don't denigrate the product or the contributors who have been doing an amazing job for a very long time.

diffxx 4y ago

> The few core developers they have do not work with modern tools like git and collaboration tools like Github, and don’t accept contributions, although they may or may not accept your suggestion for a new feature request. The funny thing about this comment is that SQLite is as close to the gold standard of software quality that we have in the open source world. SQLite is the only program that I've ever used that reliably gets better with every release and never regresses. Could it be that this is precisely because they don't use "modern tools" and accept outside contributions?

Ar-Curunir 4y ago
You're conflating two different arguments: not using modern tools, and not accepting outside contributions. It's certainly possible that limiting contributions to a set of trusted contributors helps things move smoothly. However it's not clear at all that using old tools has the same effect.
- notRobot 4y ago
  They don't use old tools, they use different tools. Git development started in 2005, Fossil development started in 2006, they'ré equally modern, and largely do the same things, just in different ways.
  - diffxx 4y ago
    Also, Fossil is essentially implemented _in_ SQLite. Fossil is used to develop SQLite which is used to implement Fossil. It's a virtuous cycle. For the SQLite project, using Fossil is obviously superior to git. This doesn't mean that arbitrary projects should use Fossil over git.
    - ThunderSizzle 4y ago
      The opposite can be said though, but git arbitrarily won over a bunch of other DVCS systems (another e.g. hg), mainly because of bandwagon and marketing.

resoluteteeth 4y ago

All of the projects listed are clearly outside the scope of what would potentially be merged into sqlite even if it accepted contributions. That said there is probably room for someone to maintain a sqlite extension distribution with standard build settings for use across different languages or something.

outworlder 4y ago

Ew. If you want to say something, say it. "I don't like that my contributions aren't accepted, so I'm forking the codebase". That's going to generate some discussion, but it's fine. Public domain and all. However, "look what happened to qemu, same thing will happen to sqlite (and I'm contributing to the problem and forking it)". One can't say "no contributions led to fragmentation" while _at the same time_ contributing to fragmentation by making a hard fork! Also, > "However, edge computing also means that your code will be running in many geographical locations, as close as possible to the user for the better possible latency. Which means that the data that hits a single SQLite instance now needs to be replicated to all the others." Then replicate away. Leave that stuff out of SQLite. If that's a really important use-case, go use couchdb or something similar.

chasil 4y ago
It's wonderful that there is a fork of SQLite. It's good to see new ideas. Is Airbus going to use it in the A350? No. Why not? It's not compliant with DO-178B, because it has not been confirmed correct with "Test Harness 3" (TH3).
- dathinab 4y ago
  people are not annoyed by anyone forking SQLite they are annoyed by people misrepresenting facts in a subtle manipulative way to make their fork and the reasons around it look like something it isn't what the article intentionally or unintentionally conveys in its tone and formulation is: "sqlite is badly outdate beyond saving and needs to be replaced" what actually is the case: "we want something like SQLite but with some core changes incompatible with SQLites core principles which should be API compatible enough to allow reusing a lot of DB tooling".

nowtern 4y ago

Just looking through the libsql repository on GitHub, turns out they've changed literally zero actual code. All they've done is added a README, some long-winded 'code of conduct' document, and a configuration file for CI builds. That's it. What is even the point of announcing this? There's nothing new, nothing different. Just empty promises and hot air.

glommer 4y ago
yes, we haven't changed any code yet. There are essentially two approaches here: one of them is to present a finished thing, and the other is to announce your intentions as soon as possible and build every single thing in public. We chose the latter in this case, as we think community is the most important aspect of this. but you are right to be suspicious! recommend checking back later
- throwaway0x7E6 4y ago
  in other words, the only thing you had to contribute was the coc.
- Kwpolska 4y ago
  No code changes and inflammatory/defamatory blog posts aren’t a good way to build a community. If you had shown some code that solves a real problem SQLite has, then people would take you seriously. But if all your “contributions” consist of ideas that 99% of SQLite users won’t need, and a few text files, your project doesn’t look too serious to me.
  - chasil 4y ago
    I hope both projects are successful. I'm going to reserve judgment; perhaps something profound will emerge.
- DoneWithAllThat 4y ago
  What does “build everything in public” mean? You’re literally describing open source software which SQLite already is.

gray_-_wolf 4y ago

> they have do not work with modern tools like git Isn't fossil younger than git?

sgbeal 4y ago
> Isn't fossil younger than git? They were initially developed in parallel. To the very best of my recollection, Richard had no knowledge of git when he started fossil. Fossil's initial development dates back to 2005, IIRC, (edit: maybe 2006) initially as a TCL prototype. It wasn't developed in earnest in C until 2007 and was self-hosting by the end of 2007 (i discovered it around Christmas that year).

gjvc 4y ago

This will be a flop.

cryptonector 4y ago
Any fork of SQLite will need serious funding to replace SQLite itself.

as4hretret 4y ago

tarasglek 4y ago

SQLite has a lot of rough edges. As amazing as it is, it would be better if the core devs were more open to outside world. I contributed a somewhat deep performance fix/workaround, it was one of the more terrible open source experiences. Hugely excited for libsql as an sqlite successor with a more modern architecture.

glommer 4y ago
Would love to see your contributions on libSQL!

xet7 4y ago

> The few core developers they have do not work with modern tools like git and collaboration tools like Github, and don’t accept contributions, although they may or may not accept your suggestion for a new feature request. Git is not modern. It is many executeables, and Git repos are a many files. GitHub is not modern. It is closed source huge RoR application. Fossil SCM is features of Git and GitHub much improved, and in small portable fast executeable, storing repos in much smaller space in one SQLite file. Apache is not https://copyfree.org https://copyfree.org license. I prefer copyfree licenses.

Liquid_Fire 4y ago
> Git is not modern. It is many executeables, and Git repos are a many files. What makes many executables and many repo files "not modern"?

jpgvm 4y ago

One one hand, SQLite would probably be a more expansive project if it allowed contributions in the same way PostgreSQL does. On the other hand it would likely make it both a larger and more complex project. I think a better tradeoff would be making the case for why SQLite should have the hooks needed to implement the edge computing bits desired without forking the project itself or attempting to replace what has made it so great for so many years. This approach would allow the extra behaviors to be managed out of tree and allow more free experimentation as a result without prematurely landing on "blessed" approaches to these very complex and frankly not yet well solved problems.

chasil 4y ago
Microsoft paid SQLite for Windows extensions to the database. Symbian paid SQLite for the original mobile functionality. AOL was the original customer for extending SQLite. This does not appear to be an unresponsive project. It just has to be their way, if it's their code.

Joel_Mckay 4y ago

If someone cares about their project, and chooses to share it with the community... no one should feel entitled to pressure them into another development model... I find some of SQLite's library could be cleaner, but appreciate the philosophy of keeping it consistent, understandable, and simple... It is nice to have something reliable completed in an afternoon. =)

wg0 4y ago

What they are missing here is that with QEMU only emulating and not virtualising in a way, was a far bigger limitation counterpart of which does not exist in SQLite. SQLite delivers for the process embedded database promise and delivers it well. Sure, can have column oriented flavour like DuckDB etc but they don't call themselves SQLite or its fork. Lastly, like many pointed out, SQLite is not even licensed (Apache, MIT etc) it is just in public domain so anything is possible.

losfair 4y ago

This project looks really exciting! I'm working on mvsqlite [1], a distributed SQLite based on FoundationDB. When doing the VFS integration I have always wanted to patch SQLite itself, but didn't because of uncertainty around correctness of the patched version... A few features on my wishlist: 1. Asynchronous I/O. mvsqlite is currently doing its own prefetch prediction that is not very accurate. I assume higher layers in SQLite have more information that can help with better prediction. 2. Custom page allocator. SQLite internally uses a linked list to manage database pages - this causes contention on any two transactions that both allocate or free pages. 3. Random ROWID, without the `max(int64)` row trick. Sequentially increasing ROWIDs is a primary source of contention, and causes significant INSERT slowdown in my benchmark [2]. [1] https://github.com/losfair/mvsqlite https://github.com/losfair/mvsqlite [2] https://univalence.me/posts/mvsqlite-bench-20220930 https://univalence.me/posts/mvsqlite-bench-20220930

chasil 4y ago
My guess is that, with the oncoming changes for concurrent writers, these will be implemented in time. Still, Rome was not built in a day, and functionality must take a backseat to reliability and legacy compatibility. https://www.sqlite.org/cgi/src/doc/begin-concurrent/doc/begin_concurrent.md https://www.sqlite.org/cgi/src/doc/begin-concurrent/doc/begi...

justin66 4y ago

I wonder if they'll attempt to replicate the portion of SQLite's test harnesses [0] that are proprietary but nevertheless beneficial to all SQLite users, tests comprising over a million lines of C code. Those tests are possibly among what the blog author does not consider to be "modern tools," which I don't even... [0] https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html

cryptonector 4y ago
That would be very expensive, and would require extensive funding, and would take years to complete.

kazinator 4y ago

Re: > * We would like to use io_uring and asynchronous interfaces > * We would like to provide replication hooks for distributed systems > * We would like to allow for WASM-based user-defined functions This shit isn't needed in SqLite any more than QEMU needs to have anything whatsoever to do with virtualization.

ENOTTY 4y ago

Frankly, as someone more interested in emulation than virtualization, I have been occasionally unhappy that most of the energy behind QEMU goes to virtualization. Some of the things QEMU has done to better adapt to virtualization do not translate well to the emulation use case.

bonzini 4y ago
This is not true, for example a lot of the work to enable concurrent emulation of multiple CPUs started on the virtualization side. It is also mentioned in the article that both emulation and virtualization benefited from the innovation.

mrtweetyhack 4y ago

Am4TIfIsER0ppos 4y ago

> do not work with [...] tools like Github feature > with a clear code of conduct bug > Contributor Covenant Code of Conduct terminal infection

edfletcher_t137 4y ago

This whole thing reads like a slighted, uppity, smarter-than-thou* child taking their ball and going home. It's off-putting. *what's the point of pointing out other "geniuses": to encourage the reader to think "oh well then our author must be in the same group too!"? Transparently manipulative.

thekozmo 4y ago

Listening to SQLite creator podcast (https://corecursive.com/066-sqlite-with-richard-hipp/# https://corecursive.com/066-sqlite-with-richard-hipp/# ), it does feel that Glauber is right about weird collaboration standards. The guy is against gmail, git, etc. Fossil may be better for SQlite today, w/o many contributors, that's the problem Glauber is trying to solve

deleted 4y ago
[deleted]

jasonhansel 4y ago

SQLite and QEMU have been successful in large part because of their limited scope. Further expanding the scope of a project puts it at risk of becoming unfocused and unmaintainable. It makes a lot more sense to start a new project to support this use case than to try to reshape SQLite.

throwawaylinux 4y ago

Paravirtualization was not pioneered by Xen in 2004, that was probably done by IBM with mainframe virtualization in the early 70s, if not before.

dathinab 4y ago

> do not work with modern tools like git That is VERY misleading and intentionally so, doesn't put a good light on the author. Sure SQLite isn't open for contribution but their tolling (VCS, issue tracking) is in no way "less modern" then git+github, it is just focused on a different approach. Putting a widely used open source but not open contribution project on GitHub is a nightmare. Similar if your are having no open contribution and only a small trusted team then you can use change flows not viable otherwise and their VCS is designed for such design flows for which git has quite a bunch of unnecessary overhead and food guns. Lastly for many project choosing no open contribution is a very sane approach properly maintaining contributions for a widely used project is a lot of work basically forcing you to delegate a lot of work to people you hardly know. If you don't have time for this or don't feel good with trusting people you hardly know then open contribution can easily become a night mare. The fact that you AFIK have to be very disciplined to write C doesn't make this better. Putting this aside another thing I found of putting in the article is that the author fails to mention that basically all of the things he wants to have in SQLit are in direct conflict with core design principles of SQLite. So they need a fork anyway independent of weather or not SQLite accepts contributions, their changes wouldn't be accepted anyway... Through without question for their use cases a new database is needed which is similar to SQLite but also in subtle but very fundamental ways very different. Starting it with a fork doesn't seem a bad idea. But they really should not say/imply it's SQLite it's not sqlite, it's just very similar and once was forked of SQLite.

kmavm 4y ago

The idea that "virtualization" began with Zen in 2004 is rather difficult to read as an early VMware employee. Before QEMU independently discovered it, VMware was JIT'ing unrestricted x86 to a safe x86 subset from 1999 on[1]. Hardware support for trap-and-emulate virtualization came to the market in the early 'aughts after VMware had proven the market demand for it. [1] https://www.vmware.com/pdf/asplos235_adams.pdf https://www.vmware.com/pdf/asplos235_adams.pdf

zackees 4y ago

draw_down 4y ago

larsnystrom 4y ago

I believe the reason this post rubs so many the wrong way is because the author keeps praising his “opponents”, trying to sound positive while being inherently negative. It’s an ugly way of argumenting as it forces the other side to be the negative one, when in fact the author is the one who’s trying to bring something down. In the end the author just sounds dishonest, like a politician with a smile plastered over their face all the time. I believe there are two ways this post could have been written which would have given a better impression of the author and his intentions. Either openly criticize SQLite, with well-formulated arguments, or don’t criticize SQLite and just announce that you’re doing something new (building on top of SQLite is not at all uncommon after all).

huino 4y ago

There's an interesting set of comments by keithmadams on Twitter https://twitter.com/keithmadams/status/1577483553281310721 https://twitter.com/keithmadams/status/1577483553281310721, regarding the accuracy of this: > This article's history of "virtualization" begins with paravirt in Zen in 2004. Then, after hardware support emerged, v12n started working with with unmodified OSes. No part of this is true. VMware was running binary OSes without hardware support in 1999. > We did this by JIT'ing unrestricted x86 to a safe x86 subset. User-level code, with few exceptions, was directly executed, but kernel-level code ran through a little JIT. We described it in https://www.vmware.com/pdf/asplos235_adams.pdf https://www.vmware.com/pdf/asplos235_adams.pdf > It performs pretty well, as described above. The only problem is that tiny JITs that run in kernel mode and use x86 segmentation to hide themselves in the top 4MB of linear memory are ... hard to write. And VMware was making a lot of money. > Zen invented paravirt; AMD invented SVM; and Intel invented VT in parallel all around 2003-2004, not to make virtualization on x86 practical, but to make it easy enough to disrupt VMware's emerging monopoly. > As a VC focused on companies exploiting a tech advantage, VMware remains my favorite case study. For a time, they could do something that was undeniably valuable, that nobody else could do. And, moats like that are always time-limited; their value invites their destruction.

Dowwie 4y ago

Can any of the contributors to ChiselStore [1] , mentioned in the article, elaborate on the comment made about it? What's the big deal about not fitting well with SQLx? This isn't a requirement for such a library to be successful. What other problems is it allegedly suffering from? It could be more of a problem of perspective. [1] https://github.com/chiselstrike/chiselstore https://github.com/chiselstrike/chiselstore

zvr 4y ago

From the README in https://github.com/libsql/libsql https://github.com/libsql/libsql: > Could SQLite be embedded in the Linux kernel? Choosing to license the fork under Apache-2.0 ensures that this will not happen (via this fork, at least), as the license is considered incompatible with the kernel's GPL-2.0-only license.

yencabulator 4y ago

> LiteFS: [...] I personally think that if distributed filesystems were easy, we’d have a good one by now. The part that makes that "easy" for LiteFS is that it's a single-writer system. That's a pretty rare trade-off in general purpose distributed filesystems.