5 ms·

I've noticed a large number of stories on HN related to SQLite over the past few weeks. Maybe it's just random or I'm only now just noticing it, but is there so

I've noticed a large number of stories on HN related to SQLite over the past few weeks. Maybe it's just random or I'm only now just noticing it, but is there some renewed/newfound interest in SQLite lately? If so, what's behind that?

Jarwain 4y ago
I find topics tend to surge and wane, and sqlite is a popular one. They don't always surge for a reason other than "someone posted something new about it and sent people down rabbit holes"
IshKebab 4y ago
Yeah definitely renewed interest. I think there's been a collective realisation that it's pretty good, and it has received some notable new features recently like JSON queries and proper type checking.
sgbeal 4y ago
> ... is there some renewed/newfound interest in SQLite lately? If so, what's behind that? Though i share that observation, in this particular case it's very likely coincidence. i've worked with Richard since 2008 on his Fossil SCM project, so have been "in that circle" for a long time without having ever actually contributed directly to sqlite. About two weeks ago the topic of wasm came up in a dev chat and it sounded to me like something interesting to experiment with (me being Fossil's "JavaScript Guy"), so i ran with it.
krferriter 4y ago
I think some people are realizing that SQLite has everything they need in their database use case, and they don't need a heavier, more complicated client-server database with more features.
- bob1029 4y ago
  This is exactly the reason. Some of us went off the beaten path and proved that you can actually put many users on a carefully-tuned SQLite instance (despite their official docs at the time). In many cases, you are able to exceed the capabilities of a one-node hosted DB solution due to latency reduction of in-process happiness.
- xwowsersx 4y ago
  Thanks. I thought that may have been the case, but I wasn't sure if there were also some recent improvements or developments.
matthewaveryusa 4y ago
I think so. I attribute it to dqlite, rqlite and litestream that are making sqlite viable server side. It's always been viable server-side in tandem with a paxos/raft changestream (etcd streaming the changelog to an sqlite database.) but with the new projects providing out-of-the-box replication solutions it's even more-so viable. It brings me so much joy to see sqlite get the love it deserves -- it's by far the best library I've ever used.
rzzzt 4y ago
I think it's this one (for context, the submission title was "Fly.io buys Litestream" before changing to the blog post title; you can still find it indexed that way in search engines): https://news.ycombinator.com/item?id=31318708 https://news.ycombinator.com/item?id=31318708
srcreigh 4y ago
SQLite is quite famous for new developers as the default db in Rails and Django. There's a widespread misconception that SQLite is a toy DB and that it's necessary to switch to a real DB like Postgres or MySQL. I suspect this fuels some of the hype--trying to correct the misconception.
- rrix2 4y ago
  well, let's be honest here: some of this is legitimate due to the nature of those frameworks, not necessarily sqlite. If you deploy a rails app with 8 process workers powered by a sqlite db even on a single host you're going to have database locking problems. You have to be able to architect your application to send all write traffic to a single process and have a language that can keep up with your database while doing that. There are "better" architectures for this stuff now that will enable that which have come together in the last few years, and a resurgence in smaller indie projects willing to deploy sqlite on servers and do weird cool things like this fiddle or datasette
  - sgbeal 4y ago
    > You have to be able to architect your application to send all write traffic to a single process ... Counter-point: sqlite's sister project and SCM, the Fossil SCM, is a self-hosting sqlite client application and has been since 2007. Every hit on the fossil-scm.org website is its own standalone fossil process accessing the same copy of the same db file, all while the developers are actively pushing and pulling changes to/from that same db and while half a dozen or so folks are logged in to its /chat room (each instance of which is long-polling that same db 24/7). In 14 years of using that db i've encountered _maybe_ two locking errors. Similarly, sqlite3's own forum is a fossil instance hosting a single sqlite3 repository. That all writes "have" to be channeled through a single manager is demonstrably not the case.
    - rrix2 4y ago
      Does fossil or the SQLite forum use rails or Django?
dinvlad 4y ago
I think this may also have to do with the indie community - SQLite seems to literally fit the bill much better than hosted DBs. A lot of what "the biggest companies" do doesn't go anywhere close to where most small (or even medium) teams' needs are. A lot of that additional complexity is simply not worth the extra effort and manpower (and oftentimes creates it!). "Lean programming" has become cool again :-)
dgb23 4y ago
SQLite has been improving quite significantly over the years and continually impresses people in terms of performance, features and robustness. When I first heard about it 10y ago, my boss (web shop) said it "isn't a real database", and that notion prevails in many circles. However more and more web developers recognize its benefits for production use. There are many use cases for it. Most web projects should default to it IMO, because of its low operational costs and great performance, both in terms speed and reliability. It's incredibly straight forward to use, set up, back up and so on. As an example: Wordpress, could absolutely default to SQLite as the vast majority of installations run on a single, shared host (apache/litespeed/nginx) and the data model is very simple and almost throughout stringly typed anyways. Running MySQL just adds friction and overhead for these use cases.
fideloper 4y ago
Lots of new stuff happening lately, yep! Mostly around distributed systems? I've been keeping track of a few things here: https://gist.github.com/fideloper/ac9b81cee85003a59c8ad1a591d5aa6d https://gist.github.com/fideloper/ac9b81cee85003a59c8ad1a591...
ludwigvan 4y ago
Probably a reaction to the ever growing complexity of software systems.
heavyset_go 4y ago
It was a meme in the "webscale" era that SQLite couldn't scale to many users, and was effectively something to use in desktop and client-side applications like mobile apps. There have been times I've been talked down to in the past for using SQLite, so I just learned to shut up about it around ideologues. Now the tide is turning when it comes to webscale assumptions, and some of those ideologues' ideas have been proven wrong in many aspects. SQLite performs well in read-heavy loads, even in web apps with many users, so it can fit those use cases well. I've been using it in a reverse search engine that gets a lot of traffic for going on 8 years now.
- bdlowery 4y ago
  “Sqlite doesn’t scale” https://twitter.com/levelsio/status/1520357256373874688 https://twitter.com/levelsio/status/1520357256373874688 Dude gets 172 million requests a month.
- edgyquant 4y ago
  To be fair, during that time the NoSQL crowd was saying that about all relational databases.
  - heavyset_go 4y ago
    Yeah, at the time SQLite existed both as an archetypal example of "relational databases bad" and a case of "especially this one that isn't even a 'real' RDBMS".

Jarwain 4y ago

I find topics tend to surge and wane, and sqlite is a popular one. They don't always surge for a reason other than "someone posted something new about it and sent people down rabbit holes"

IshKebab 4y ago

Yeah definitely renewed interest. I think there's been a collective realisation that it's pretty good, and it has received some notable new features recently like JSON queries and proper type checking.

sgbeal 4y ago

> ... is there some renewed/newfound interest in SQLite lately? If so, what's behind that? Though i share that observation, in this particular case it's very likely coincidence. i've worked with Richard since 2008 on his Fossil SCM project, so have been "in that circle" for a long time without having ever actually contributed directly to sqlite. About two weeks ago the topic of wasm came up in a dev chat and it sounded to me like something interesting to experiment with (me being Fossil's "JavaScript Guy"), so i ran with it.

krferriter 4y ago

I think some people are realizing that SQLite has everything they need in their database use case, and they don't need a heavier, more complicated client-server database with more features.

bob1029 4y ago
This is exactly the reason. Some of us went off the beaten path and proved that you can actually put many users on a carefully-tuned SQLite instance (despite their official docs at the time). In many cases, you are able to exceed the capabilities of a one-node hosted DB solution due to latency reduction of in-process happiness.
xwowsersx 4y ago
Thanks. I thought that may have been the case, but I wasn't sure if there were also some recent improvements or developments.

matthewaveryusa 4y ago

I think so. I attribute it to dqlite, rqlite and litestream that are making sqlite viable server side. It's always been viable server-side in tandem with a paxos/raft changestream (etcd streaming the changelog to an sqlite database.) but with the new projects providing out-of-the-box replication solutions it's even more-so viable. It brings me so much joy to see sqlite get the love it deserves -- it's by far the best library I've ever used.

rzzzt 4y ago

I think it's this one (for context, the submission title was "Fly.io buys Litestream" before changing to the blog post title; you can still find it indexed that way in search engines): https://news.ycombinator.com/item?id=31318708 https://news.ycombinator.com/item?id=31318708

srcreigh 4y ago

SQLite is quite famous for new developers as the default db in Rails and Django. There's a widespread misconception that SQLite is a toy DB and that it's necessary to switch to a real DB like Postgres or MySQL. I suspect this fuels some of the hype--trying to correct the misconception.

rrix2 4y ago
well, let's be honest here: some of this is legitimate due to the nature of those frameworks, not necessarily sqlite. If you deploy a rails app with 8 process workers powered by a sqlite db even on a single host you're going to have database locking problems. You have to be able to architect your application to send all write traffic to a single process and have a language that can keep up with your database while doing that. There are "better" architectures for this stuff now that will enable that which have come together in the last few years, and a resurgence in smaller indie projects willing to deploy sqlite on servers and do weird cool things like this fiddle or datasette
- sgbeal 4y ago
  > You have to be able to architect your application to send all write traffic to a single process ... Counter-point: sqlite's sister project and SCM, the Fossil SCM, is a self-hosting sqlite client application and has been since 2007. Every hit on the fossil-scm.org website is its own standalone fossil process accessing the same copy of the same db file, all while the developers are actively pushing and pulling changes to/from that same db and while half a dozen or so folks are logged in to its /chat room (each instance of which is long-polling that same db 24/7). In 14 years of using that db i've encountered _maybe_ two locking errors. Similarly, sqlite3's own forum is a fossil instance hosting a single sqlite3 repository. That all writes "have" to be channeled through a single manager is demonstrably not the case.
  - rrix2 4y ago
    Does fossil or the SQLite forum use rails or Django?

dinvlad 4y ago

I think this may also have to do with the indie community - SQLite seems to literally fit the bill much better than hosted DBs. A lot of what "the biggest companies" do doesn't go anywhere close to where most small (or even medium) teams' needs are. A lot of that additional complexity is simply not worth the extra effort and manpower (and oftentimes creates it!). "Lean programming" has become cool again :-)

dgb23 4y ago

SQLite has been improving quite significantly over the years and continually impresses people in terms of performance, features and robustness. When I first heard about it 10y ago, my boss (web shop) said it "isn't a real database", and that notion prevails in many circles. However more and more web developers recognize its benefits for production use. There are many use cases for it. Most web projects should default to it IMO, because of its low operational costs and great performance, both in terms speed and reliability. It's incredibly straight forward to use, set up, back up and so on. As an example: Wordpress, could absolutely default to SQLite as the vast majority of installations run on a single, shared host (apache/litespeed/nginx) and the data model is very simple and almost throughout stringly typed anyways. Running MySQL just adds friction and overhead for these use cases.

fideloper 4y ago

Lots of new stuff happening lately, yep! Mostly around distributed systems? I've been keeping track of a few things here: https://gist.github.com/fideloper/ac9b81cee85003a59c8ad1a591d5aa6d https://gist.github.com/fideloper/ac9b81cee85003a59c8ad1a591...

ludwigvan 4y ago

Probably a reaction to the ever growing complexity of software systems.

heavyset_go 4y ago

It was a meme in the "webscale" era that SQLite couldn't scale to many users, and was effectively something to use in desktop and client-side applications like mobile apps. There have been times I've been talked down to in the past for using SQLite, so I just learned to shut up about it around ideologues. Now the tide is turning when it comes to webscale assumptions, and some of those ideologues' ideas have been proven wrong in many aspects. SQLite performs well in read-heavy loads, even in web apps with many users, so it can fit those use cases well. I've been using it in a reverse search engine that gets a lot of traffic for going on 8 years now.

bdlowery 4y ago
“Sqlite doesn’t scale” https://twitter.com/levelsio/status/1520357256373874688 https://twitter.com/levelsio/status/1520357256373874688 Dude gets 172 million requests a month.
edgyquant 4y ago
To be fair, during that time the NoSQL crowd was saying that about all relational databases.
- heavyset_go 4y ago
  Yeah, at the time SQLite existed both as an archetypal example of "relational databases bad" and a case of "especially this one that isn't even a 'real' RDBMS".