3 ms·

> If the app becomes hugely popular, needs to serve massive volumes of data and sync across thousands of instances, well, it won't This argument is so overused

by ratww 4y ago

> If the app becomes hugely popular, needs to serve massive volumes of data and sync across thousands of instances, well, it won't This argument is so overused for justifying any kind of complexity that it would border on comical if it weren't causing so many issues in our industry. There is absolutely no guarantee that the MongoDB + Kafka will also work with the hypothetical "massive volumes of data" you're talking about. Those two tools aren't set-and-forget: they require fine tuning and a good architecture to begin with. Ironically, I've seen plenty of cases where MongoDB couldn't handle the production load and had to be put behind some Varnish or Nginx disk cache (ironically bringing it closer to the SQLite app example). But we don't even have to go that far: depending on the situation, there is a huge chance that the hypothetical app won't see "massive amounts of data" anyway. And if it does, there's an also massive chance that it won't happen overnight, so there will be time to build more. And if we get there, the SQLite app can just be bumped up to PostgreSQL or something similar, which scales vertically beautifully, or some other relational DB that scales horizontally.

simonh 4y ago
Right, but if you're scaling it up to meet changing requirements, well, that's now a different use case. How do you do that scaling up effectively? That's where discussion here comes in. I'm not at all arguing that you need MongoDB and Kafka, those are not my examples, but pretending that SQLite and plain HTML are the be-all, end-all is bizarre. OK, you're a big fan of PostgreSQL. Cool. Now we're talking. In what situations and why is a useful discussion to have. That's grounds for fertile and relevant architectural debate, and here is a good place to do that. Arguing that isn't the case is, well, ok... problem solved then. I suppose there's not much more to talk about.
- ratww 4y ago
  I don't think GP is claiming that SQLite is the "be-all, end-all" for all cases. But it definitely is for when a website doesn't need to scale, which are the vast majority of cases. Disks (or rather SSDs + memory backed cache) are extremely fast. For cases when it doesn't work anymore, there's an easy migration path to whatever relational DB that I mentioned. The argument isn't that SQLite is the "be-all, end-all", it's more that MongoDB, Kafka and other tools are the opposite of it: they don't hit a sweet spot for small sites (and also often doesn't for growing ones, or or big ones too), so there's no much point in starting there, as you'll only get problems and might hamper the website growth.

by ratww 4y ago

simonh 4y ago

Right, but if you're scaling it up to meet changing requirements, well, that's now a different use case. How do you do that scaling up effectively? That's where discussion here comes in. I'm not at all arguing that you need MongoDB and Kafka, those are not my examples, but pretending that SQLite and plain HTML are the be-all, end-all is bizarre. OK, you're a big fan of PostgreSQL. Cool. Now we're talking. In what situations and why is a useful discussion to have. That's grounds for fertile and relevant architectural debate, and here is a good place to do that. Arguing that isn't the case is, well, ok... problem solved then. I suppose there's not much more to talk about.

ratww 4y ago
I don't think GP is claiming that SQLite is the "be-all, end-all" for all cases. But it definitely is for when a website doesn't need to scale, which are the vast majority of cases. Disks (or rather SSDs + memory backed cache) are extremely fast. For cases when it doesn't work anymore, there's an easy migration path to whatever relational DB that I mentioned. The argument isn't that SQLite is the "be-all, end-all", it's more that MongoDB, Kafka and other tools are the opposite of it: they don't hit a sweet spot for small sites (and also often doesn't for growing ones, or or big ones too), so there's no much point in starting there, as you'll only get problems and might hamper the website growth.