3 ms·

>Discussing the technologies or patterns involved without building anything and especially without building a complete working product with your favourite elega

by simonh 4y ago

>Discussing the technologies or patterns involved without building anything and especially without building a complete working product with your favourite elegant tool.... But surely, if anywhere, this is the place to talk to people who have built complete working system, are building them, and intend to build more of them. This isn't a forum for a bunch of undergrads chatting about stuff they just learned in college. I'm sure there are some of those here, but if you're going to find seasoned web engineers anywhere, this is a good place to start. > Let's say you use a basic HTML Golang app backed by SQLite... therefore your website is simple, usable, never goes down. That's a complete non-sequitur. If the app becomes hugely popular, needs to serve massive volumes of data and sync across thousands of instances, well, it won't. You can't say that because it uses simple tech therefore it will be wonderful, that's not how that works. There are use cases where the MongoDB + Kafka thing will work much better. That's what these discussions are about.

ratww 4y ago
> If the app becomes hugely popular, needs to serve massive volumes of data and sync across thousands of instances, well, it won't This argument is so overused for justifying any kind of complexity that it would border on comical if it weren't causing so many issues in our industry. There is absolutely no guarantee that the MongoDB + Kafka will also work with the hypothetical "massive volumes of data" you're talking about. Those two tools aren't set-and-forget: they require fine tuning and a good architecture to begin with. Ironically, I've seen plenty of cases where MongoDB couldn't handle the production load and had to be put behind some Varnish or Nginx disk cache (ironically bringing it closer to the SQLite app example). But we don't even have to go that far: depending on the situation, there is a huge chance that the hypothetical app won't see "massive amounts of data" anyway. And if it does, there's an also massive chance that it won't happen overnight, so there will be time to build more. And if we get there, the SQLite app can just be bumped up to PostgreSQL or something similar, which scales vertically beautifully, or some other relational DB that scales horizontally.
- simonh 4y ago
  Right, but if you're scaling it up to meet changing requirements, well, that's now a different use case. How do you do that scaling up effectively? That's where discussion here comes in. I'm not at all arguing that you need MongoDB and Kafka, those are not my examples, but pretending that SQLite and plain HTML are the be-all, end-all is bizarre. OK, you're a big fan of PostgreSQL. Cool. Now we're talking. In what situations and why is a useful discussion to have. That's grounds for fertile and relevant architectural debate, and here is a good place to do that. Arguing that isn't the case is, well, ok... problem solved then. I suppose there's not much more to talk about.
  - ratww 4y ago
    I don't think GP is claiming that SQLite is the "be-all, end-all" for all cases. But it definitely is for when a website doesn't need to scale, which are the vast majority of cases. Disks (or rather SSDs + memory backed cache) are extremely fast. For cases when it doesn't work anymore, there's an easy migration path to whatever relational DB that I mentioned. The argument isn't that SQLite is the "be-all, end-all", it's more that MongoDB, Kafka and other tools are the opposite of it: they don't hit a sweet spot for small sites (and also often doesn't for growing ones, or or big ones too), so there's no much point in starting there, as you'll only get problems and might hamper the website growth.
strken 4y ago
> That's a complete non-sequitur. It doesn't need to sequitur. You disagree with the (unimportant, arbitrary, and probably wrong) choice of "good" technology, for the (valid, fair, and important) reason of scalability. This is fine: whether a specific technology is fit for a hypothetical purpose is irrelevant to what I believe the article is saying. I don't read the article as saying "never talk about the technology behind scalability on HN or anywhere else," I read it as saying "people learn scalability from exemplars, so the way you improve overall scalability is by finding better examples, not discussing what's already out there." The reason we have use-cases of Kafka scaling is because LinkedIn[0] needed durable distributed pubsub, so they built it. When the linked article was written it was handling 500 billion events a day. That was kind of cool, because 500 billion is a big number, but also kind of shocking, because that was 68 daily events for each human being alive at the time. Perhaps in 2022 with the hindsight of the last decade, someone with different problems and priorities could approach scalability from a different angle and come up with something else that works better for them. [0] https://engineering.linkedin.com/architecture/brief-history-scaling-linkedin https://engineering.linkedin.com/architecture/brief-history-...

by simonh 4y ago

ratww 4y ago

> If the app becomes hugely popular, needs to serve massive volumes of data and sync across thousands of instances, well, it won't This argument is so overused for justifying any kind of complexity that it would border on comical if it weren't causing so many issues in our industry. There is absolutely no guarantee that the MongoDB + Kafka will also work with the hypothetical "massive volumes of data" you're talking about. Those two tools aren't set-and-forget: they require fine tuning and a good architecture to begin with. Ironically, I've seen plenty of cases where MongoDB couldn't handle the production load and had to be put behind some Varnish or Nginx disk cache (ironically bringing it closer to the SQLite app example). But we don't even have to go that far: depending on the situation, there is a huge chance that the hypothetical app won't see "massive amounts of data" anyway. And if it does, there's an also massive chance that it won't happen overnight, so there will be time to build more. And if we get there, the SQLite app can just be bumped up to PostgreSQL or something similar, which scales vertically beautifully, or some other relational DB that scales horizontally.

simonh 4y ago
Right, but if you're scaling it up to meet changing requirements, well, that's now a different use case. How do you do that scaling up effectively? That's where discussion here comes in. I'm not at all arguing that you need MongoDB and Kafka, those are not my examples, but pretending that SQLite and plain HTML are the be-all, end-all is bizarre. OK, you're a big fan of PostgreSQL. Cool. Now we're talking. In what situations and why is a useful discussion to have. That's grounds for fertile and relevant architectural debate, and here is a good place to do that. Arguing that isn't the case is, well, ok... problem solved then. I suppose there's not much more to talk about.
- ratww 4y ago
  I don't think GP is claiming that SQLite is the "be-all, end-all" for all cases. But it definitely is for when a website doesn't need to scale, which are the vast majority of cases. Disks (or rather SSDs + memory backed cache) are extremely fast. For cases when it doesn't work anymore, there's an easy migration path to whatever relational DB that I mentioned. The argument isn't that SQLite is the "be-all, end-all", it's more that MongoDB, Kafka and other tools are the opposite of it: they don't hit a sweet spot for small sites (and also often doesn't for growing ones, or or big ones too), so there's no much point in starting there, as you'll only get problems and might hamper the website growth.

strken 4y ago

> That's a complete non-sequitur. It doesn't need to sequitur. You disagree with the (unimportant, arbitrary, and probably wrong) choice of "good" technology, for the (valid, fair, and important) reason of scalability. This is fine: whether a specific technology is fit for a hypothetical purpose is irrelevant to what I believe the article is saying. I don't read the article as saying "never talk about the technology behind scalability on HN or anywhere else," I read it as saying "people learn scalability from exemplars, so the way you improve overall scalability is by finding better examples, not discussing what's already out there." The reason we have use-cases of Kafka scaling is because LinkedIn[0] needed durable distributed pubsub, so they built it. When the linked article was written it was handling 500 billion events a day. That was kind of cool, because 500 billion is a big number, but also kind of shocking, because that was 68 daily events for each human being alive at the time. Perhaps in 2022 with the hindsight of the last decade, someone with different problems and priorities could approach scalability from a different angle and come up with something else that works better for them. [0] https://engineering.linkedin.com/architecture/brief-history-scaling-linkedin https://engineering.linkedin.com/architecture/brief-history-...