4 ms·

Why do you need Kafka, Redis, Mongo, AND Postgres? Arguably it could all be done with Postgres or Redis: both have great pubsub implementations. But what could

Why do you need Kafka, Redis, Mongo, AND Postgres? Arguably it could all be done with Postgres or Redis: both have great pubsub implementations. But what could you possibly need two databases for and a queue implementation alongside a popular backend for queues? I don't mean to criticize, but the complexity of this is what I'd expect for something handling hundreds if not thousands of qps. Even then I'd expect you could probably fine-tune a Redis Cluster setup and cross out 2/3 of that architecture diagram. Hell, you don't even need WebSockets: a system like this polling for an integer or two should be scalable to tens of thousands of qps.

nevodavid10 3y ago
As for Postgres, I chose Timescale because you can easily change it at any time for Postgres. I plan to have it as a managed service in the future, which means that it will hold a lot of clients - some big, some small. But I think I should def make another smaller version without Kafka and Timescale.
- bee_rider 3y ago
  When it is a managed service, will a single individual be feeding into a single sort of taste profile over multiple sites? Actually am I crazy to thing that wouldn’t be the worst thing in the world? Tracking people across multiple sites without their informed consent is of course totally unethical spying, but if it is clear that they are consciously interacting with UI like yours and they know it is shared, that doesn’t seem so bad. (?) And it could be nice if Pandora would start suggesting soundtracks to movies I’ve upvoted on Amazon or something like that…
  - badtuple 3y ago
    It's hard enough to view the content I want to see when everyone has "smart" content feeds shoving ads in my face. I work hard to isolate what I look at on Amazon from what I see on YouTube so marketers don't ruin the little bits of the web I'm able to enjoy intentionally. Creating cross-site bleed of account information behind the scenes just makes everything worse for the user who can no longer drive your app...they just have to plop in and let it drive them. You handwaved the idea of consent by saying it's "clear" to the user. But if there isn't a way to opt out then consent isn't addressed at all.
- bastawhiz 3y ago
  Even if you were managing this for third parties, there's no reason to assume your scale would necessitate this complexity. One of my Postgres databases stores analytics data for podcasts. 800gb worth in two tables. It's just vanilla Postgres with indexes. P90 queries complete in 100ms. I serve >10k customers with one write instance and one read-only replica. Some podcasts get thousands of listens per second at peak times. Stripe relied entirely on Mongo for storage and queueing at a scale that you'll almost assuredly never reach. That's probably inadvisable today, but the point is that the more systems you have, the more systems you need to maintain and the more things that can break and the more places you'll eventually find bottlenecks.
- upon_drumhead 3y ago
  You chose timescale because you can swap it out for Postgres? Why not start with Postgres then? Is there anything timescale provides that is beneficial over Postgres?
astura 3y ago
Résumé Driven Development.
- nefarious_ends 3y ago
  oh wow this is a term I’ve been looking for for ages!
  - nevodavid10 3y ago
    Haha, it's def me!
klysm 3y ago
That is an absurd set of technologies to choose. The cost of having multiple persistent backends is so high I’ve never understood the motivation.
ransom1538 3y ago
[flagged]
- wizzwizz4 3y ago
  Some tools are good for some jobs. Other tools are good for other jobs. There's no "war".
- pjot 3y ago
  pg_dump --schema-only --table=tablename databasename
  - renewiltord 3y ago
    I have to say that this style of pointless tech flamewar is actually quite refreshing to encounter. It's like Slashdot in the old days Linux v BSD. Someone fires off a command or a patch that adds the functionality. Sort of nostalgic. Ultimately, though, it would be nice to have a `.schema` equivalent in `psql` just as a convenience tool.
- bastawhiz 3y ago
  This is an awfully petty hill to die on, especially for this website. The material difference between Postgres and MySQL for the overwhelming majority of users is insignificant. They're both good pieces of software and there are hardly any features missing in either that anyone who's not a DBA would miss when comparing one to the other. It's like arguing against using Firefox because Chrome "won the war". Who do you think you're convincing?
shri_krishna 3y ago
Even better is to just use Clickhouse (maybe Kafka if there are lots of inserts and async_insert isn't sufficient) and Postgres.
- nevodavid10 3y ago
  Thought about Clickhouse without Kafka even because it can handle streams of data. But now sure what to do on failures.

nevodavid10 3y ago

As for Postgres, I chose Timescale because you can easily change it at any time for Postgres. I plan to have it as a managed service in the future, which means that it will hold a lot of clients - some big, some small. But I think I should def make another smaller version without Kafka and Timescale.

bee_rider 3y ago
When it is a managed service, will a single individual be feeding into a single sort of taste profile over multiple sites? Actually am I crazy to thing that wouldn’t be the worst thing in the world? Tracking people across multiple sites without their informed consent is of course totally unethical spying, but if it is clear that they are consciously interacting with UI like yours and they know it is shared, that doesn’t seem so bad. (?) And it could be nice if Pandora would start suggesting soundtracks to movies I’ve upvoted on Amazon or something like that…
- badtuple 3y ago
  It's hard enough to view the content I want to see when everyone has "smart" content feeds shoving ads in my face. I work hard to isolate what I look at on Amazon from what I see on YouTube so marketers don't ruin the little bits of the web I'm able to enjoy intentionally. Creating cross-site bleed of account information behind the scenes just makes everything worse for the user who can no longer drive your app...they just have to plop in and let it drive them. You handwaved the idea of consent by saying it's "clear" to the user. But if there isn't a way to opt out then consent isn't addressed at all.
bastawhiz 3y ago
Even if you were managing this for third parties, there's no reason to assume your scale would necessitate this complexity. One of my Postgres databases stores analytics data for podcasts. 800gb worth in two tables. It's just vanilla Postgres with indexes. P90 queries complete in 100ms. I serve >10k customers with one write instance and one read-only replica. Some podcasts get thousands of listens per second at peak times. Stripe relied entirely on Mongo for storage and queueing at a scale that you'll almost assuredly never reach. That's probably inadvisable today, but the point is that the more systems you have, the more systems you need to maintain and the more things that can break and the more places you'll eventually find bottlenecks.
upon_drumhead 3y ago
You chose timescale because you can swap it out for Postgres? Why not start with Postgres then? Is there anything timescale provides that is beneficial over Postgres?

astura 3y ago

Résumé Driven Development.

nefarious_ends 3y ago
oh wow this is a term I’ve been looking for for ages!
- nevodavid10 3y ago
  Haha, it's def me!

klysm 3y ago

That is an absurd set of technologies to choose. The cost of having multiple persistent backends is so high I’ve never understood the motivation.

ransom1538 3y ago

[flagged]

wizzwizz4 3y ago
Some tools are good for some jobs. Other tools are good for other jobs. There's no "war".
pjot 3y ago
pg_dump --schema-only --table=tablename databasename
- renewiltord 3y ago
  I have to say that this style of pointless tech flamewar is actually quite refreshing to encounter. It's like Slashdot in the old days Linux v BSD. Someone fires off a command or a patch that adds the functionality. Sort of nostalgic. Ultimately, though, it would be nice to have a `.schema` equivalent in `psql` just as a convenience tool.
bastawhiz 3y ago
This is an awfully petty hill to die on, especially for this website. The material difference between Postgres and MySQL for the overwhelming majority of users is insignificant. They're both good pieces of software and there are hardly any features missing in either that anyone who's not a DBA would miss when comparing one to the other. It's like arguing against using Firefox because Chrome "won the war". Who do you think you're convincing?

shri_krishna 3y ago

Even better is to just use Clickhouse (maybe Kafka if there are lots of inserts and async_insert isn't sufficient) and Postgres.

nevodavid10 3y ago
Thought about Clickhouse without Kafka even because it can handle streams of data. But now sure what to do on failures.