5 ms·

Such a server is 400$/mo, a backend developer that can confidently maintain kafka in production is significantly more expensive!

macksd 5y ago
I think the point of interest was 32 cores to handle what sounds like 10 messages per second at most. That's not really a ton of throughput... It's certainly a valid point that an awful lot of uses cases don't need Twitter-scale firehoses or Google-size Hadoop clusters.
- rowanG077 5y ago
  It said nothing about the distribution of traffic. It might well be thousands and thousands of pub sub messages at some point of the day and 0 for others.
- colinchartier 5y ago
  Ah, the database does a lot more than just pub/sub - especially since the high traffic pub/sub goes through redis. I guess my point was that we never regretted setting up postgres as the "default job queue" and it never required much engineering work to maintain. For an example, it handles stripe webhooks when users change their pricing tier - if you drop that message, users would be paying for something they wouldn't receive.
rockwotj 5y ago
Fwiw I don't know the shape of the data, but I feel like you could do this with Firebase for a few bucks a month...
- daenz 5y ago
  you 100% could, and this thread feels like the twilight zone with how many people are advocating for using a rdbms for (what seems like) most peoples queuing needs.
  - tata71 5y ago
    Does Firebase offer self-hosting these days? What do you say to those who don't want Google to know their usage info?
    - pricci 5y ago
      Checkout supabase.com. It is based on postgres.
  - tomc1985 5y ago
    Dude you are seriously underestimating postgres' versatility. It does so many different things, and well!
    - daenz 5y ago
      I'm not underestimating anything. I am advocating for the right tool for the job. I have a hard time believing, despite the skewed sample size in this thread, that most people think using postgres as a message queue for most cases makes the most sense.
      - tomc1985 5y ago
        What is your idea of 'most cases'? I've personally written real-time back-of-house order-tracking with rails and postgres pubsub (no redis!), and wrote a record synchronization queuing system with a table and some clever lock semantics that has been running in production for several years now -- which marketing relies upon as it oversees 10+ figures of yearly topline revenue. Neither of those projects were FAANG scale, but they work fine for what is needed and scale relatively cleanly with postgres itself. Besides, in a lot of environments corporate will only approve the use of certain tools. And if you already have one approved that does the job, then why not?
        daenz 5y ago
        >some clever lock semantics Most senior+ engineers that I know would hear that and recoil. Getting "clever" with concurrency handling in your home-rolled queuing system is not something that coworkers, especially more senior coworkers, will appreciate inheriting, adapting, and maintaining. Believe me. I get that you're trying to flex some cool thing that you built, but it doesn't really have any bearing on the concept of "most cases" because it's an anecdote. Queuing systems are a thing for a reason, and in most cases, using them makes more sense than writing your own.
lern_too_spel 5y ago
It probably fits within the free tier limits of a managed pubsub service.
threeseed 5y ago
But Kafka does significantly more. And if your needs are simpler like in this case then there are dozens of smaller pub/sub/queue systems that you could compare this to.
- speed_spread 5y ago
  Limit the types of server used to reduce system complexity. If you can have all your business state in the same place, ops are much easier. Kafka does more for streaming data, but doesn't do squat for relational data. You always need a database, but you sometimes can get by without a queuing system.
- moneywoes 5y ago
  Briefly what are some mandatory kafka use cases?
- akvadrako 5y ago
  I would say postgres does much more. What use case can only Kafka handle?
remram 5y ago
It's that much on a popular cloud platform, you can buy this for 3-4 times that amount and use it for years.
- discordance 5y ago
  Got a 128gb 32 core xeon workstation sitting under my desk off eBay and it was $400
  - isbvhodnvemrwvn 5y ago
    That's not exactly a setup suitable for reliable production usage though.
- tluyben2 5y ago
  Or rent it for a lot less at a traditional hosting company.
pts_ 5y ago
That's the job of a DevOps engineer not a backend developer attempted to be overworked.

macksd 5y ago

I think the point of interest was 32 cores to handle what sounds like 10 messages per second at most. That's not really a ton of throughput... It's certainly a valid point that an awful lot of uses cases don't need Twitter-scale firehoses or Google-size Hadoop clusters.

rowanG077 5y ago
It said nothing about the distribution of traffic. It might well be thousands and thousands of pub sub messages at some point of the day and 0 for others.
colinchartier 5y ago
Ah, the database does a lot more than just pub/sub - especially since the high traffic pub/sub goes through redis. I guess my point was that we never regretted setting up postgres as the "default job queue" and it never required much engineering work to maintain. For an example, it handles stripe webhooks when users change their pricing tier - if you drop that message, users would be paying for something they wouldn't receive.

rockwotj 5y ago

Fwiw I don't know the shape of the data, but I feel like you could do this with Firebase for a few bucks a month...

daenz 5y ago
you 100% could, and this thread feels like the twilight zone with how many people are advocating for using a rdbms for (what seems like) most peoples queuing needs.
- tata71 5y ago
  Does Firebase offer self-hosting these days? What do you say to those who don't want Google to know their usage info?
  - pricci 5y ago
    Checkout supabase.com. It is based on postgres.
- tomc1985 5y ago
  Dude you are seriously underestimating postgres' versatility. It does so many different things, and well!
  - daenz 5y ago
    I'm not underestimating anything. I am advocating for the right tool for the job. I have a hard time believing, despite the skewed sample size in this thread, that most people think using postgres as a message queue for most cases makes the most sense.
    - tomc1985 5y ago
      What is your idea of 'most cases'? I've personally written real-time back-of-house order-tracking with rails and postgres pubsub (no redis!), and wrote a record synchronization queuing system with a table and some clever lock semantics that has been running in production for several years now -- which marketing relies upon as it oversees 10+ figures of yearly topline revenue. Neither of those projects were FAANG scale, but they work fine for what is needed and scale relatively cleanly with postgres itself. Besides, in a lot of environments corporate will only approve the use of certain tools. And if you already have one approved that does the job, then why not?
      - daenz 5y ago
        >some clever lock semantics Most senior+ engineers that I know would hear that and recoil. Getting "clever" with concurrency handling in your home-rolled queuing system is not something that coworkers, especially more senior coworkers, will appreciate inheriting, adapting, and maintaining. Believe me. I get that you're trying to flex some cool thing that you built, but it doesn't really have any bearing on the concept of "most cases" because it's an anecdote. Queuing systems are a thing for a reason, and in most cases, using them makes more sense than writing your own.

lern_too_spel 5y ago

It probably fits within the free tier limits of a managed pubsub service.

threeseed 5y ago

But Kafka does significantly more. And if your needs are simpler like in this case then there are dozens of smaller pub/sub/queue systems that you could compare this to.

speed_spread 5y ago
Limit the types of server used to reduce system complexity. If you can have all your business state in the same place, ops are much easier. Kafka does more for streaming data, but doesn't do squat for relational data. You always need a database, but you sometimes can get by without a queuing system.
moneywoes 5y ago
Briefly what are some mandatory kafka use cases?
akvadrako 5y ago
I would say postgres does much more. What use case can only Kafka handle?

remram 5y ago

It's that much on a popular cloud platform, you can buy this for 3-4 times that amount and use it for years.

discordance 5y ago
Got a 128gb 32 core xeon workstation sitting under my desk off eBay and it was $400
- isbvhodnvemrwvn 5y ago
  That's not exactly a setup suitable for reliable production usage though.
tluyben2 5y ago
Or rent it for a lot less at a traditional hosting company.

pts_ 5y ago

That's the job of a DevOps engineer not a backend developer attempted to be overworked.