8 ms·
> doing hundreds of thousands of messages per day > The postgres instance now runs on 32 cores and 128gb of memory and has scaled well. Am I the only one?
by boomskats 5y ago
> doing hundreds of thousands of messages per day
> The postgres instance now runs on 32 cores and 128gb of memory and has scaled well.
Am I the only one?
- colinchartier 5y agoSuch a server is 400$/mo, a backend developer that can confidently maintain kafka in production is significantly more expensive!
- macksd 5y agoI think the point of interest was 32 cores to handle what sounds like 10 messages per second at most. That's not really a ton of throughput... It's certainly a valid point that an awful lot of uses cases don't need Twitter-scale firehoses or Google-size Hadoop clusters.
- rowanG077 5y agoIt said nothing about the distribution of traffic. It might well be thousands and thousands of pub sub messages at some point of the day and 0 for others.
- colinchartier 5y agoAh, the database does a lot more than just pub/sub - especially since the high traffic pub/sub goes through redis. I guess my point was that we never regretted setting up postgres as the "default job queue" and it never required much engineering work to maintain. For an example, it handles stripe webhooks when users change their pricing tier - if you drop that message, users would be paying for something they wouldn't receive.
- rockwotj 5y agoFwiw I don't know the shape of the data, but I feel like you could do this with Firebase for a few bucks a month...
- daenz 5y agoyou 100% could, and this thread feels like the twilight zone with how many people are advocating for using a rdbms for (what seems like) most peoples queuing needs.
- tata71 5y agoDoes Firebase offer self-hosting these days? What do you say to those who don't want Google to know their usage info?
- pricci 5y agoCheckout supabase.com. It is based on postgres.
- tomc1985 5y agoDude you are seriously underestimating postgres' versatility. It does so many different things, and well!
- daenz 5y agoI'm not underestimating anything. I am advocating for the right tool for the job. I have a hard time believing, despite the skewed sample size in this thread, that most people think using postgres as a message queue for most cases makes the most sense.
- tomc1985 5y agoWhat is your idea of 'most cases'? I've personally written real-time back-of-house order-tracking with rails and postgres pubsub (no redis!), and wrote a record synchronization queuing system with a table and some clever lock semantics that has been running in production for several years now -- which marketing relies upon as it oversees 10+ figures of yearly topline revenue. Neither of those projects were FAANG scale, but they work fine for what is needed and scale relatively cleanly with postgres itself. Besides, in a lot of environments corporate will only approve the use of certain tools. And if you already have one approved that does the job, then why not?
- lern_too_spel 5y agoIt probably fits within the free tier limits of a managed pubsub service.
- threeseed 5y agoBut Kafka does significantly more. And if your needs are simpler like in this case then there are dozens of smaller pub/sub/queue systems that you could compare this to.
- speed_spread 5y agoLimit the types of server used to reduce system complexity. If you can have all your business state in the same place, ops are much easier. Kafka does more for streaming data, but doesn't do squat for relational data. You always need a database, but you sometimes can get by without a queuing system.
- moneywoes 5y agoBriefly what are some mandatory kafka use cases?
- akvadrako 5y agoI would say postgres does much more. What use case can only Kafka handle?
- remram 5y agoIt's that much on a popular cloud platform, you can buy this for 3-4 times that amount and use it for years.
- discordance 5y agoGot a 128gb 32 core xeon workstation sitting under my desk off eBay and it was $400
- isbvhodnvemrwvn 5y agoThat's not exactly a setup suitable for reliable production usage though.
- tluyben2 5y agoOr rent it for a lot less at a traditional hosting company.
- pts_ 5y agoThat's the job of a DevOps engineer not a backend developer attempted to be overworked.
- johnisgood 5y agoAre you implying that given the specs, hundreds of thousands of messages per day is not good enough? I think you are, or at least that is what I was thinking myself.
- marcosdumay 5y agoOnly for hundreds of thousands of messages per day, that's way too big of a server. But if you look on the rest of the thread, it doesn't do only that. Anyway, for a server that only does pub/sub with ACID guarantees, those specs are so large that there is certainly a bottleneck before they matter. So it wouldn't be strange if somebody gets one that can't even handle that, it just would mean that there is some issue somewhere we don't see.
- edmundsauto 5y agoIs your point that the server has room to grow? Or that you just “ain’t impressed by that”?
- Lhiw 5y agoThe point is that the update doesn't really tell us much about how well the approach scales despite aiming to do so.
- marcosdumay 5y agoI guess the point is that the scale is actually not that large, but that's perfectly ok because most problems will never need that large scale either. In fact, the article makes a very good point how just doing it in postgres is great, it doesn't really scale (because of ACID), and adapting it for scale after you need it will lead to a better design than what you would do if you started optimizing without any information.
- SahAssar 5y agoI assume that is their main database for everything, not just for pub/sub. One of the big benefits of doing it that way is that you have proper transaction handling across jobs and their related data.
- jbverschoor 5y agoCome on man… you can run the whole thing off if a few Gb instance. Such a huge instance should be able to do about 100k a second!
- aniforprez 5y agoPotential and actual usage aren't related. They might be having a lot of records and read/writes but maybe the actual pub/sub isn't that intensive. They seem to be using the same DB for everything
- stillicidious 5y agoBe careful not to confuse average load with peak instantaneous load. Bursty workloads are the bane of capacity planners everywhere.
- akvadrako 5y agoDoes postgres scale that well? I would be interested in case studies as I've not seen much achieving beyond 10k records per second.
- michaelcampbell 5y agoWhose to say it can't?
- dreyfan 5y agoIt’s such a low throughput requirement I think even bitcoin could support it.
- ivalm 5y agoNo, this is prob still spiky enough to have more than 7 transactions per second, that’s too much for Bitcoin.
- michelpp 5y agoTo cherry pick two details of the post and insinuate something about it? No.
- ClumsyPilot 5y agoYeah, every home IoT hub processes more messages than that with less thsn raspberri pi worth of compute
- earleybird 5y agoI certainly appreciate the sentiment though I'm pretty sure I don't have the same reliability and uptime guarantees on my little Rpi3/MQTT/NodeRed/SQLite/ESP8266 home system :-) That said, it's been running for upwards of 4 years and accumulated an insane number of temperature readings inside and above heating vents (heat source is heat pump) SELECT count() as count FROM temperatures : msg : Object { _msgid: "421b3777.908118", topic: "SELECT count() as count FROM …", payload: 23278637 } Ok, I need therapy for my data hoarding - 23 million temp samples is not a good sign :-)
- brodouevencode 5y agoCurious as to why you aren’t tracking that with a time series database?
- earleybird 5y agoIt's hobby diversion so minimal effort is a factor. That and that SQL query comes back in seconds. The initial experimentation was with ESP8266's and the MQTT/NodeRed/SQLite played a supporting roll. My experience with SQLite is that it can take you a long ways before needing to look elsewhere.
- ClumsyPilot 5y agoThe IoT hubs are an embedded system, built with a minimal memory footprint and overhead, 512 mb of ram is typical, sometimes less. Here is an example: https://www.gl-inet.com/products/gl-s1300/ https://www.gl-inet.com/products/gl-s1300/ That means you can't have docker and different versions of Java, node and .Net all running in parallel. You run a single process and Sqlite is a library that allows SQL operations and database to be inbuilt. You 'budget' is like 100 mb of Ram, becauae other stuff has to run too. All the time-series databases I know are a large, memory hungry hippo, built for distributed compute/kubernetes. Just very different usecase. If one was built with minimalism in mind, then it could be used.
- tyingq 5y agoI imagine their scaling problem isn't messages/day, it's probably lots of concurrent, persistent connections. And I don't think a connection pooler would work with this job queue setup.
- tmountain 5y agoPeople are jumping on this. Question is—do the resource requirements outlined align with usage you described, or is that combined workload? By combined workload, I mean working set plus messaging. It’s not a useful exercise to criticize a service that’s multifaceted based on a single use case. Full disclosure—-not a Postgres user, nor am I invested in the tech.
- colinchartier 5y agoI should've clarified, the database handles more than just the "regular" pub/sub, some of the tables have over a billion rows.
- boomskats 5y agoThank you. That makes a lot more sense and explains the value of the approach much better.