7 ms·
Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with P
by uberduper 11mo ago
Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with PG are absurdly slow.
If you don't need what kafka offers, don't use it. But don't pretend you're on to something with your custom 5k msg/s PG setup.
- loire280 11mo agoIn fact, a properly-configured Kafka cluster on minimal hardware will saturate its network link before it hits CPU or disk bottlenecks.
- j45 11mo agoBut it can do so many processes a second I’ll be able to scale to the moon before I ever launch.
- theK 11mo agoIsn't that true for everything on the cloud? I thought we are long into the era where your disk comes over the network there.
- UltraSane 11mo agoA network link can be anything from 1Gbps to 800Gbps.
- altcognito 11mo agoThis doesn't even make sense. How do you know what the network links or the other bottlenecks are like? There are a grandiose number of assumptions being made here.
- loire280 11mo agoThere is a finite and relatively narrow range of ratios of CPU, memory, and network throughput in both modern cloud offerings and bare hardware configurations. Obviously it's possible to build, for example, a machine with 2 cores, a 10Gbps network link, and a single HDD that would falsify my statement.
- altcognito 11mo agoBut the workload matters. Even the comment in the article doesn't completely make sense for me in that way -- if your workload is 50 operations per byte transferred versus 5000 operations per byte transferred, there is a considerable difference in hardware requirements.
- enether 11mo agoExactly. "a properly-configured Kafka cluster" implies you have very properly configured your clients too, which is almost never the case because it's practically very hard to do in the messy reality of a large-scale organization. Even if you somehow get everyone to follow best-practices, you most likely still won't get to saturate the network on "minimal hardware". The number of client connections and requests per second will likely saturate your "minimal CPU". It's true that minimal hardware on Kafka can saturate the network, but this mostly happens in low-digit client scenarios. In practice, orgs pushing serious data have serious client counts.
- EdwardDiego 11mo agoDepends on how you configure the clients, ask me how I know that using a K8s pod id in a consumer group id is a really bad idea - or how setting batch size to 1 and linger to 0 is a really bad idea - the former blows up disk (all those unique consumer groups cause the backing topic to consume a lot of space, as the topic is by default only compacted) and the latter thrashes request handler CPU time.
- 010101010101 11mo ago> If you don't need what kafka offers, don't use it. This is literally the point the author is making.
- uberduper 11mo agoIt seems like their point was to criticize people for using new tech instead of hacking together unscalable solutions with their preferred database.
- blenderob 11mo agoThat wasn't their point. Instead of posting snarky comments, please review the site guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize."
- lenkite 11mo agoBut honestly, isn't that the strongest plausible interpretation according to the "site guidelines" ? When one explicitly says that the one camp chases "buzzwords" and the other chases "common sense", how else are you supposed to interpret it ?
- blenderob 11mo ago> how else are you supposed to interpret it? It's not so hard. You interpret it how it is written. Yes, they say one camp chases buzzwords and another chases common sense. Critique that if you want to. That's fine. But what's not written in the OP is some sort of claim that Postgres performs better than Kafka. The opposite is written. The OP acknowledges that Kafka is fast. Right there in the title! What's written is OP's experiments and data that shows Postgres is slow but can be practical for people who don't need Kafka. Honestly I don't see anything bewildering about it. But if you think they're wrong about Postgres being slow but practical that's something nice to talk about. What's not nice is to post snarky comments insinuating that the OP is asking you to design unscalable solutions.
- PeterCorless 11mo agoExactly. Just yesterday someone posted how they can do 250k messages/second with Redpanda (Kafka-compatible implementation) on their laptop. https://www.youtube.com/watch?v=7CdM1WcuoLc https://www.youtube.com/watch?v=7CdM1WcuoLc Getting even less than that throughput on 3x c7i.24xlarge — a total of 288 vCPUs – is bafflingly wasteful. Just because you can do something with Postgres doesn't mean you should. > 1. One camp chases buzzwords. > 2. The other camp chases common sense In this case, is "Postgres" just being used as a buzzword? [Disclosure: I work for Redpanda; we provide a Kafka-compatible service.]
- mxey 11mo agoDoesn’t Kafka/Redpanda have to fsync for every message?
- kragen 11mo agoDefinitely not in the case of Kafka. Even with SSD that would limit it to around 100kHz. Batch commit allows Kafka (and Postgres) to amortize fsync overhead over many messages.
- PeterCorless 11mo agoYes, for Redpanda. There's a blog about that: "The use of fsync is essential for ensuring data consistency and durability in a replicated system. The post highlights the common misconception that replication alone can eliminate the need for fsync and demonstrates that the loss of unsynchronized data on a single node still can cause global data loss in a replicated non-Byzantine system." However, for all that said, Redpanda is still blazingly fast. https://www.redpanda.com/blog/why-fsync-is-needed-for-data-safety-in-kafka-or-non-byzantine-protocols https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s...
- uberduper 11mo agoI'm highly skeptical of the method employed to simulate unsync'd writes in that example. Using a non-clustered zookeeper and then just shutting it down, breaking the kafka controller and preventing any kafka cluster state management (not just preventing partition leader election) while manually corrupting the log file. Oof. Is it really _that_ hard to lose ack'd data from a kafka cluster that you had to go to such contrived and dubious lengths?
- joaohaas 11mo agoHad the same thoughts, weird it didn't include Kafka numbers. Never used Kafka myself, but we extensively use Redis queues with some scripts to ensure persistency, and we hit throughputs much higher than those in equivalent prod machines. Same for Redis pubsubs, but those are just standard non-persistent pubsubs, so maybe that gives it an upper edge.
- roozbeh18 11mo agoJust checked my single node Kafka setup which currently handles 695.27k e/s (average daily) into elasticsearch without breaking a sweat. kafka has been the only stable thing in this whole setup. zeek -> kafka -> logstash -> elastic
- ozgrakkurt 11mo agoThis is why benchmarks should be hardware limit based IMO. Like I am maxing IOPS/throughput of this ssd or maxing out the network card etc. CPU is more tricky but I’m sure it can be shown somehow
- darth_avocado 11mo agoThe 96 vcpu setup with 24xlarge instance costs about $20k/month on AWS before discounts. And one thing you don’t want in a pub sub system is a single instance taking all the read/writes. You can run a sizeable Kafka cluster for that kind of money in AWS.
- jaimebuelta 11mo agoI may be reading a bit extra, but my main take on this is: "in your app, you probably already have PostgreSQL. You don't need to set up an extra piece of infrastructure to cover your extra use case, just reuse the tool you already have" It's very common to start adding more and more infra for use cases that, while technically can be better cover with new stuff, it can be served by already existing infrastructure, at least until you have proof that you need to grow it.
- blenderob 11mo ago> Has this person actually benchmarked kafka? Is anyone actually reading the full article, or just reacting to the first unimpressive numbers you can find and then jumping on the first dismissive comment you can find here? Benchmarking Kafka isn't the point here. The author isn't claiming that Postgres outperforms Kafka. The argument is that Postgres can handle modest messaging workloads well enough for teams that don't want the operational complexity of running Kafka. Yes, the throughput is astoundingly low for such a powerful CPU but that's precisely the point. Now you know how well or how bad Postgres performs on a beefy machine. You don't always need Kafka-level scale. The takeaway is that Postgres can be a practical choice if you already have it in place. So rather than dismissing it over the first unimpressive number you find, maybe respond to that actual matter of TFA. Where's the line where Postgres stops being "good enough"? That'll be something nice to talk about.
- adamtulinius 11mo agoThe problem is benchmarking on the 96 vcpu server, because at that point the author seems to miss the point of Kafka. That's just a waste of money for that performance.
- blenderob 11mo agoAnd if the OP hadn't done that, someone here would complain, why couldn't the OP use a larger CPU and test if Postgres performs better? Really, there is no way the OP can win here, can they? I'm glad the OP benchmarked on the 96 vCPU server. So now I know how well Postgres performs on a large CPU. Not very well. But if the OP had done their benchmark on a low CPU, I wouldn't have learned this.
- cheikhcheikh 11mo agoyou're missing the point. Postgres performs well on large CPU. Postgres as-used by OP does not and is a waste of money. It's great that he benchmarked for a larger CPU, that's not what people are disputing, they are disputing the ridiculous conclusion.
- 11mo ago
- adamtulinius 11mo agoI remember doing 900k writes/s (non-replicated) already back on kafka 0.8 with a random physical server with an old fusionio drive (says something about how long ago this was :D). It's a fair point that if you already have a pgsql setup, and only need a few messages here and there, then pg is fine. But yeah, the 96 vcpu setup is absurd.
- ljm 11mo agoI wonder if OP could have got different results if they implemented a different schema as opposed to mimicking Kafka's setup with the partitions, consumer offsets, etc. I might well be talking out of my arse but if you're going to implement pub/sub in Postgres, it'd be worth designing around its strengths and going back to basics on event sourcing.