7 ms·
Finding Kafka’s throughput limit in Dropbox infrastructure
- deanCommie 8y ago"At Dropbox, Kafka clusters are managed by the Jetstream team, whose primary responsibility is to provide high quality Kafka services. " And this is why AWS's "Managed Kafka" service has a place. I get why Dropbox moved off S3 - storage is their core competency and they thought they could/should do better/cheaper. But I'm surprised they would be wasting valuable dev time with this kind of overhead.
- Zecar 8y ago> storage is their core competency Not so sure about that. I post in every one of these threads, but they deleted our honeymoon photos on my wife's account and then deleted my account several months later without notice. I wouldn't trust them with a swap file. Move fast and break things is a terrible mantra for a file storage company.
- pwaivers 8y agoTheir core product is still storage, even if you had a bad experience. You may think Dropbox is a bad product, but OP's point still stands.
- gitgud 8y agoCan't say that I've ever had an issue with the reliability or service at Dropbox. Not sure they're known for moving fast either, as the product has changed very little in 10 years.
- toomuchtodo 8y ago> But I'm surprised they would be wasting valuable dev time with this kind of overhead. I'm not surprised at all. They have reached a scale where managed services are looked at with a critical eye, and are no longer the default.
- foobiekr 8y agoThis. AWS is most valuable when the management overhead is a double digit percentage of your revenue. At some point the AWS margin itself is genuinely larger than DIY. That should be obvious as it has to be so unless AWS’s own costs are dramatically lower which is only possible if their value is smoke-and-mirrors.
- halbritt 8y agoI don't think it's that hard to reach a scale where DIY makes sense. As was stated up above, the AWS kafka offering isn't complete. One could do something similar with a cloud formation template. The ElasticSearch offering on the other hand is fairly complete. At my scale, it would cost me ~$15-20k/mo for ES in AWS. However, I DIY my own ES cluster (in k8s) which costs me something like $5k/mo. The delta is roughly $120-180k/yr. An engineer costs me roughly ~$200k year with salary and benefits, so let's say the delta is .75 of an engineer. There are also risks to DIY that are hard to put an number on. ES is definitely a PITA to run, but it's core enough to our infrastructure offering that dedicating .25-.5 of an engineer to it seems like a worthy trade-off to me. It is also a unique enough type of workload that it yields an interesting learning opportunity for my team.
- toomuchtodo 8y agoI don’t disagree at all. Far too often, people wait too long to optimize away from a vendor’s offering. Their margin is your opex. Put that money back in your pocket.
- technion 8y agoIt also means you're in a position to pick up your stack and deploy it elsewhere if such a thing makes business sense in future. A lot of organisations are putting themselves in a position where that's not an option.
- halbritt 8y agoThe AWS kafka product is a non-product. One would have to reproduce the same effort there as anywhere else. 60MB/s per broker Took me about two hours worth of testing to reach that same conclusion. I like the Dropbox approach though, it is very thorough. One presumes that kafka is going to become a fundamental part of their infrastructure, in which case, this effort is a prudent investment.
- vaishaksuresh 8y agoAs it stands today, AWS' managed kafka is very limited. Single region and no TLS is a non-starter for a lot of places.
- deleted 8y ago[deleted]
- cbsmith 8y agoI'd use the Confluent managed Kafka service before I'd touch the AWS one.
- pwaivers 8y agoThe overload indicators are strange to me. For example, one indicator is "IO thread idle below 20%: this means the pool of worker threads used by kafka for handling client requests are too busy to handle any more workload". How did they determine this? Is it true? I think a much better way would be to monitor the JMX reporter for dropped messages or high latency: https://docs.confluent.io/current/kafka/monitoring.html https://docs.confluent.io/current/kafka/monitoring.html.
- pram 8y agoThere are JMX beans for the socket server and request handler idle. If they even dip below 80% you’re probably looking at poorly configured producers timing out.
- pandemic_region 8y agoLine is doing 150 billion msgs/day on their Kafka infra, about 3million per second. https://www.slideshare.net/linecorp/building-a-companywide-data-pipeline-on-apache-kafka-engineering-for-150-billion-messages-per-day https://www.slideshare.net/linecorp/building-a-companywide-d...
- tudelo 8y agoThat seems... like a lot. Thanks for the link.
- buro9 8y agoAt Cloudflare a year ago we said we were doing 6m per second ingest rate on a cluster of 106 brokers with x3 replication factor, 106 partitions: https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p... The more interesting blog post IMHO is this one on Kafka compression https://blog.cloudflare.com/squeezing-the-firehose/ https://blog.cloudflare.com/squeezing-the-firehose/
- Rebelgecko 8y agoThe compression article was really useful for me when I was trying to squeeze a bit more performance out of our brokers at work. Any plans to revisit the article now that Kafka has zstd support?
- nosequel 8y agoThey talk about zstd in the post and they were the ones who added support to Sarama.
- halbritt 8y agoI'm using zstd with my kafka implementation and it works great. v2.1 is required. We're using librdkafka, which works, but I believe we had to compile it in to get that and idempotent producer.
- anotherhue 8y ago50MB/s is tiny, you can push 500 on commodity disks. https://www.slideshare.net/ConfluentInc/kafka-on-zfs-better-living-through-filesystems?ref=https://www.confluent.io/kafka-summit-sf18/kafka-on-zfs https://www.slideshare.net/ConfluentInc/kafka-on-zfs-better-... Full talk: https://www.confluent.io/kafka-summit-sf18/kafka-on-zfs https://www.confluent.io/kafka-summit-sf18/kafka-on-zfs (shameless plug)
- halbritt 8y agoWith TLS?
- pmalynin 8y agoAfter the asymmetric phase, TLS is pretty cheap on modern hardware. About 1500mb/s per core (at least with AES-NI). This is why all modern Macs do transparent encryption even if you didn’t explicitly turn on FDE.
- halbritt 8y agoSure, sure, but did you get that kind of throughput with kafka on java? Most folks (Confluent, for example) are still on java 8 which has some very severe limitations with regards to throughput.
- pmalynin 8y agoI'm not exactly sure what that has to do with the throughput of TLS, especially considering that it's 99% likely that its implemented in a native library (openssl, etc).
- halbritt 8y agoThe problem is that it's not implemented in the native library. That's why the throughput is so abysmal.
- Rafuino 8y ago60 MB/s per broker at what latency though? Does latency even factor into the performance consideration?
- halbritt 8y agoFor some use cases, it absolutely does. I can't remember the exact use case, but a friend of mine is adopting kafka with an end-to-end latency goal of 100ms. Absolutely requires SSDs and for the producers and consumers to be colocated, but is otherwise not that difficult to achieve.
- Zecar 8y agoDropbox deleted my honeymoon photos
- shanemhansen 8y agoI regularly do 200MB/s per node with NSQ (I know, I know, apples to oranges but people should be aware there are choices other than apples depending on your use case).
- hsong 8y agoIt would be great to see analysis on where the bottleneck is. Is it disk, network, compression, etc.