4 ms·
> What was the reason to stick on 0.10.0 for so long? Aforementioned self-packaging, we were mangling the .tar.gz files into .debs, and we had to remember to u
by lizthegrey 5y ago
> What was the reason to stick on 0.10.0 for so long?
Aforementioned self-packaging, we were mangling the .tar.gz files into .debs, and we had to remember to update the debs and then push them out onto our systems, instead of just using Apt. Thus why Confluent's prebuilt distro helped a lot! But also the team was just _afraid_ of Kafka and didn't want to touch it unnecessarily.
> I see you like Cruise Control in the Confluent Platform, did you try it earlier?
We definitely should have. We tried Datadog's Kafka-kit but found adapting it to use Wavefront or Honeycomb Metrics products was more problematic than it needed to be.
> Tiered storage is slowly coming to FOSS Kafka, hopefully in 3.2.0, thanks to some very nice developers from AirBnB. Credit to the StreamNative team, that FOSS Pulsar has tiered storage built-in.
Yeah, we're glad the rest of the world gets to have it, and also glad we paid upfront for Confluent's enterprise feature version to get us out of the immediate bind we had in 2020. Those EBS/instance storage bills were adding up fast.
- EdwardDiego 5y ago> we paid upfront for Confluent's enterprise feature version to get us out of the immediate bind we had in 2020. Definitely agree it's an essential feature for large datasets - in the past I've used Kafka Connect to stream data to S3 for longer term retention, but it's something else to manage, and getting data back into a topic if needed can be a bit painful.
- lizthegrey 5y agogetting to just use the same consistent API without rewriting clients was AMAZING.
- rad_gruchalski 5y agoThis always reminds me of this thing I’ve written some time ago : https://gruchalski.com/posts/2021-04-02-kafka-infinite-retention/ https://gruchalski.com/posts/2021-04-02-kafka-infinite-reten...
- ewhauser421 5y agoThe irony of Honeycomb using an open source tool from Datadog is not lost on me =)
- lizthegrey 5y agostand on the shoulders of giants!
- ryan_lane 5y agoThis is one of the major benefits of open source. You can share engineering resources, even with your competitors, for things that aren't your bread & butter.
- lizthegrey 5y agoYup! In fact, we collaborate with Datadog, New Relic, Splunk, Lightstep, et al on OpenTelemetry (https://news.ycombinator.com/item?id=28997275 https://news.ycombinator.com/item?id=28997275 / opentelemetry.io)