39 ms·

Kafka is dead, long live Kafka

392 points by richieartoul 3y ago

poorman 3y ago
Oh man, I can't wait until these people get their first S3 bill for the number or GET requests.
- ryanworl 3y ago
  [WarpStream co-founder here] My co-founder and I worked at Datadog for over 3 years where we built Husky, an event storage and query system built directly on top of S3 as well. We know what we're doing here, I promise ;) https://www.datadoghq.com/blog/engineering/introducing-husky/ https://www.datadoghq.com/blog/engineering/introducing-husky...
  - poorman 3y ago
    What happens if I pull the plug on a server that hasn't flushed to S3 in the configured time? Do I loose that event?
    - ryanworl 3y ago
      With an appropriately configured client (i.e. one that retries and waits for requests to be acknowledged), another Agent would receive the retry and the event would be written to the topic at that time.
  - madmax108 3y ago
    "My co-founder and I worked at Datadog" If there's one company that's known for blowing up budgets as you scale, it's Datadog. Hoping WarpStream doesn't follow that playbook. Best wishes!
- richieartoul 3y ago
  (WarpStream co-founder) Check the end of our blog post, we brought receipts :)
88913527 3y ago
It's refreshing to see cloud computing costs matter. Hopefully it'll cut down of needless cloud complexity and the proliferation of indirection in software abstractions of the last decade.
- richieartoul 3y ago
  (WarpStream co-founder) Cheers! We really want to drive the incremental cost (both in terms of $ and management overhead) per GiB down as low as possible so people can start using these systems for more use-cases.
deleted 3y ago
[deleted]
liminal 3y ago
Can this be installed locally and run on Minio?
- richieartoul 3y ago
  Will make this possible soon, oversight on our part that the agent doesn't expose a way to let you do that right now
yingjunwu 3y ago
I am the founder of RisingWave (http://risingwave.com/ http://risingwave.com/), an open-source SQL streaming database. I am happy to see the launch of Warpstream! I just reviewed the project and here's my personal opinion: * Apache Kafka is undoubtedly the leading product in the streaming platform space. It offers a simple yet effective API that has become the golden standard. All streaming/messaging vendors need to adhere to Kafka protocol. * The original Kafka only used local storage to store data, which can be extremely expensive if the data volume is large. That's why many people are advocating for the development of Kafka Tiered Storage (KIP-405: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...). To my best knowledge, there are at least five vendors selling Kafka or Kafka-compatible products with tiered storage support: -- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/ https://www.confluent.io/10x-apache-kafka/; -- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafka https://github.com/Aiven-Open/tiered-storage-for-apache-kafk...); -- Redpanda Data, which cuts your TCO by 6X (https://redpanda.com/platform-tco https://redpanda.com/platform-tco); -- DataStax, which commercializes Apache Pulsar (https://pulsar.apache.org/ https://pulsar.apache.org/); -- StreamNative, which commercializes Apache Pulsar (https://pulsar.apache.org/ https://pulsar.apache.org/). * WarpStream claims to be "built directly on top of S3," which I believe is a very aggressive approach that has the potential to drastically reduce costs, even compared to tiered storage. The potential tradeoff is system performance, especially in terms of latency. As new technology, WarpStream brings novelty, and definitely it also needs to convince users that the service is robust and reliable. * BYOC (Bring Your Own Cloud) is becoming the default option. Most of the vendors listed above offer BYOC, where data is stored in customers' cloud accounts, addressing concerns about data privacy and security. I believe WarpStream is new technology to this market, and and would encourage the team to publish some detailed numbers to confirm its performance and efficiency!
- mdaniel 3y ago
  If you're still within the edit window for your comment, the ");" in them are attached to the URLs, making them problematic to click ed -- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/ https://www.confluent.io/10x-apache-kafka/ -- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafka https://github.com/Aiven-Open/tiered-storage-for-apache-kafk... -- Redpanda Data, which cuts your TCO by 6X https://redpanda.com/platform-tco https://redpanda.com/platform-tco -- DataStax, which commercializes Apache Pulsar https://pulsar.apache.org/ https://pulsar.apache.org/ -- StreamNative, which commercializes Apache Pulsar https://pulsar.apache.org/ https://pulsar.apache.org/
hintymad 3y ago
So the idea is that S3 offers practically unlimited throughput, so we can certainly use S3 for a scalable queue? I wish the open-source community could have something competing with S3, but then S3 has made it really hard for companies to have a convincing justification to build their own alternative[1]. [1] There are a few alternatives, such as Apache Ozone, Ceph, and Good'ol HDFS, but none seems have an accelerating momentum for S3's workload.
Exuma 3y ago
Care to summarize the 'contact us' pricing?
- ryanworl 3y ago
  We're aiming for per-GB usage-based pricing that is significantly cheaper than the alternatives, but the BYOC model combined with our extremely efficient cloud control plane gives us a lot of flexibility here. That's why we can offer a free tier at all. We're mostly just not sure yet, so your input would be appreciated.
pfannkuchen 3y ago
I feel like people misuse the X is dead, long live X construction, but maybe I’m missing something. I’ve always taken it to be a reference to the stability and near-programmatic nature of British royal succession. So while “the king is dead” in some places historically would be a disaster with conflict surely following, in Britain it automatically and instantly becomes “long live the king” for the next king and there is no panic.
EdwardDiego 3y ago
Dear richieartoul, your blog post is a tad oversalted. Kafka doesn't inherently require dedicated teams of experts and millions of dollars until you're running very large clusters. But fully agree that 3-AZ stretch clusters suck money through those inter-AZ xfer fees. Which is how AWS sells MSK, the inter-AZ xfer is "free"! That is, already priced in...
jjwiseman 3y ago
This comment from 4 months ago helped me understand when you might want to use kafka: https://news.ycombinator.com/item?id=35160555 https://news.ycombinator.com/item?id=35160555 (The comment is by rad_gruchalski, who I see is also active in this thread. Thanks, rad_gruchalski!)
siliconc0w 3y ago
It's a neat idea for a SaaS to just host the control-plane/metadata and allow customers to write to their own data store via S3. I can't think of too many other systems work that way - it certainly is operationally simpler but I wonder if it also gets you some compliance wins. I wonder if we'll eventually standardize on a cloud-agnostic S3-like protocol but for locking and fast transactional writes so we can just have companies building pure software again but using standardized Cloud instead of POSIX APIs.
edude03 3y ago
Okay, I hate to be that guy but I'm given two options for pricing > Free but with only 24 Hours of Retention or > Contact us. Pricing is hard for sure, but if I'm being honest neither of these options make me want to try the product.
- richieartoul 3y ago
  (WarpStream co-founder) I know that is super frustrating, I’m sorry. We’re just really early and unsure how to begin even discussing public pricing. We are 100% committed to that free tier. We designed our entire architecture around making that free tier so cost effective that we could offer it for free. We want to get to a point where we can have transparent pricing on our website, it will just take us some time to figure that out after speaking with initial customers. We would love to hear your feedback though. You won’t get handed off to a sales guy or anything, it’ll just be a direct conversation with me (CEO) and my cofounder Ryan (CTO)
topaz0 3y ago
Kafka's been dead almost 100 years.
backendanon 3y ago
I didn't check the site. I have nightmares debugging kafka from years ago and it's only huger now. No thank you, give me Apache MQ or RabbitMQ, even IBM's MQ and it's all good and don't try to tell me I need a streaming thingamajig.
rupurt 3y ago
This is super interesting @richieartoul. I specced out something similar myself and was going to implement it in Zig https://github.com/fremantle-industries/transit https://github.com/fremantle-industries/transit. FWIW I came to a similar conclusion that a lot of the power in Kafka comes from the API and that eventually much of the complexity of managing the cluster will eventually be abstracted away with multiple implementations. I also felt that if I could implement Kafka persistent over the S3 keyspace then I could start with persistence direct to S3 like you've done with warpstream and then layer on a faster hot disk and in memory tiering mechanism to eventually lower end to end latencies. I love where you're going with this so hit me up on twitter if you ever want to chat more in-depth https://twitter.com/rupurt https://twitter.com/rupurt.
- richieartoul 3y ago
  DMd you on twitter
d_t_w 3y ago
Hi @richieartoul, does warpstream support the standard Kafka Admin API? We build an admin console / dev tooling for Kafka (https://kpow.io https://kpow.io) that supports Kafka 1.0+ including Redpanda due to their fairly strict adherence to those API. Warpstream seems like a cool idea, I'd like to see what happens if we plug Kpow on top of it, if that's possible.
- richieartoul 3y ago
  Our kafka protocol support is still incomplete. We’ve documented our progress here: https://docs.warpstream.com/warpstream/reference/kafka-protocol-and-feature-support https://docs.warpstream.com/warpstream/reference/kafka-proto... That said we’d be stoked to get this working as an additional tool for people. Do you want to shoot me an email or join our slack so we can discuss further? I can probably prioritize whatever protocol features were missing to get it working. founders@warpstreamlabs.com
kabochya 3y ago
We've built something probably very similar to this product at my previous job. We had double digit TB daily ML traffic that didn't require realtime latency, so we moved those all onto S3 and also saw approx ~90% cost savings. This was built on top of JVM and still used a 6 broker Kafka cluster to keep metadata (vs probably 300 when it was originally all on Kafka). Kafka's compute and storage model doesn't scale too well for the extreme use cases that can tolerate latency, and the Apache Pulsar model sort of worked better (though at that time Pulsar wasn't too stable for us to use in prod). One of the keys to cost efficiency for us was that the size of the data was large enough that we didn't need to wait too long before hitting an economic file size to upload. Trying to imagine how a pipeline with less than 10 MB/s would work with this efficiently.
- richieartoul 3y ago
  (WarpStream co-founder) Yeah we’ve run into a number of people who’ve rolled their own solution in this space. The “push pointers to S3 through traditional Kafka” approach is a very practical one. Was this memq at Pinterest, or something else?
  - kabochya 3y ago
    Yeah it is, sort of glad that it's actually known. I've since left the ingestion side of data infra so not very familiar of the landscape after one year.
steve76 3y ago
[dead]
Spiwux 3y ago
> how many partitions do I need? Unclear, but better get it right because you can never change it! That is.. simply wrong. You can change the number of partitions. Plus I really don't get the "you need an entire team of engineers to operate Kafka" claim you keep repeating. That is simply not true, speaking from experience. It is certainly expensive to run, but doesn't require a lot of engineering hours in our team.
FridgeSeal 3y ago
> requires teams to operate it I mean, managed Kafka has been a thing for a long while now. I’ve used it plenty and haven’t had any issues…
jooz 3y ago
The criticism I have through kafka is the "at-least-once" semantics. How do you manage exactly-once semantics? If kafka performance is based on reading small batches of 50 messages, in case of crash of the consumer, some of them will be processed twice. Depending on your business logic this may be ok, or may be create a new problem that must be solved farther in the process by adding an external data store.
- kilotaras 3y ago
  You can't have exactly-once in distributed system. Crashes happen, network or power goes down and your ack get's lost[0]. The best one can do is accept that and make your processing idempotent. [0] The most interesting case I've witnessed was power going down because a hot air balloon crashed into power line.
benbuick 3y ago
Hey Richie, As you seem to support the Kafka + Kinesis(?) API: We have developed Kadeck (kadeck.com), a control plane and collaboration layer for Kafka & Kinesis teams. I have looked at the documentation and reckon we should be supporting a lot of Warpstream already. Is there any quick way to test this out? Would love to share this with our users! Whether dead or alive, there's no denying that Kafka, or data streaming in general, is one of the most exciting fields today. Happy to welcome a new member!
- richieartoul 3y ago
  (WarpStream Cofounder) https://docs.WarpStream.com https://docs.WarpStream.com is the best document we have right now. This sounds interesting though, can you jump in our slack or shoot me an email at founders@warpstreamlabs.com ? Happy to support you any way we can!
substation13 3y ago
Is there any source available for this?
bobby_the_whale 3y ago
[dead]
throw4847285 3y ago
I was hoping this would be about Franz.
aftermodern 3y ago
The object storage backend is interesting and I’m wondering if Kafka will have a native option for that, eventually. Confluent hasn’t outright advertised it, but one of their engineers mentioned in a q&a that their Cloud offering is using object storage for their own backend now.
- relay23 3y ago
  They always have.. you get it but you don't know it.
dtjohnnymonkey 3y ago
I've seen both your FoundationDB talks, curious if you are using FoundationDB under the hood or how much of the metadata store is homegrown? Also, how did you implement the parsing of the Kafka protocol? From scratch or pieced together with open source Go bits? Having a nice programmable API for building Kafka-compatible servers could be a huge boon to the Kafka community (wink wink).
michaelcampbell 3y ago
> WarpStream is 5-10x cheaper than Kafka <sigh> I'm guessing they mean 1/10 to 1/5 the cost? "1x cheaper" would be free.
ryanworl 3y ago
I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges. If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.
- kbumsik 3y ago
  Does it support S3-compatible services, notably Cloudflare R2? I heard that there might be special handling for each S3-compatible providers, due the slightly different API behavior and different consistency models, and etc. If it support Cloudflare R2 then it would be great for multi-cloud too.
- geekone 3y ago
  What was used to generate the diagrams in the post?
- dottedmag 3y ago
  Do you have a reference documentation for S3 data layout?
  - richieartoul 3y ago
    (WarpStream founder here) Not currently. One of the things we're looking to do next is make it so any topic can be "automatically" turned into a standard format in S3, something like Parquet/Iceberg/Deltalake so its easier to consume for application that don't particularly care about the Kafka protocol.
    - EamonnMR 3y ago
      That would be a killer feature. Once we switched to MSK back at a previous gig, our biggest care and feeding task was adding new event types to our kafka-to-redshift ETL thing. Well, that and dealing with scaling our http wrapper...
cduzz 3y ago
Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clusters, etc. 3. What's done to preserve order-of-operation within a kafka partition?
- cbsmith 3y ago
  The way Kafka works, it naturally buffers & coalesces messages even before they get to the brokers, so yes, of course the messages are being coalesced. There is no problem with "just" running a Kafka cluster in each AZ and only replicating data between AZs until it's time to pull it all together. It's just that when presented with a distributed system and AZs, engineers (and in fairness the business requirements) are more than likely to go with a multi-AZ solution. Same goes for regions. So the vast majority of Kafka clusters are multi-AZ but probably shouldn't be, and Kafka gets the bill for that, even though it shouldn't. The Kafka protocol doesn't really preserve order-of-operation within a Kafka partition. It preserves the order of operations within a producer-partition pair (and even then, only if you configure it a certain way). The standard implementation does this by preserving the order-of-broker-receipt-of-messages from producers, but from an external system's vantage point, it really only means that (if configured the right way) messages with any given key, from any given producer, will be preserved in the order they are received.
  - cduzz 3y ago
    The architect mentioned a 100ms (or 10 times per second) buffer / flush rate; presumably there's some windowing settings so I can flush at 10,000 records or after 1/10th of a second, and can chose 10 records or 1/100th of a second if I don't mind the bill. Kafka's nice because there are a lot of knobs to adjust how you prioritize availability or durability or latency. Kafka's tedious because there are a lot of knobs... I was curious as to the nature of the knobs on this product. I probably should have also asked "and by S3 do you mean S3 or any S3 like object store?" probably that's answered elsewhere.
slantedview 3y ago
This article seems to be spreading some FUD, in particular this comment: > Cloud economics – by design, Kafka’s replication strategy will rack up massive inter AZ bandwidth costs. You're no more or less forced to put Kafka replicas in different AZs than you would be with an alternative.
- richieartoul 3y ago
  (WarpStream Founder) That's fair, but we think people shouldn't have to choose between multiple AZs and costs. All cloud providers recommend that critical workloads running in cloud environments should run in more than one AZ.
- crimsonpowder 3y ago
  Of course, as soon as an AZ goes down that had all of your Kafka nodes in it, you get eviscerated on HN for running such a setup.
  - EdwardDiego 3y ago
    3 AZ Kafka is the safest, but I've run several 2.5 clusters just fine. (cluster stretched across 2 AZs with a tiebreaker ZK in a 3rd AZ to maintain quorum in the event that you lose an AZ)
rcarmo 3y ago
Well, one thing's for sure. Running Kafka on discrete VMs on a cloud provider "by the book" is ludicrously expensive. I remember having a very simple discussion with quite a few customers about both Kafka and Hadoop that boiled down to this: Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? (in this case it was Azure storage, which provides locally, redundant, AZ-redundant or globally redundant storage, most of which are available to run managed disks upon). This is why properly designed Hadoop/Kafka cloud managed services employ storage adapters to leverage the provider's baked in redundancy. And why some cloud providers have Kafka-compatible event brokers. The rest of what WarpStream does is just icing on the cake (although I'm curious as to the internals and how they avoid inter-AZ charges). (full disclosure: I work at Microsoft, but built Hadoop/Spark/Kafka clusters before joining nearly a decade ago.)
- chaotic-good 3y ago
  > Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? That's easy. EBS and similar solutions comes with the price. They're very expensive. Especially, when you need a lot of IOPs. You may be saving on cross-AZ traffic but you will pay ridiculous amount of money on storage. If you have replication you can use attached storage which is way cheaper.
  - rcarmo 3y ago
    Azure Managed Disks are inherently replicated. But there are other storage solutions (like redundant WebHDFS) you can take advantage of. At the time, the customers I was talking to wanted to have identical deployments in the cloud (to what they had on-premises, down to the number and size of disks).
- hinkley 3y ago
  In the good ol' days, we used to implement redundant systems such that there was tie breaker process that used only a fraction of the resources of the true processes. Some Raft implementations allow for nodes that are voters but cannot be quorum leaders (example: the branch office where all traffic is funneled through an asymmetric VPN tunnel should never be elected leader, but knows which candidates it can and cannot see) So the table stakes for running a cluster was not 3x as much hardware but closer to 2.2x, which is a huge deal for solutions in the small and developer sandboxes. It also matters when 3 shards don't quite cover your load but 5 is too many. Or 6 vs 7. The problem is that with geographic replication, this doesn't fix either of the problems articulated as part of the thesis of this article: 1. Cloud economics – by design, Kafka’s replication strategy will rack up massive inter AZ bandwidth costs. 2. Operational overhead – running your own Kafka cluster literally requires a dedicated team and sophisticated custom tooling. Still, we need this functionality back for cloud, particularly as the pendulum swings back to self hosting, which it always has in the past.
ant6n 3y ago
I keep seeing these HN titles about technologies named for other concepts, thinking it's talking about those other contexts, without any indication it's about those obscure technologies. Then I'm disappointed its just about some random, badly named technology. So, in honor of actual Zombie Kafka, here a short news blurb about Kafka international airport being Named most Alienating Airport: https://www.youtube.com/watch?v=gEyFH-a-XoQ https://www.youtube.com/watch?v=gEyFH-a-XoQ
- tekla 3y ago
  Seriously. I ordered Windows and all I got was a piece of cardboard with some random letters and numbers on it, and not a frame with glass.
MapleWalnut 3y ago
It'd be great if there was an rss feed for your blog
whalesalad 3y ago
Running Kafka is an exercise in sadomasochism. This on the other hand sounds pretty neat.
- 8organicbits 3y ago
  I've heard good things about Red Panda[1], but I haven't tried it. [1] https://redpanda.com/ https://redpanda.com/
- whatscooking 3y ago
  What? It’s pretty easy to set up and get going
  - whalesalad 3y ago
    The last time I was running a cluster it was 2014-2015 and it was not too bad to get up and running, but it felt so heavy handed for what it was. The zookeeper requirement was frustrating. The fact that consumers needed to maintain offset state was frustrating (although... understandable). For the right use case I suppose it is 100% worthwhile but more often than not I have seen it used in places where other tools would be more appropriate.
    - lmm 3y ago
      They've made Zookeeper optional now, although they're being understandably conservative about that (e.g. only declared it production-ready a year or two ago).
  - insanitybit 3y ago
    Getting a distributed set up plugged into your k8s/whatever framework can be a pain.
    - EdwardDiego 3y ago
      Have you tried Strimzi?
    - 3y ago
davewritescode 3y ago
Is the plan to eventually open source this and provide paid support?
waynesonfire 3y ago
company competing with kafka says kafka is dead, is that accurate?
- tiew9Vii 3y ago
  Kafka the Apache open source project is kind of dead/legacy for Confluent. Confluent re-wrote the backend of Kafka for their SASS product and use the public Kafka wire protocol for public integration https://www.confluent.io/blog/cloud-native-data-streaming-kafka-engine/ https://www.confluent.io/blog/cloud-native-data-streaming-ka... . I don’t think the Apache Kafka distribution is going away, Confluent will still have paying customers with support contracts wanting features and the protocol will likely still need to evolve but the open source version will likely be a burden on Confluent with their own private implementation so it’ll be interesting to see how things look in a few years. https://www.confluent.io/blog/cloud-native-data-streaming-kafka-engine/ https://www.confluent.io/blog/cloud-native-data-streaming-ka...
- insanitybit 3y ago
  No
dmytton 3y ago
The blog post mentions "P99 of ~1s of producer-to-consumer latency". What about just producer latency i.e. message successfully received into the queue ready to be picked up? S3 writes seem be in the low 100s of ms so I assume that's part of the quoted end-to-end latency.
- richieartoul 3y ago
  (WarpStream co-founder) Yeah our P99 for producer latency is ~400ms right now.
  - otterley 3y ago
    Sounds like you're currently optimized for throughput, not latency. (I work with customers for whom 10ms of latency for a 128kB write is too high.) What are your plans to improve latency?
    - richieartoul 3y ago
      (WarpStream co-founder) None currently, if you need latency that low WarpStream is not a good fit and you should probably stick with something more traditional like a very well tuned Kafka cluster.
nfw2 3y ago
The title of the article should be "Kafka is dead. Long live Warpstream." The long live part refers to the successor.
- lxe 3y ago
  You're right! I've only heard it being used as a contradictory phrase though. https://en.wikipedia.org/wiki/The_king_is_dead,_long_live_the_king https://en.wikipedia.org/wiki/The_king_is_dead,_long_live_th...!
  - jkaplowitz 3y ago
    As per your link, it's only "seemingly" contradictory - that is, it emphasizes that as soon as the (previous) king dies, the (new) king is automatically immediately in office to be wished a long life, without needing to wait for any coronation ceremony. This is the rule in many monarchies past and present, though not all. So, imagine that the (old) king dies at the end of the first clause and therefore the (new) king takes office at the beginning of the second.
- ryanworl 3y ago
  (WarpStream CTO here) This is a bit subtle I will admit, but we view the Kafka protocol as a successor here because it will outlive Kafka the implementation.
pshirshov 3y ago
Ok, intsead of providing commit log guarantees on their own they rely on S3. I'm not sure if S3 can actually provide durable commit log guarantees. There is timestamp-based last write wins policy for concurrent writes, so I'm not sure if this thingy can actually replace Kafka in all the usecases providing same guarantees.
- richieartoul 3y ago
  (WarpStream co-founder) We use S3 for data replication, availability, and durability, but not for the log ordering guarantees. For that, we have our own custom metadata store that "imposes" ordering guarantees on top of the data that is written to S3. We don't rely on any kind of timestamp-based last write wins policy for concurrent writes. In fact the agents will never write a file to S3 with the same name more than once or overwrite an existing file.
  - pshirshov 3y ago
    So, in fact it's not really stateless, you have your locking/ordering layer and you implement some consensus for that layer?
    - richieartoul 3y ago
      Its stateless from the customer's perspective since they don't have to manage the consensus layer (which is only possible because we separated data from metadata). Obviously there is state somewhere since the cluster is storing data, its just offloaded to S3 + our control plane, while still keeping all the data in the customer's cloud account.
      - pshirshov 3y ago
        I think you should write some detailed docs about your design and consistency guarantees.
endisneigh 3y ago
This looks great but reading the accidental SRE gave me two questions: We’ve had bare metal for a very long time now and it seems that managing your own bare metal hasn’t become much easier. If it were very easy, we’d see more of these sort of things be managed by the end user. That being said how are you managing this service? A cloud provider or bare metal? Both you and Ryan have much experience with foundationdb which generally is managed yourself. Speaking of which, did you go with that for your metadata store again? Why or why not?
- richieartoul 3y ago
  (WarpStream co-founder) WarpStream's current offering is a hybrid BYOC approach. The customer runs the agents in their cloud account, and we manage the metadata store for them remotely. This keeps all the customer data in their cloud account and their S3 bucket where we can't see it or touch it. It does mean that the customer has to run the WarpStream agents themselves, but they're just stateless containers that are pretty easy to manage. We considered using FoundationDB for our metadata store, but ended up not in the end. In order to make our free tier cost effective we really need to make our metadata store as efficient as possible for this specific use-case which required something a bit more custom. That said FoundationDB is a fantastic piece of technology. Best distributed database I've ever used, and I've used many :)
bothra90 3y ago
Exciting work! Some questions: 1. Any plans on open-sourcing this? 2. Why not have a tiered architecture that can provide lower latency? p99 of 1s can be too high for some use-cases. 3. Related to 2, how does WarpStream compare to tiered storage in Pulsar? [Edit 1] Added (3)
- smrtinsert 3y ago
  Yeah really kind of surprised the free tier isn't the self hosted kind. That will keep me looking.
  - richieartoul 3y ago
    (WarpStream Cofounder) FWIW we’re considering a version where you can host the metadata yourself for enterprise users. For the free tier though we didn’t think it made sense since for a workload that could fit into our free tier, it didn’t seem like anyone would want to be responsible for the metadata layer themselves. Would love your feedback on that.
    - banditelol 3y ago
      I'm also quite surprised there's no option for self hosting metadata on the free tier. To be fair, in my experience having managed metadata server for prefect (while they have similar Bring Your Own Server model) is quite hard to get right. But at the end we decided to keep maintaining it because our company prefer having all the data in our servers (including metadata). But I'm still excited to try this, since at least now I could play around (and learn) with a partially kafka compatible system, without the burden of maintaining all of Kafka parts (and costs). Thanks!
rad_gruchalski 3y ago
I’m so happy to see that someone has finally built such thing. I’ve been asking for tiered storage in Kafka since 2016 (https://gruchalski.com/posts/2016-05-08-the-case-for-kafka-cold-storage/ https://gruchalski.com/posts/2016-05-08-the-case-for-kafka-c...) but this approach is so much better. Kafka itself could solve this with tiered storage OR, at least, allowing adding volumes at runtime. But neither is possible. Very cool, have to find some time to take this for a test drive.
- richieartoul 3y ago
  (WarpStream co-founder) Thanks for the kind words! I'll just add that tiered storage is not quite the same thing because it means you still have to manage local disks and replication carefully, however briefly. Please reach out with questions any time!
  - rad_gruchalski 3y ago
    Indeed. The way I was thinking about it was to implement a different file system for it.
- Sphax 3y ago
  Just fyi, tiered storage is coming at some point in Kafka: https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=97554472#KIP405:KafkaTieredStorage-Status https://cwiki.apache.org/confluence/plugins/servlet/mobile?c...
  - rad_gruchalski 3y ago
    It’s been like that for a while.
    - EdwardDiego 3y ago
      The Uber devs are working hard at it.
bsaul 3y ago
Question : does this architecture have any difference regarding ordering or durability of messages, compared to kafka ?
- richieartoul 3y ago
  (WarpStream co-founder) Nope, we provide the same durability and ordering messages as Kafka, albeit with higher latency. We never acknowledge produce requests until they've been durably persisted in object storage (P99 ~400ms), so its the same guarantee as running a 3AZ Kafka cluster with fsync enabled.
deleted 3y ago
[deleted]
EamonnMR 3y ago
Have you considered making a plain HTTPS frontend like Confluent's REST proxy? Or even just designating a bucket as 'incoming messages' and letting people use the S3 API to write messages? Kafka is great but if you're targeting Amazon customers, you're competing with SQS too.
- ryanworl 3y ago
  [WarpStream co-founder here] Please sign up for our mailing list! We'll have some interesting things to announce related to this soon.
insanitybit 3y ago
Pretty cool. I'm definitely going to check this out.
skybrian 3y ago
As a hobbyist I probably don’t need this but I’m curious. I haven’t used any of the S3-compatible data stores. Any ideas which would be easiest for messing around with? Also, what databases are there built on S3? I think there is sqlite replication?
rickette 3y ago
Will this only exist as a managed service (SaaS) or will there be an open source (or open core) version?
slotrans 3y ago
> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it. Linkedin built Kafka for massive-scale problems that 99% of us don't have. Though technologists have a well-earned reputation for using tech they don't need, my perception is that most of us are succeeding in avoiding the use of Kafka.
- benjaminwootton 3y ago
  I’m not sure how anyone can hate Kafka? It does what it says on the tin - move data from A to B with publish/subscribe semantics. It’s quite easy to just use it as a dumb message broker with no retention if that’s all you need but if you do want to do something funky with persistence then go down that route. I’m not sure how anyone could have a negative feeling towards a vanilla, but rock solid and wildly popular open source tool. If they do then it will be about some niche feature or use case. I actually think this message reflects badly on the vendor here. Criticise or compete with Kafka on its technical merits if you like, but this is just a misrepresentation of their position in the market.
  - richieartoul 3y ago
    (WarpStream co-founder) I'm not sure what you mean. Message persistence is a fundamental feature of Kafka that almost everyone using it relies on, its not some esoteric feature no one uses. We're each coming from our own network bias here, but in my experience a lot of people are really unhappy with the operational toil associated with running Kafka at scale in production.
    - benjaminwootton 3y ago
      Yes it’s fundamental, but it’s not generally that significant to users of Kafka. As a developer, Kafka is a place to publish and subscribe to data with reliability and performance. As a developer, the fact that messages are persistent is nothing more than a cool feature in that I can replay messages if I need to. Things like consumer groups and offsets are features of the API, but they aren’t complex. Every similar tool whether it be RabbitMQ or IBM MQ has its own API abstractions and features. Likewise, I need to learn about failover semantics, but that’s the same with any API dependency. It seems that you and the other posters here have a concensus that it’s hard to operate. Rather than saying that Kafka is dead or a polarising technology, a better line of argument is that it’s simply hard or expensive to operate at scale. (I personally think that’s par for the course with a technology like this, but that’s an aside.) You have to remember that for everyone operating Kafka, there will be on average tens or hundreds of developers using it. And the vast, vast majority of those will not find it to be particularly polarising. Instead, they’ll find it a de-facto choice.
esafak 3y ago
How do Pulsar and Kafka compare these days?
sgt 3y ago
Can this be used without S3?
- richieartoul 3y ago
  (WarpStream co-founder) It requires some form of object storage (S3, GCS, Azure blob storage, etc). You could use minio if you want to host the storage yourself, although I think I need to make a few changes to make that configurable in the agent. For local development you can use an in-memory or local file storage, but not for "real" usage.
  - sgt 3y ago
    Okay - yes, it would make a lot of sense as a Kafka alternative to those of us who prefer to stay away from the public clouds.
computershit 3y ago
Can I use minio?
- richieartoul 3y ago
  (WarpStream co-founder) That does not work currently because there is no way to configure the agent with a Minio URL. That's just an oversight on our part though, I'll get that working within a few days.
deleted 3y ago
[deleted]
derefr 3y ago
As someone who's been doing async-fanout physical-replication of Postgres instances for the longest time using pgBackRest's S3 repo support (the primary writes to S3; the replicas read from S3; the primary's uplink doesn't get saturated serving the writes) I've always wondered where the equivalent for message-queue systems was. Glad to see it :)
jolux 3y ago
Haven’t seen it mentioned yet so I’ll just say we use Kinesis Data Streams with enhanced fanout at work and it seems very minimal in terms of operational overhead if it fits your needs.
- richieartoul 3y ago
  (WarpStream co-founder) Yeah if you can fit your workload into it, it's pretty solid from an operational perspective since AWS manages the whole thing (except scaling up shards and stuff like that). It gets pretty pricey though for large volumes of Data.
nathas 3y ago
Is WarpStream considering hiring Aphyr to do a Jepsen test?
- guywithabike 3y ago
  WarpStream relies on a proprietary metadata store hosted within their internal network to operate, so it's pretty unlikely that Jepsen tests could cover that. If you're ok with the externally hosted metadata stores as well as the high per-request latencies (p99 of 400ms, according to WarpStream), it's highly likely that things like liveness and safety properties are pretty far from your mind. So, I wouldn't bank on them submitting to a Jepsen test. :)
  - abhinav-t 3y ago
    I think that WarpStream relying on a proprietary metadata store isn't an issue for Jepsen tests. If I understand correctly, Jepsen tests treat the distributed databases (or logs like Kafka) pretty much as a black box. Jepsen tests introduce partitions and look for missing/unexpected items against the items that were acknowledged as written successfully by the system. If you look at Kyle's blog post, https://aphyr.com/posts/293-jepsen-kafka https://aphyr.com/posts/293-jepsen-kafka, there is no mention of looking into a broker's storage or any storage for that matter.
  - richieartoul 3y ago
    (WarpStream co-founder) FWIW we subject WarpStream to continuous chaos/fault injection in our integration tests and staging environment to verify correctness and liveness properties. I wouldn't say they're far from our mind, we've just made a big trade off around latency that we think will make sense for a lot of people.
Hixon10 3y ago
I wonder, will we have any solution for local development? Or we always will depend on closed control plane?
- richieartoul 3y ago
  (WarpStream co-founder) We're thinking about making the `warpstream playground` command run an in-memory local version of control plane. Come chat with us in our Slack or Discord if you want to discuss more!
onetimeuse92304 3y ago
I think the focus on the $/GB is wrong. The main goal of Kafka is providing certain processing guarantees and connecting consumers with producers of events. What Kafka is not for is dealing with enormous number of events or enormous amount of data. Not that it is particularly slow at it (actually, quite fast) but I see a lot of people needlessly try to push more data and events than they need and then complain they have problems. One easy trick to deal with enormous numbers of events and amounts of data is batching. Just batch some number of events into a single event so that a consumer can pick it up in one go. Removes a lot of cost of transferring the event. Too much data to transfer? Don't be stupid and try to transfer those huge documents through the Kafka topic -- there is no need for this. Just upload them to S3 and pass through a reference. Or even better, take 10k events, extract data, zip it up, push to S3, and THEN pass a single event that describes 10k large events.
- ec109685 3y ago
  You’ve described the tricks WarpSream does in their implementation, buffering data until a large enough chunk can be written to S3.
  - onetimeuse92304 3y ago
    I didn't know that. I worked with some pretty large scale projects and solutions like this are my daily bread and butter to the point where it seems obvious to me but I have no idea what is/isn't obvious to general population of developers.
purpleblue 3y ago
What's to stop Confluent/Kafka from simply implementing a feature like this themselves?
- mdaniel 3y ago
  their recent blog post doesn't sound that far off from the OP's architecture: https://www.confluent.io/blog/cloud-native-data-streaming-kafka-engine/#networking-and-routing https://www.confluent.io/blog/cloud-native-data-streaming-ka... (discussed: https://news.ycombinator.com/item?id=36870278 https://news.ycombinator.com/item?id=36870278 )

392 points by richieartoul 3y ago

poorman 3y ago

Oh man, I can't wait until these people get their first S3 bill for the number or GET requests.

ryanworl 3y ago
[WarpStream co-founder here] My co-founder and I worked at Datadog for over 3 years where we built Husky, an event storage and query system built directly on top of S3 as well. We know what we're doing here, I promise ;) https://www.datadoghq.com/blog/engineering/introducing-husky/ https://www.datadoghq.com/blog/engineering/introducing-husky...
- poorman 3y ago
  What happens if I pull the plug on a server that hasn't flushed to S3 in the configured time? Do I loose that event?
  - ryanworl 3y ago
    With an appropriately configured client (i.e. one that retries and waits for requests to be acknowledged), another Agent would receive the retry and the event would be written to the topic at that time.
- madmax108 3y ago
  "My co-founder and I worked at Datadog" If there's one company that's known for blowing up budgets as you scale, it's Datadog. Hoping WarpStream doesn't follow that playbook. Best wishes!
richieartoul 3y ago
(WarpStream co-founder) Check the end of our blog post, we brought receipts :)

88913527 3y ago

It's refreshing to see cloud computing costs matter. Hopefully it'll cut down of needless cloud complexity and the proliferation of indirection in software abstractions of the last decade.

richieartoul 3y ago
(WarpStream co-founder) Cheers! We really want to drive the incremental cost (both in terms of $ and management overhead) per GiB down as low as possible so people can start using these systems for more use-cases.

deleted 3y ago

[deleted]

liminal 3y ago

Can this be installed locally and run on Minio?

richieartoul 3y ago
Will make this possible soon, oversight on our part that the agent doesn't expose a way to let you do that right now

yingjunwu 3y ago

I am the founder of RisingWave (http://risingwave.com/ http://risingwave.com/), an open-source SQL streaming database. I am happy to see the launch of Warpstream! I just reviewed the project and here's my personal opinion: * Apache Kafka is undoubtedly the leading product in the streaming platform space. It offers a simple yet effective API that has become the golden standard. All streaming/messaging vendors need to adhere to Kafka protocol. * The original Kafka only used local storage to store data, which can be extremely expensive if the data volume is large. That's why many people are advocating for the development of Kafka Tiered Storage (KIP-405: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...). To my best knowledge, there are at least five vendors selling Kafka or Kafka-compatible products with tiered storage support: -- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/ https://www.confluent.io/10x-apache-kafka/; -- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafka https://github.com/Aiven-Open/tiered-storage-for-apache-kafk...); -- Redpanda Data, which cuts your TCO by 6X (https://redpanda.com/platform-tco https://redpanda.com/platform-tco); -- DataStax, which commercializes Apache Pulsar (https://pulsar.apache.org/ https://pulsar.apache.org/); -- StreamNative, which commercializes Apache Pulsar (https://pulsar.apache.org/ https://pulsar.apache.org/). * WarpStream claims to be "built directly on top of S3," which I believe is a very aggressive approach that has the potential to drastically reduce costs, even compared to tiered storage. The potential tradeoff is system performance, especially in terms of latency. As new technology, WarpStream brings novelty, and definitely it also needs to convince users that the service is robust and reliable. * BYOC (Bring Your Own Cloud) is becoming the default option. Most of the vendors listed above offer BYOC, where data is stored in customers' cloud accounts, addressing concerns about data privacy and security. I believe WarpStream is new technology to this market, and and would encourage the team to publish some detailed numbers to confirm its performance and efficiency!

mdaniel 3y ago
If you're still within the edit window for your comment, the ");" in them are attached to the URLs, making them problematic to click ed -- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/ https://www.confluent.io/10x-apache-kafka/ -- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafka https://github.com/Aiven-Open/tiered-storage-for-apache-kafk... -- Redpanda Data, which cuts your TCO by 6X https://redpanda.com/platform-tco https://redpanda.com/platform-tco -- DataStax, which commercializes Apache Pulsar https://pulsar.apache.org/ https://pulsar.apache.org/ -- StreamNative, which commercializes Apache Pulsar https://pulsar.apache.org/ https://pulsar.apache.org/

hintymad 3y ago

So the idea is that S3 offers practically unlimited throughput, so we can certainly use S3 for a scalable queue? I wish the open-source community could have something competing with S3, but then S3 has made it really hard for companies to have a convincing justification to build their own alternative[1]. [1] There are a few alternatives, such as Apache Ozone, Ceph, and Good'ol HDFS, but none seems have an accelerating momentum for S3's workload.

Exuma 3y ago

Care to summarize the 'contact us' pricing?

ryanworl 3y ago
We're aiming for per-GB usage-based pricing that is significantly cheaper than the alternatives, but the BYOC model combined with our extremely efficient cloud control plane gives us a lot of flexibility here. That's why we can offer a free tier at all. We're mostly just not sure yet, so your input would be appreciated.

pfannkuchen 3y ago

I feel like people misuse the X is dead, long live X construction, but maybe I’m missing something. I’ve always taken it to be a reference to the stability and near-programmatic nature of British royal succession. So while “the king is dead” in some places historically would be a disaster with conflict surely following, in Britain it automatically and instantly becomes “long live the king” for the next king and there is no panic.

EdwardDiego 3y ago

Dear richieartoul, your blog post is a tad oversalted. Kafka doesn't inherently require dedicated teams of experts and millions of dollars until you're running very large clusters. But fully agree that 3-AZ stretch clusters suck money through those inter-AZ xfer fees. Which is how AWS sells MSK, the inter-AZ xfer is "free"! That is, already priced in...

jjwiseman 3y ago

This comment from 4 months ago helped me understand when you might want to use kafka: https://news.ycombinator.com/item?id=35160555 https://news.ycombinator.com/item?id=35160555 (The comment is by rad_gruchalski, who I see is also active in this thread. Thanks, rad_gruchalski!)

siliconc0w 3y ago

It's a neat idea for a SaaS to just host the control-plane/metadata and allow customers to write to their own data store via S3. I can't think of too many other systems work that way - it certainly is operationally simpler but I wonder if it also gets you some compliance wins. I wonder if we'll eventually standardize on a cloud-agnostic S3-like protocol but for locking and fast transactional writes so we can just have companies building pure software again but using standardized Cloud instead of POSIX APIs.

edude03 3y ago

Okay, I hate to be that guy but I'm given two options for pricing > Free but with only 24 Hours of Retention or > Contact us. Pricing is hard for sure, but if I'm being honest neither of these options make me want to try the product.

richieartoul 3y ago
(WarpStream co-founder) I know that is super frustrating, I’m sorry. We’re just really early and unsure how to begin even discussing public pricing. We are 100% committed to that free tier. We designed our entire architecture around making that free tier so cost effective that we could offer it for free. We want to get to a point where we can have transparent pricing on our website, it will just take us some time to figure that out after speaking with initial customers. We would love to hear your feedback though. You won’t get handed off to a sales guy or anything, it’ll just be a direct conversation with me (CEO) and my cofounder Ryan (CTO)

topaz0 3y ago

Kafka's been dead almost 100 years.

backendanon 3y ago

I didn't check the site. I have nightmares debugging kafka from years ago and it's only huger now. No thank you, give me Apache MQ or RabbitMQ, even IBM's MQ and it's all good and don't try to tell me I need a streaming thingamajig.

rupurt 3y ago

This is super interesting @richieartoul. I specced out something similar myself and was going to implement it in Zig https://github.com/fremantle-industries/transit https://github.com/fremantle-industries/transit. FWIW I came to a similar conclusion that a lot of the power in Kafka comes from the API and that eventually much of the complexity of managing the cluster will eventually be abstracted away with multiple implementations. I also felt that if I could implement Kafka persistent over the S3 keyspace then I could start with persistence direct to S3 like you've done with warpstream and then layer on a faster hot disk and in memory tiering mechanism to eventually lower end to end latencies. I love where you're going with this so hit me up on twitter if you ever want to chat more in-depth https://twitter.com/rupurt https://twitter.com/rupurt.

richieartoul 3y ago
DMd you on twitter

d_t_w 3y ago

Hi @richieartoul, does warpstream support the standard Kafka Admin API? We build an admin console / dev tooling for Kafka (https://kpow.io https://kpow.io) that supports Kafka 1.0+ including Redpanda due to their fairly strict adherence to those API. Warpstream seems like a cool idea, I'd like to see what happens if we plug Kpow on top of it, if that's possible.

richieartoul 3y ago
Our kafka protocol support is still incomplete. We’ve documented our progress here: https://docs.warpstream.com/warpstream/reference/kafka-protocol-and-feature-support https://docs.warpstream.com/warpstream/reference/kafka-proto... That said we’d be stoked to get this working as an additional tool for people. Do you want to shoot me an email or join our slack so we can discuss further? I can probably prioritize whatever protocol features were missing to get it working. founders@warpstreamlabs.com

kabochya 3y ago

We've built something probably very similar to this product at my previous job. We had double digit TB daily ML traffic that didn't require realtime latency, so we moved those all onto S3 and also saw approx ~90% cost savings. This was built on top of JVM and still used a 6 broker Kafka cluster to keep metadata (vs probably 300 when it was originally all on Kafka). Kafka's compute and storage model doesn't scale too well for the extreme use cases that can tolerate latency, and the Apache Pulsar model sort of worked better (though at that time Pulsar wasn't too stable for us to use in prod). One of the keys to cost efficiency for us was that the size of the data was large enough that we didn't need to wait too long before hitting an economic file size to upload. Trying to imagine how a pipeline with less than 10 MB/s would work with this efficiently.

richieartoul 3y ago
(WarpStream co-founder) Yeah we’ve run into a number of people who’ve rolled their own solution in this space. The “push pointers to S3 through traditional Kafka” approach is a very practical one. Was this memq at Pinterest, or something else?
- kabochya 3y ago
  Yeah it is, sort of glad that it's actually known. I've since left the ingestion side of data infra so not very familiar of the landscape after one year.

steve76 3y ago

[dead]

Spiwux 3y ago

> how many partitions do I need? Unclear, but better get it right because you can never change it! That is.. simply wrong. You can change the number of partitions. Plus I really don't get the "you need an entire team of engineers to operate Kafka" claim you keep repeating. That is simply not true, speaking from experience. It is certainly expensive to run, but doesn't require a lot of engineering hours in our team.

FridgeSeal 3y ago

> requires teams to operate it I mean, managed Kafka has been a thing for a long while now. I’ve used it plenty and haven’t had any issues…

jooz 3y ago

The criticism I have through kafka is the "at-least-once" semantics. How do you manage exactly-once semantics? If kafka performance is based on reading small batches of 50 messages, in case of crash of the consumer, some of them will be processed twice. Depending on your business logic this may be ok, or may be create a new problem that must be solved farther in the process by adding an external data store.

kilotaras 3y ago
You can't have exactly-once in distributed system. Crashes happen, network or power goes down and your ack get's lost[0]. The best one can do is accept that and make your processing idempotent. [0] The most interesting case I've witnessed was power going down because a hot air balloon crashed into power line.

benbuick 3y ago

Hey Richie, As you seem to support the Kafka + Kinesis(?) API: We have developed Kadeck (kadeck.com), a control plane and collaboration layer for Kafka & Kinesis teams. I have looked at the documentation and reckon we should be supporting a lot of Warpstream already. Is there any quick way to test this out? Would love to share this with our users! Whether dead or alive, there's no denying that Kafka, or data streaming in general, is one of the most exciting fields today. Happy to welcome a new member!

richieartoul 3y ago
(WarpStream Cofounder) https://docs.WarpStream.com https://docs.WarpStream.com is the best document we have right now. This sounds interesting though, can you jump in our slack or shoot me an email at founders@warpstreamlabs.com ? Happy to support you any way we can!

substation13 3y ago

Is there any source available for this?

bobby_the_whale 3y ago

[dead]

throw4847285 3y ago

I was hoping this would be about Franz.

aftermodern 3y ago

The object storage backend is interesting and I’m wondering if Kafka will have a native option for that, eventually. Confluent hasn’t outright advertised it, but one of their engineers mentioned in a q&a that their Cloud offering is using object storage for their own backend now.

relay23 3y ago
They always have.. you get it but you don't know it.

dtjohnnymonkey 3y ago

I've seen both your FoundationDB talks, curious if you are using FoundationDB under the hood or how much of the metadata store is homegrown? Also, how did you implement the parsing of the Kafka protocol? From scratch or pieced together with open source Go bits? Having a nice programmable API for building Kafka-compatible servers could be a huge boon to the Kafka community (wink wink).

michaelcampbell 3y ago

> WarpStream is 5-10x cheaper than Kafka <sigh> I'm guessing they mean 1/10 to 1/5 the cost? "1x cheaper" would be free.

ryanworl 3y ago

I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges. If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.

kbumsik 3y ago
Does it support S3-compatible services, notably Cloudflare R2? I heard that there might be special handling for each S3-compatible providers, due the slightly different API behavior and different consistency models, and etc. If it support Cloudflare R2 then it would be great for multi-cloud too.
geekone 3y ago
What was used to generate the diagrams in the post?
dottedmag 3y ago
Do you have a reference documentation for S3 data layout?
- richieartoul 3y ago
  (WarpStream founder here) Not currently. One of the things we're looking to do next is make it so any topic can be "automatically" turned into a standard format in S3, something like Parquet/Iceberg/Deltalake so its easier to consume for application that don't particularly care about the Kafka protocol.
  - EamonnMR 3y ago
    That would be a killer feature. Once we switched to MSK back at a previous gig, our biggest care and feeding task was adding new event types to our kafka-to-redshift ETL thing. Well, that and dealing with scaling our http wrapper...

cduzz 3y ago

Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clusters, etc. 3. What's done to preserve order-of-operation within a kafka partition?

cbsmith 3y ago
The way Kafka works, it naturally buffers & coalesces messages even before they get to the brokers, so yes, of course the messages are being coalesced. There is no problem with "just" running a Kafka cluster in each AZ and only replicating data between AZs until it's time to pull it all together. It's just that when presented with a distributed system and AZs, engineers (and in fairness the business requirements) are more than likely to go with a multi-AZ solution. Same goes for regions. So the vast majority of Kafka clusters are multi-AZ but probably shouldn't be, and Kafka gets the bill for that, even though it shouldn't. The Kafka protocol doesn't really preserve order-of-operation within a Kafka partition. It preserves the order of operations within a producer-partition pair (and even then, only if you configure it a certain way). The standard implementation does this by preserving the order-of-broker-receipt-of-messages from producers, but from an external system's vantage point, it really only means that (if configured the right way) messages with any given key, from any given producer, will be preserved in the order they are received.
- cduzz 3y ago
  The architect mentioned a 100ms (or 10 times per second) buffer / flush rate; presumably there's some windowing settings so I can flush at 10,000 records or after 1/10th of a second, and can chose 10 records or 1/100th of a second if I don't mind the bill. Kafka's nice because there are a lot of knobs to adjust how you prioritize availability or durability or latency. Kafka's tedious because there are a lot of knobs... I was curious as to the nature of the knobs on this product. I probably should have also asked "and by S3 do you mean S3 or any S3 like object store?" probably that's answered elsewhere.

slantedview 3y ago

This article seems to be spreading some FUD, in particular this comment: > Cloud economics – by design, Kafka’s replication strategy will rack up massive inter AZ bandwidth costs. You're no more or less forced to put Kafka replicas in different AZs than you would be with an alternative.

richieartoul 3y ago
(WarpStream Founder) That's fair, but we think people shouldn't have to choose between multiple AZs and costs. All cloud providers recommend that critical workloads running in cloud environments should run in more than one AZ.
crimsonpowder 3y ago
Of course, as soon as an AZ goes down that had all of your Kafka nodes in it, you get eviscerated on HN for running such a setup.
- EdwardDiego 3y ago
  3 AZ Kafka is the safest, but I've run several 2.5 clusters just fine. (cluster stretched across 2 AZs with a tiebreaker ZK in a 3rd AZ to maintain quorum in the event that you lose an AZ)

rcarmo 3y ago

Well, one thing's for sure. Running Kafka on discrete VMs on a cloud provider "by the book" is ludicrously expensive. I remember having a very simple discussion with quite a few customers about both Kafka and Hadoop that boiled down to this: Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? (in this case it was Azure storage, which provides locally, redundant, AZ-redundant or globally redundant storage, most of which are available to run managed disks upon). This is why properly designed Hadoop/Kafka cloud managed services employ storage adapters to leverage the provider's baked in redundancy. And why some cloud providers have Kafka-compatible event brokers. The rest of what WarpStream does is just icing on the cake (although I'm curious as to the internals and how they avoid inter-AZ charges). (full disclosure: I work at Microsoft, but built Hadoop/Spark/Kafka clusters before joining nearly a decade ago.)

chaotic-good 3y ago
> Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? That's easy. EBS and similar solutions comes with the price. They're very expensive. Especially, when you need a lot of IOPs. You may be saving on cross-AZ traffic but you will pay ridiculous amount of money on storage. If you have replication you can use attached storage which is way cheaper.
- rcarmo 3y ago
  Azure Managed Disks are inherently replicated. But there are other storage solutions (like redundant WebHDFS) you can take advantage of. At the time, the customers I was talking to wanted to have identical deployments in the cloud (to what they had on-premises, down to the number and size of disks).
hinkley 3y ago
In the good ol' days, we used to implement redundant systems such that there was tie breaker process that used only a fraction of the resources of the true processes. Some Raft implementations allow for nodes that are voters but cannot be quorum leaders (example: the branch office where all traffic is funneled through an asymmetric VPN tunnel should never be elected leader, but knows which candidates it can and cannot see) So the table stakes for running a cluster was not 3x as much hardware but closer to 2.2x, which is a huge deal for solutions in the small and developer sandboxes. It also matters when 3 shards don't quite cover your load but 5 is too many. Or 6 vs 7. The problem is that with geographic replication, this doesn't fix either of the problems articulated as part of the thesis of this article: 1. Cloud economics – by design, Kafka’s replication strategy will rack up massive inter AZ bandwidth costs. 2. Operational overhead – running your own Kafka cluster literally requires a dedicated team and sophisticated custom tooling. Still, we need this functionality back for cloud, particularly as the pendulum swings back to self hosting, which it always has in the past.

ant6n 3y ago

I keep seeing these HN titles about technologies named for other concepts, thinking it's talking about those other contexts, without any indication it's about those obscure technologies. Then I'm disappointed its just about some random, badly named technology. So, in honor of actual Zombie Kafka, here a short news blurb about Kafka international airport being Named most Alienating Airport: https://www.youtube.com/watch?v=gEyFH-a-XoQ https://www.youtube.com/watch?v=gEyFH-a-XoQ

tekla 3y ago
Seriously. I ordered Windows and all I got was a piece of cardboard with some random letters and numbers on it, and not a frame with glass.

MapleWalnut 3y ago

It'd be great if there was an rss feed for your blog

whalesalad 3y ago

Running Kafka is an exercise in sadomasochism. This on the other hand sounds pretty neat.

8organicbits 3y ago
I've heard good things about Red Panda[1], but I haven't tried it. [1] https://redpanda.com/ https://redpanda.com/
whatscooking 3y ago
What? It’s pretty easy to set up and get going
- whalesalad 3y ago
  The last time I was running a cluster it was 2014-2015 and it was not too bad to get up and running, but it felt so heavy handed for what it was. The zookeeper requirement was frustrating. The fact that consumers needed to maintain offset state was frustrating (although... understandable). For the right use case I suppose it is 100% worthwhile but more often than not I have seen it used in places where other tools would be more appropriate.
  - lmm 3y ago
    They've made Zookeeper optional now, although they're being understandably conservative about that (e.g. only declared it production-ready a year or two ago).
- insanitybit 3y ago
  Getting a distributed set up plugged into your k8s/whatever framework can be a pain.
  - EdwardDiego 3y ago
    Have you tried Strimzi?
  - 3y ago

davewritescode 3y ago

Is the plan to eventually open source this and provide paid support?

waynesonfire 3y ago

company competing with kafka says kafka is dead, is that accurate?

tiew9Vii 3y ago
Kafka the Apache open source project is kind of dead/legacy for Confluent. Confluent re-wrote the backend of Kafka for their SASS product and use the public Kafka wire protocol for public integration https://www.confluent.io/blog/cloud-native-data-streaming-kafka-engine/ https://www.confluent.io/blog/cloud-native-data-streaming-ka... . I don’t think the Apache Kafka distribution is going away, Confluent will still have paying customers with support contracts wanting features and the protocol will likely still need to evolve but the open source version will likely be a burden on Confluent with their own private implementation so it’ll be interesting to see how things look in a few years. https://www.confluent.io/blog/cloud-native-data-streaming-kafka-engine/ https://www.confluent.io/blog/cloud-native-data-streaming-ka...
insanitybit 3y ago
No

dmytton 3y ago

The blog post mentions "P99 of ~1s of producer-to-consumer latency". What about just producer latency i.e. message successfully received into the queue ready to be picked up? S3 writes seem be in the low 100s of ms so I assume that's part of the quoted end-to-end latency.

richieartoul 3y ago
(WarpStream co-founder) Yeah our P99 for producer latency is ~400ms right now.
- otterley 3y ago
  Sounds like you're currently optimized for throughput, not latency. (I work with customers for whom 10ms of latency for a 128kB write is too high.) What are your plans to improve latency?
  - richieartoul 3y ago
    (WarpStream co-founder) None currently, if you need latency that low WarpStream is not a good fit and you should probably stick with something more traditional like a very well tuned Kafka cluster.

nfw2 3y ago

The title of the article should be "Kafka is dead. Long live Warpstream." The long live part refers to the successor.

lxe 3y ago
You're right! I've only heard it being used as a contradictory phrase though. https://en.wikipedia.org/wiki/The_king_is_dead,_long_live_the_king https://en.wikipedia.org/wiki/The_king_is_dead,_long_live_th...!
- jkaplowitz 3y ago
  As per your link, it's only "seemingly" contradictory - that is, it emphasizes that as soon as the (previous) king dies, the (new) king is automatically immediately in office to be wished a long life, without needing to wait for any coronation ceremony. This is the rule in many monarchies past and present, though not all. So, imagine that the (old) king dies at the end of the first clause and therefore the (new) king takes office at the beginning of the second.
ryanworl 3y ago
(WarpStream CTO here) This is a bit subtle I will admit, but we view the Kafka protocol as a successor here because it will outlive Kafka the implementation.

pshirshov 3y ago

Ok, intsead of providing commit log guarantees on their own they rely on S3. I'm not sure if S3 can actually provide durable commit log guarantees. There is timestamp-based last write wins policy for concurrent writes, so I'm not sure if this thingy can actually replace Kafka in all the usecases providing same guarantees.

richieartoul 3y ago
(WarpStream co-founder) We use S3 for data replication, availability, and durability, but not for the log ordering guarantees. For that, we have our own custom metadata store that "imposes" ordering guarantees on top of the data that is written to S3. We don't rely on any kind of timestamp-based last write wins policy for concurrent writes. In fact the agents will never write a file to S3 with the same name more than once or overwrite an existing file.
- pshirshov 3y ago
  So, in fact it's not really stateless, you have your locking/ordering layer and you implement some consensus for that layer?
  - richieartoul 3y ago
    Its stateless from the customer's perspective since they don't have to manage the consensus layer (which is only possible because we separated data from metadata). Obviously there is state somewhere since the cluster is storing data, its just offloaded to S3 + our control plane, while still keeping all the data in the customer's cloud account.
    - pshirshov 3y ago
      I think you should write some detailed docs about your design and consistency guarantees.

endisneigh 3y ago

This looks great but reading the accidental SRE gave me two questions: We’ve had bare metal for a very long time now and it seems that managing your own bare metal hasn’t become much easier. If it were very easy, we’d see more of these sort of things be managed by the end user. That being said how are you managing this service? A cloud provider or bare metal? Both you and Ryan have much experience with foundationdb which generally is managed yourself. Speaking of which, did you go with that for your metadata store again? Why or why not?

richieartoul 3y ago
(WarpStream co-founder) WarpStream's current offering is a hybrid BYOC approach. The customer runs the agents in their cloud account, and we manage the metadata store for them remotely. This keeps all the customer data in their cloud account and their S3 bucket where we can't see it or touch it. It does mean that the customer has to run the WarpStream agents themselves, but they're just stateless containers that are pretty easy to manage. We considered using FoundationDB for our metadata store, but ended up not in the end. In order to make our free tier cost effective we really need to make our metadata store as efficient as possible for this specific use-case which required something a bit more custom. That said FoundationDB is a fantastic piece of technology. Best distributed database I've ever used, and I've used many :)

bothra90 3y ago

Exciting work! Some questions: 1. Any plans on open-sourcing this? 2. Why not have a tiered architecture that can provide lower latency? p99 of 1s can be too high for some use-cases. 3. Related to 2, how does WarpStream compare to tiered storage in Pulsar? [Edit 1] Added (3)

smrtinsert 3y ago
Yeah really kind of surprised the free tier isn't the self hosted kind. That will keep me looking.
- richieartoul 3y ago
  (WarpStream Cofounder) FWIW we’re considering a version where you can host the metadata yourself for enterprise users. For the free tier though we didn’t think it made sense since for a workload that could fit into our free tier, it didn’t seem like anyone would want to be responsible for the metadata layer themselves. Would love your feedback on that.
  - banditelol 3y ago
    I'm also quite surprised there's no option for self hosting metadata on the free tier. To be fair, in my experience having managed metadata server for prefect (while they have similar Bring Your Own Server model) is quite hard to get right. But at the end we decided to keep maintaining it because our company prefer having all the data in our servers (including metadata). But I'm still excited to try this, since at least now I could play around (and learn) with a partially kafka compatible system, without the burden of maintaining all of Kafka parts (and costs). Thanks!

rad_gruchalski 3y ago

I’m so happy to see that someone has finally built such thing. I’ve been asking for tiered storage in Kafka since 2016 (https://gruchalski.com/posts/2016-05-08-the-case-for-kafka-cold-storage/ https://gruchalski.com/posts/2016-05-08-the-case-for-kafka-c...) but this approach is so much better. Kafka itself could solve this with tiered storage OR, at least, allowing adding volumes at runtime. But neither is possible. Very cool, have to find some time to take this for a test drive.

richieartoul 3y ago
(WarpStream co-founder) Thanks for the kind words! I'll just add that tiered storage is not quite the same thing because it means you still have to manage local disks and replication carefully, however briefly. Please reach out with questions any time!
- rad_gruchalski 3y ago
  Indeed. The way I was thinking about it was to implement a different file system for it.
Sphax 3y ago
Just fyi, tiered storage is coming at some point in Kafka: https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=97554472#KIP405:KafkaTieredStorage-Status https://cwiki.apache.org/confluence/plugins/servlet/mobile?c...
- rad_gruchalski 3y ago
  It’s been like that for a while.
  - EdwardDiego 3y ago
    The Uber devs are working hard at it.

bsaul 3y ago

Question : does this architecture have any difference regarding ordering or durability of messages, compared to kafka ?

richieartoul 3y ago
(WarpStream co-founder) Nope, we provide the same durability and ordering messages as Kafka, albeit with higher latency. We never acknowledge produce requests until they've been durably persisted in object storage (P99 ~400ms), so its the same guarantee as running a 3AZ Kafka cluster with fsync enabled.

deleted 3y ago

[deleted]

EamonnMR 3y ago

Have you considered making a plain HTTPS frontend like Confluent's REST proxy? Or even just designating a bucket as 'incoming messages' and letting people use the S3 API to write messages? Kafka is great but if you're targeting Amazon customers, you're competing with SQS too.

ryanworl 3y ago
[WarpStream co-founder here] Please sign up for our mailing list! We'll have some interesting things to announce related to this soon.

insanitybit 3y ago

Pretty cool. I'm definitely going to check this out.

skybrian 3y ago

As a hobbyist I probably don’t need this but I’m curious. I haven’t used any of the S3-compatible data stores. Any ideas which would be easiest for messing around with? Also, what databases are there built on S3? I think there is sqlite replication?

rickette 3y ago

Will this only exist as a managed service (SaaS) or will there be an open source (or open core) version?

slotrans 3y ago

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it. Linkedin built Kafka for massive-scale problems that 99% of us don't have. Though technologists have a well-earned reputation for using tech they don't need, my perception is that most of us are succeeding in avoiding the use of Kafka.

benjaminwootton 3y ago
I’m not sure how anyone can hate Kafka? It does what it says on the tin - move data from A to B with publish/subscribe semantics. It’s quite easy to just use it as a dumb message broker with no retention if that’s all you need but if you do want to do something funky with persistence then go down that route. I’m not sure how anyone could have a negative feeling towards a vanilla, but rock solid and wildly popular open source tool. If they do then it will be about some niche feature or use case. I actually think this message reflects badly on the vendor here. Criticise or compete with Kafka on its technical merits if you like, but this is just a misrepresentation of their position in the market.
- richieartoul 3y ago
  (WarpStream co-founder) I'm not sure what you mean. Message persistence is a fundamental feature of Kafka that almost everyone using it relies on, its not some esoteric feature no one uses. We're each coming from our own network bias here, but in my experience a lot of people are really unhappy with the operational toil associated with running Kafka at scale in production.
  - benjaminwootton 3y ago
    Yes it’s fundamental, but it’s not generally that significant to users of Kafka. As a developer, Kafka is a place to publish and subscribe to data with reliability and performance. As a developer, the fact that messages are persistent is nothing more than a cool feature in that I can replay messages if I need to. Things like consumer groups and offsets are features of the API, but they aren’t complex. Every similar tool whether it be RabbitMQ or IBM MQ has its own API abstractions and features. Likewise, I need to learn about failover semantics, but that’s the same with any API dependency. It seems that you and the other posters here have a concensus that it’s hard to operate. Rather than saying that Kafka is dead or a polarising technology, a better line of argument is that it’s simply hard or expensive to operate at scale. (I personally think that’s par for the course with a technology like this, but that’s an aside.) You have to remember that for everyone operating Kafka, there will be on average tens or hundreds of developers using it. And the vast, vast majority of those will not find it to be particularly polarising. Instead, they’ll find it a de-facto choice.

esafak 3y ago

How do Pulsar and Kafka compare these days?

sgt 3y ago

Can this be used without S3?

richieartoul 3y ago
(WarpStream co-founder) It requires some form of object storage (S3, GCS, Azure blob storage, etc). You could use minio if you want to host the storage yourself, although I think I need to make a few changes to make that configurable in the agent. For local development you can use an in-memory or local file storage, but not for "real" usage.
- sgt 3y ago
  Okay - yes, it would make a lot of sense as a Kafka alternative to those of us who prefer to stay away from the public clouds.

computershit 3y ago

Can I use minio?

richieartoul 3y ago
(WarpStream co-founder) That does not work currently because there is no way to configure the agent with a Minio URL. That's just an oversight on our part though, I'll get that working within a few days.

deleted 3y ago

[deleted]

derefr 3y ago

As someone who's been doing async-fanout physical-replication of Postgres instances for the longest time using pgBackRest's S3 repo support (the primary writes to S3; the replicas read from S3; the primary's uplink doesn't get saturated serving the writes) I've always wondered where the equivalent for message-queue systems was. Glad to see it :)

jolux 3y ago

Haven’t seen it mentioned yet so I’ll just say we use Kinesis Data Streams with enhanced fanout at work and it seems very minimal in terms of operational overhead if it fits your needs.

richieartoul 3y ago
(WarpStream co-founder) Yeah if you can fit your workload into it, it's pretty solid from an operational perspective since AWS manages the whole thing (except scaling up shards and stuff like that). It gets pretty pricey though for large volumes of Data.

nathas 3y ago

Is WarpStream considering hiring Aphyr to do a Jepsen test?

guywithabike 3y ago
WarpStream relies on a proprietary metadata store hosted within their internal network to operate, so it's pretty unlikely that Jepsen tests could cover that. If you're ok with the externally hosted metadata stores as well as the high per-request latencies (p99 of 400ms, according to WarpStream), it's highly likely that things like liveness and safety properties are pretty far from your mind. So, I wouldn't bank on them submitting to a Jepsen test. :)
- abhinav-t 3y ago
  I think that WarpStream relying on a proprietary metadata store isn't an issue for Jepsen tests. If I understand correctly, Jepsen tests treat the distributed databases (or logs like Kafka) pretty much as a black box. Jepsen tests introduce partitions and look for missing/unexpected items against the items that were acknowledged as written successfully by the system. If you look at Kyle's blog post, https://aphyr.com/posts/293-jepsen-kafka https://aphyr.com/posts/293-jepsen-kafka, there is no mention of looking into a broker's storage or any storage for that matter.
- richieartoul 3y ago
  (WarpStream co-founder) FWIW we subject WarpStream to continuous chaos/fault injection in our integration tests and staging environment to verify correctness and liveness properties. I wouldn't say they're far from our mind, we've just made a big trade off around latency that we think will make sense for a lot of people.

Hixon10 3y ago

I wonder, will we have any solution for local development? Or we always will depend on closed control plane?

richieartoul 3y ago
(WarpStream co-founder) We're thinking about making the `warpstream playground` command run an in-memory local version of control plane. Come chat with us in our Slack or Discord if you want to discuss more!

onetimeuse92304 3y ago

I think the focus on the $/GB is wrong. The main goal of Kafka is providing certain processing guarantees and connecting consumers with producers of events. What Kafka is not for is dealing with enormous number of events or enormous amount of data. Not that it is particularly slow at it (actually, quite fast) but I see a lot of people needlessly try to push more data and events than they need and then complain they have problems. One easy trick to deal with enormous numbers of events and amounts of data is batching. Just batch some number of events into a single event so that a consumer can pick it up in one go. Removes a lot of cost of transferring the event. Too much data to transfer? Don't be stupid and try to transfer those huge documents through the Kafka topic -- there is no need for this. Just upload them to S3 and pass through a reference. Or even better, take 10k events, extract data, zip it up, push to S3, and THEN pass a single event that describes 10k large events.

ec109685 3y ago
You’ve described the tricks WarpSream does in their implementation, buffering data until a large enough chunk can be written to S3.
- onetimeuse92304 3y ago
  I didn't know that. I worked with some pretty large scale projects and solutions like this are my daily bread and butter to the point where it seems obvious to me but I have no idea what is/isn't obvious to general population of developers.

purpleblue 3y ago

What's to stop Confluent/Kafka from simply implementing a feature like this themselves?

mdaniel 3y ago
their recent blog post doesn't sound that far off from the OP's architecture: https://www.confluent.io/blog/cloud-native-data-streaming-kafka-engine/#networking-and-routing https://www.confluent.io/blog/cloud-native-data-streaming-ka... (discussed: https://news.ycombinator.com/item?id=36870278 https://news.ycombinator.com/item?id=36870278 )