12 ms·
Redpanda – A Kafka-compatible streaming platform for mission-critical workloads
- agallego 6y agoI'm Alex, the founder. Happy to answer any questions.
- sorenbs 6y agoIt sounds like Vectorized is a streaming platform built on top of Redpanda. Is that the right way to think about it? If yes, then why not build it directly on top of Kafka instead?
- agallego 6y agoWe kind of started on a different note. We wanted to solve some fundamental problems (no Zookeeper - similar to KIP-500) and most of all we wanted a single binary we can ship around, something that is easy to run. I think people LOVE the kafka _api_ but they have a hard time operating clusters at scale. So we decided to keep the same API but solve the problem of operational complexity.
- MrBuddyCasino 6y ago> I think people LOVE the kafka _api_ but they have a hard time operating clusters at scale. That is very true, and this is what you should emphasise. Wish you the best of luck! The actual speed of Kafka has rarely been a concern (but huge numbers of partitions are, which makes rebalancing a pain) in my experience, in fact it was mostly overkill. But operational complexity was definitely an issue!
- agallego 6y agoWill do. Thank you. Do you have any thoughts here. How would you say it differently. We tried that on the landing page
- MrBuddyCasino 6y agoLanding page isn't bad, when you scroll the important points are there, but you have scroll. I'm not a "landing page optimisation guru", so take this with a grain of salt, but I would change it as follows. Without scrolling, all the text thats displayed is this: "Redpanda A Kafka® API compatible streaming platform for mission-critical workloads. Try Redpanda today" It tells me what it is, which is good. It does not tell me why it is better, for that I have to scroll. Kafka is already suited for mission-critical workloads, so that is not a unique value proposition. Maybe: "Redpanda - 100% Kafka API compatible, but without the headaches. Forget Zookeeper, forget rebalancing issues. Instead, enjoy reliable message delivery, 10x faster speed and ultra-low latencies due to our thread-per-core architecture." Something like that, plus a visible "call to action" button, maybe "try it out" or "download". Could also think about a pretty graph comparing latencies or smth. People love pretty graphs.
- AlejandroM_E 6y agoHey Alex, Alejandro here :wave: I was a bit surprised to see the announcement about making Redpanda open source (although very interesting for poking around!). Question: are you, as a company (Vectorized) pursuing the same business model as the Confluent one? OSS Kafka, but paid official support, cloud, connectors and registry?
- agallego 6y agoWe had success with very large customers (enterprise) and also some success with finance company (looking for 0 data loss). The idea is that we can give the kafka API compat system with 10x lower tail latencies for free and still monetize the high end of the market. I think there is a big shift when you have a single binary. i.e.: no one really complains from running ngix, etc because it's easy to get up and running. so the gist we wanted to let everyone use it and reserve the right to be the only hosted redpanda provider.
- agallego 6y agoalso i'd note one more thing which is the WASM API is going to give us an opportunity to add more enterprise value. i.e.: think GDPR compliance as long as your data streams are in JSON or some predefined format. WASM basically allows us to push computational guarantees to the storage engine itself without a separate cluster.
- cookiesboxcar 6y agowhat does WASM mean in this context?
- sorenbs 6y agohttps://webassembly.org/ https://webassembly.org/
- cookiesboxcar 6y ago
- tomp 6y agoHi Alex, I'm an amateur programming language enthusiast & designer, so I've a question related to this. Looks like the project is mostly written in C++ and Go. What was the reason for this choice? Have you considered other languages, like Rust, Zig or similar instead of C++? TBH not sure what an alternative to Go would be, maybe JVM AOT-compiled with NativeImage (but AFAIK that's still experimental). Did Go's GC and/or C++'s lack of GC help/impede the project? IMO memory management is one of the main differences between languages... the other is concurrency / memory model / undefined behavior, where JVM is significantly ahead of the rest (no undefined behavior), I'm not sure exactly where Go stands (there seems to be a memory model, but no mention of undefined behavior or lack thereof).
- agallego 6y agoHi there! Sure thing. I basically was playing with an RPC framework to make my old company project fast in 2017 see github.com/smfrpc/smf I wanted to play with dpdk then. When I started this in 2019 I wanted to use a framework that was battle tested, so seastar.io fit the bill. There is also a big part of it that I had been professionally programming in c++ for many many years. Last, this evolved from a prototype on my laptop in miami to company. I think rust would he am excellent choice as well and I bet we'll write a bunch of rust in the not too distant future.
- bashcoder 6y agoHi Alex - This looks a lot like what Scylladb did to unleash the potential of the Cassandra space, with a fully optimized C++ rewrite of an Apache project. Did you draw some inspiration from their efforts? As a satisfied Scylla convert, I'm looking forward to trying Redpanda.
- agallego 6y agohi there! scylla is great :) we are built on the same framework seastar.io so we literally share code bases in that regard. Main departure is we are only API compatible. It was an explicit choice to use Raft vs ISR and to not use ZK, etc. but indeed, seastar is a really fun framework to build storage systems in. I know ceph is also doing a re-write of a subsystem in seastar for example.
- runT1ME 6y agoI'm a fan of scylla too, but if I could go back in time I'd have recommended waiting until mid 2019 to migrate. 'Fully optimized C++ rewrite's tend to take years to become battle tested.
- bashcoder 6y agoIn fact, we couldn't switch to Scylla prior to 2019 because their support of incremental counters wasn't yet complete.
- agallego 6y agototally. today if folks. need txns, they wouldn't be a good fit for us. What we found is about 90% of use cases are covered by the base api. For reference w/ all of the versioning there is something like 144 api calls you can make to kafka, most ppl use a small subset of those via high level clients. (java, python, librdkafka, etc)
- agallego 6y agoindeed. that's why we continuously _empirically_ prove that we are in fact safe - https://vectorized.io/validating-consistency/ https://vectorized.io/validating-consistency/ there is no substitute to testing tho there are 2 levels here. 1) raft has a proof (and a great phd dissertation from diego), but what matters is if we actually implemented it correctly. so 2) is we need to continuously test it. Denis did a lot of similar work at CosmosDB (microsoft) and has spent his career working on consensus. Hopefully these eases some concerns.
- timhigins 6y agoHey, looks really interesting. Would love to see benchmarks that are open source so we all can check it out in more detail. Seems like a lot of questions on here are just: how much better is the performance and why?, so maybe showing is better than telling. Best of luck!
- agallego 6y agohi tim. absolutely, we exect ppl to test. public benchmarks coming in December. We'll probably just pick the open messaging benchmark that has been going around pulsar+kafka communities so folks can have a frame of reference. Though what is to me more interesting, is what happens when you inject failures while the benchmarks are running.
- dilyevsky 6y agoHey Alex, this looks very interesting! From browsing the docs it appears rpk tries to mess with cgroups and some kernel settings... is there compatibility mode for running unprivileged in a container environment such as kubernetes? Thanks!
- benpope 6y agoThe tuning isn't 100% necessary, it's there primarily for running max performance on metal, and you can pick and choose what to tune. In general it probably best to run with: --smp <n> --overprovisioned for a container with <n> CPUs. These are standard Seastar flags.
- agallego 6y agobasically what ben said. we also have a container image `vectorized/redpanda:latest`
- patrec 6y agoWhat's end-to-end latency for a local cluster like? What's your business model? I'm a big fan of kafka as an abstract building block, but not so much the actual implementation, which is as painful to setup as a consultancy-based business model might make you suspect it would be, especially if you need reliability. The other problem is that performance kind of sucks, apart from potential latency spikes due to GC pauses I found even the average latencies for reliable end to end (in a fast local network and on decent sized hardware) not in right order of magnitude ballpark.
- Lachedup 6y agoRegarding biz model. -Redpanda (free) - comparable features to Kafka, no limits -Redpanda Enterprise (paid) - Additional features (security, WASM, tiered storage, support etc) -Vectorized cloud (Free and paid tiers) - Hosted in AWS+ GCP
- rhodin 6y agoWhat do you mean with "Resilient to Scan-The-World"?
- polskibus 6y agoHi, I need to decide in the near future on a scalable messaging solution with at-least-once or better guarantees for our own SaaS platform. I'm looking for something that can be deployed on-prem, that's not targeted only for the public cloud. Do you have any documentation on how best to deploy redpanda with Docker and/or Ansible? What are the best practices on rolling out the cluster in-house? I've seen some docs on tuning that referred to AWS types of instances, is there something that you could refer me to that is more generic and not tied to a specific cloud vendor? I'm asking partly because we must be able to offer closed on-prem installations as well as SaaS on a cloud. I'm looking for a low-ops component that will not fail me (as often as alternatives would) :)
- Lachedup 6y agowe are happy to help and answer any questions. Join our community slack channel or hit the contact us button on vectorized.io
- manhcuongbk56 6y agoHi Alex, i don't see any client document. Can i use kafka java client with it?
- benpope 6y agoThis should work fine. If you have any issues, please report as a bug.
- eis 6y agoHey, congrats on publishing your source. It's a very interesting project indeed. I took a look around and I think with a bit expanded documentation, especially examples using the WASM transformations and maybe a some emphasis on durability and other guarantees it could grow into a great project. I'd be interested in the write amplification since you went pretty low level in your IO layer. How do you guarantee atomic writes when virtually no disk provides guarantees other than on a page level which could result in destroying already written data if a write to the same page fails - at least in theory - and so one has to resort to writing data multiple times.
- ch 6y agoFor those eager to get a look at the code it can be found here: https://github.com/vectorizedio/redpanda https://github.com/vectorizedio/redpanda
- markpapadakis 6y agoThis is fantastic news. What would you say were the hardest aspects of the design and implementation, in terms of effort and thinking that went into those accomplishments ?
- agallego 6y agoI would say, building an scalable raft implementation has been by far the hardest thing. Mostly because we started from scratch and bypassed the page-cache with our read-ahead and write-behind strategies, etc. so building tiers of caches, etc and have a working raft system was hard. Here is a post on building a dummy key-value db to test jepsen https://vectorized.io/validating-consistency/ https://vectorized.io/validating-consistency/
- returnfalse 6y agoThis is awesome! Can’t wait to try it out. One question: I assume RedPanda using raft to replicate the topic content not just metadata. Is that correct? If so, how does it perform better compared to Kafka’s ISR? Since raft might be slow for this kind of workload. If I remember correctly, Liftbridge was using raft for log replication and switched away from that because of the performance problems.
- agallego 6y agoCheckout this _extremely_ detailed work that explains exactly that - https://vectorized.io/kafka-redpanda-availability/ https://vectorized.io/kafka-redpanda-availability/ To quote my colleague Denis - The results fit the theory. Raft’s (and Redpanda’s) performance is proportional to the best of the majority of nodes, while sync replication (Kafka) works only as well as its worst-performing node. AFAIK we can push the limits of hardware on throughput. The gist is that I'd need more details exactly on what you mean raft is slow. We support the same level of acks as kafka except acks=-1 is much stronger gurantees due to the log completeness guarantee of raft. for 0 and 1 we short circuit the raft acknowledgement and return to the client to match the exepectations of acknowledgements come to be known by users. There should be no perf penalty vis-a-vis kafka in any setting I can think of. If it is, is probably a bug on our side.
- returnfalse 6y agoIt's interesting to see that there are still opportunities for further improvements in raft. Thanks for the reply!
- espadrine 6y agoOutstanding article! Are you hoping to achieve no-downtime leader kills by going with the latest Paxos improvements, as you mention at the end?
- agallego 6y agoSo many goodies to come. First we need to finish multi partition transactions that we don't yet support. We need to balance eager cluster balancing based on saturation (network, cpu, disk, etc) with other performance improvements. As is the cluster achieves steady state in a couple of seconds for affected partitions and clients in Kafka retry.
- followtherhythm 6y agoThis is so dope, I am trying it out asap!
- agallego 6y agoLet me know if you run into issues - vectorized.io/slack the entire eng team is there.
- BenoitP 6y agoNice! It'd be great to know where do the improvements came from. Is it from a different architectural design, is it from better exploitation of new hardware, is it from JVM limitations? For example, when Valhalla and Loom land in the JDK, how much of this improvement will stay?
- fnord123 6y agoI can't find the benchmarks so it's not possible to know what they measured, how they measured it, and if it's a relevant measurement. Skimming the repository it doesn't seem to be an unreasonable claim. C++ with Seastar compared to Scala on JVM. It's in line with the Scylla/Cassandra improvements.
- fnord123 6y agoAccording to this Data Engineering interview (beginning around 8m in) the improvements begin with moving from the JVM which has a latency impact. But the real improvement is that fsync on a file handle puts in a barrier in the FS. RedPanda does some work around batch coalescing, they skip the page cache to side step a bunch of kernel locks, and adaptive allocation (preallocating FS space): https://vectorized.io/dataengineering-podcast-interview/ https://vectorized.io/dataengineering-podcast-interview/ The throughput is 4x, the latency is 10x.
- tmd83 6y agoIt says XFS only for data directory in the setup guide. Curious to why that is and how does this work in MAC if that's the case.
- steeve 6y agoRedpanda is built on top of Seastar, as is ScyllaDB, so the reasons might be closely related: https://www.scylladb.com/2016/02/09/qualifying-filesystems/ https://www.scylladb.com/2016/02/09/qualifying-filesystems/
- txdv 6y agoI was so happy about just running a single small binary for kafka, but then I saw this requirement that is harder to fulfill :/
- txdv 6y agoHow much of a hard requirement is this? Can I play around on non XFS?
- benpope 6y agoThis is not a hard requirement, but XFS is highly recommended for production.
- greenkey 6y agoWouldn’t it be valid to consider Kafka/RedPanda a bottleneck and another point of failure that may delay data getting to a destination? In some cases, the performance, efficiency, and reliability gains from caching and consolidation make sense. But, I’ve seen enough poor architectural decisions and lack of architectural oversight result in use of various log streaming, cloud messenging, app monitoring, object DBs, etc., all discounting the request overhead in time and traffic, points of failure, and overall complexity for some false sense of scalability enough to where things that seemed cool ten years ago make me physically sick now. What are some questions to use to help determine whether Kafka/RedPanda actually make sense to use, without having to first baseline, then implement, then compare request time, reliability, and data freshness to gauge whether it was worth it? BTW- I think there are valid cases for using it and appreciate all of the work!
- agallego 6y agoAbsolutely. If you use transactions, we don't support it yet. We are in active development here. Notice that folks love speed but the operational simplicity of having one binary/fault domain makes a lot of our enterprise users use the tech. Last is if you like the product direction which is to my knowledge fundamentally different from the other engines out there. WASM in particular solves around 60% of all streaming work we see in the wild. It is effectively good at one shot transformations (gdpr, simple enriching, simple connection to downstream system like elastic, etc) as well as tiered storage. Think the idea was to build something as easy as nginx - apt-get install redpanda and et voila I hope to continue to focus on the developer experience.
- bouk 6y agoWho called it Kafka-compatible and not Kafkaesque
- iAm25626 6y agoWould this work with ksqlDB out the box?
- agallego 6y agoi haven't tested myself. the kafka ecosystem is huge. but we should test.
- max_streese 6y agoHi I like that there is a competitor to Kafka in this space and also the build in capability to do transformations. I got a few questions though which I could not find in your docs: (1) Over at the Apache Arrow FAQ I read that the overhead of serialization in analytical frameworks can be around 80 to 90 % of total compute costs (r_1). While having no concrete numbers on this, from using Kafka together with Kafka Streams I can at least confirm that the overhead of serialization is (very) significant. My question therefore is: Does your WASM engine avoid (de-)serialization between your storage/stream layer and the engine and if not are there plans for this? (2) Are supported WASM transformations stateless (i.e. single message) only or can they be stateful (i.e. window-ing and stream-stream/table join functionality) (3) I could not find any reference to the WASM inline lambdas at all in the docs actually, am I missing something? r_1: https://arrow.apache.org/faq/ https://arrow.apache.org/faq/
- agallego 6y agogreat questions. (1) arrow is great! currently, it does not, but yes it will when we move out of nodejs impl into our own v8 isolates inside an alien thread (seastar concept) (2) stateful but only for a single partition (3) will be released in the next week or so. If you look in the github repo you can look into `coproc`
- max_streese 6y agoAh thanks, really exited for (1)! Though the explanation went a bit over my head just one (two) follow up question(s): How did you end up with WASM for the inline lambdas? Did you have some discussion on alternatives like Lua? I am curious about insights on choosing scripting engines/implementations, hence why I am asking. A different question for this could also be if alternatives to V8 where considered I guess as I believe there are quite a few pure WASM engine implementations out there (unless I did not get some feature that requires you to use V8 and rules out pure WASM engines).
- agallego 6y ago1) multi language support. Ppl love go and rust. So some way to open it up to that community. 2) I like Lua a lot. Such a good tool. It just didn't have the target compatibilities of wasm + the security guarantees of wasm Afaik, there are just a couple vms with superb performance. V8 and anotherone targeting x86 only. We want to support the AWS graviton instances, so only one choice
- bluestreak 6y agoare there plans to support Kafak connector API, specifically PostgreSQL?
- sorenbs 6y agoWhat would this do? What is your use case?
- marco_craveiro 6y agoI second this request, I'd be very interested in any articles that talk about PostgreSQL integration with Kafka. Last I heard on this topic was debezium [1] but I quite liked the simplicity of Bottled Water [2]. [1] https://debezium.io/documentation/reference/connectors/postgresql.html https://debezium.io/documentation/reference/connectors/postg... [2] https://github.com/confluentinc/bottledwater-pg https://github.com/confluentinc/bottledwater-pg
- bluestreak 6y agothis would persist kafka traffic to database to make it available long term and at high speed :)
- agallego 6y agonice thinking! we should test. i don't see any api incompatibility there.
- agallego 6y agoMost kafka connectors just use the kafka API. So should "just work" (tm). Hehe. Let me know otherwise
- ckdarby 6y agoConflict of interest: I'm happily using Pulsar, I come from an extensive Kafka system, and would like to see Pulsar win this entire space personally. I see some differences, instead of Pulsar functions RedPanda has gone the extra step of using WASM, suspect the Pulsar community will end up going this direction as the whole community begins this push forward. Got rid of Zookeeper, I've never truly understood the hatred toward Zookeeper aside from the viewpoint of it being one more external dependency the project requires. Compatible Kafka API, this is a smart business choice to grab up any business using Kafka that is unhappy with the operational costs of Kafka and want to move off. Pulsar has a connector for Kafka which lets a business leave their existing work entirely untouched and stream to the new source taking the strangler pattern approach. The problem with compatible API is still the fact you need to touch the running system to point over from Kafka to RedPanda and then it opens the cans of worms on how to handle aborting RedPanda roll out and switching back to Kafka without losing data. Business now needs to modify all their existing code where Kafka is producing to also now write to RedPanda. The other option I see is just the same which is RedPanda has a connector to Kafka and only streams off the existing Kafka which kind of makes the API compatible IMO pointless aside from marketing & sales standpoint with customers.
- voctor 6y ago> aside from the viewpoint of it being one more external dependency This is exactly the point. One more dependency, one more executable. Redpanda get rid of that.
- ckdarby 6y agoKafka also has a KIP to get rid of Zookeeper and I see a bunch of the issues related to the KIP resolved and looks like it should be happening this year. Doesn't this just internalize the dependency within the project itself? Isn't Redpanda taking on all the effort that Zookeeper has been doing for years, all the edge cases, all the additional support and now the coupling of it within the very project itself?
- agallego 6y ago
- tjj5036 6y agoCongrats to Alex and team on the launch!
- agallego 6y agothank you! :)
- bithavoc 6y agoIs there a compatibility table available? Are compact topics supported?
- agallego 6y agoyup compacted topics are supported. see the spill-key-index for implementation https://github.com/vectorizedio/redpanda/blob/dev/src/v/storage/spill_key_index.h https://github.com/vectorizedio/redpanda/blob/dev/src/v/stor... if you are interested.
- pachico 6y agoAwesome! Great project and best of luck!
- agallego 6y agothanks! let us know how the testing goes :D
- fancy_pantser 6y agoIs Redpanda compatible with Faust (Python stream processing from Robinhood)? I really don't want to use Kafka, but when I must, Faust makes it straightforward. In fact, I wonder if opinionated client libraries/modules are in your interest to develop as well; they could lower the "time to implement" story for your offering. https://faust.readthedocs.io https://faust.readthedocs.io
- agallego 6y agoindeed! faust uses the regular python client. :) we try to work w/ the full ecosystem. if it doesn't it's a bug. so give it a shot and let me know. feel free to jump on slack if you want real time help too.
- chrislusf 6y agoIf messages are batched, would there be any performance advantages over Kafka/Pulsar/etc with thread-per-core architecture? The context switching cost would be amortized?
- agallego 6y agomessage batching and locking can be thought of as orthogonal pieces. In practice, yes there are advantages still of using thread per core, but perhaps not for the reasons you think. It has to do with core-local metadata materialization for a subset of the requests. THere is effectively essential complexity (i.e.: to replicate data, one must replicate data), but the TpC is strictly an optimization for flatter tail latencies.
- chrislusf 6y agoThanks for the insights! Are the tail latencies critical though? Messages are expected to be async and delivery latency usually are not hyper critical.
- agallego 6y agoDepends on use cases. We are working with a couple of security companies doing intrusion detection and for them, writing to disk on anomaly + notification seems to matter. Also financial services wanting to leverage open source tooling like spark or tensor flow still care about latency.
- 02020202 6y agoi wish travis jefferies would finish his jocko project. it's in go, single binary. would be pretty sweet but he gave up on the remaining 10% for some reason :(
- atombender 6y agoThis looks awesome! I've been waiting for a long time for someone to think outside the JVM, and I really hope this is a growing trend. The "big data" industry has seemingly been joined at the hip with Java ever since Hadoop came onto the scene, and the Apache community in particular has a lot of apps that are deeply unfriendly to non-Java apps. For example, you can't use Apache BookKeeper from a non-Java app. Would you say Redpanda is ready for production use?
- agallego 6y agoWe have production users today. We hope to announce our early product partnerships in the coming months. Big Corporate law is incredibly slow.... you have no idea. haha.
- manhcuongbk56 6y agoCan i use kafka java client with it ?