11 ms·

I'm Alex, the founder. Happy to answer any questions.

sorenbs 6y ago
It sounds like Vectorized is a streaming platform built on top of Redpanda. Is that the right way to think about it? If yes, then why not build it directly on top of Kafka instead?
- agallego 6y ago
  We kind of started on a different note. We wanted to solve some fundamental problems (no Zookeeper - similar to KIP-500) and most of all we wanted a single binary we can ship around, something that is easy to run. I think people LOVE the kafka _api_ but they have a hard time operating clusters at scale. So we decided to keep the same API but solve the problem of operational complexity.
  - MrBuddyCasino 6y ago
    > I think people LOVE the kafka _api_ but they have a hard time operating clusters at scale. That is very true, and this is what you should emphasise. Wish you the best of luck! The actual speed of Kafka has rarely been a concern (but huge numbers of partitions are, which makes rebalancing a pain) in my experience, in fact it was mostly overkill. But operational complexity was definitely an issue!
    - agallego 6y ago
      Will do. Thank you. Do you have any thoughts here. How would you say it differently. We tried that on the landing page
      - MrBuddyCasino 6y ago
        Landing page isn't bad, when you scroll the important points are there, but you have scroll. I'm not a "landing page optimisation guru", so take this with a grain of salt, but I would change it as follows. Without scrolling, all the text thats displayed is this: "Redpanda A Kafka® API compatible streaming platform for mission-critical workloads. Try Redpanda today" It tells me what it is, which is good. It does not tell me why it is better, for that I have to scroll. Kafka is already suited for mission-critical workloads, so that is not a unique value proposition. Maybe: "Redpanda - 100% Kafka API compatible, but without the headaches. Forget Zookeeper, forget rebalancing issues. Instead, enjoy reliable message delivery, 10x faster speed and ultra-low latencies due to our thread-per-core architecture." Something like that, plus a visible "call to action" button, maybe "try it out" or "download". Could also think about a pretty graph comparing latencies or smth. People love pretty graphs.
        agallego 6y ago
        Sweet thanks! passing this to the designer.
        agallego 6y ago
        @MrBuddyCasino - I tried incorporating the design on the site we launched 5 seconds ago. lmk what you think (alex @ vectorized.io ) is my email in case you feel inclined :D thank you tho.
AlejandroM_E 6y ago
Hey Alex, Alejandro here :wave: I was a bit surprised to see the announcement about making Redpanda open source (although very interesting for poking around!). Question: are you, as a company (Vectorized) pursuing the same business model as the Confluent one? OSS Kafka, but paid official support, cloud, connectors and registry?
- agallego 6y ago
  We had success with very large customers (enterprise) and also some success with finance company (looking for 0 data loss). The idea is that we can give the kafka API compat system with 10x lower tail latencies for free and still monetize the high end of the market. I think there is a big shift when you have a single binary. i.e.: no one really complains from running ngix, etc because it's easy to get up and running. so the gist we wanted to let everyone use it and reserve the right to be the only hosted redpanda provider.
  - agallego 6y ago
    also i'd note one more thing which is the WASM API is going to give us an opportunity to add more enterprise value. i.e.: think GDPR compliance as long as your data streams are in JSON or some predefined format. WASM basically allows us to push computational guarantees to the storage engine itself without a separate cluster.
    - cookiesboxcar 6y ago
      what does WASM mean in this context?
      - sorenbs 6y ago
        https://webassembly.org/ https://webassembly.org/
        cookiesboxcar 6y ago
tomp 6y ago
Hi Alex, I'm an amateur programming language enthusiast & designer, so I've a question related to this. Looks like the project is mostly written in C++ and Go. What was the reason for this choice? Have you considered other languages, like Rust, Zig or similar instead of C++? TBH not sure what an alternative to Go would be, maybe JVM AOT-compiled with NativeImage (but AFAIK that's still experimental). Did Go's GC and/or C++'s lack of GC help/impede the project? IMO memory management is one of the main differences between languages... the other is concurrency / memory model / undefined behavior, where JVM is significantly ahead of the rest (no undefined behavior), I'm not sure exactly where Go stands (there seems to be a memory model, but no mention of undefined behavior or lack thereof).
- agallego 6y ago
  Hi there! Sure thing. I basically was playing with an RPC framework to make my old company project fast in 2017 see github.com/smfrpc/smf I wanted to play with dpdk then. When I started this in 2019 I wanted to use a framework that was battle tested, so seastar.io fit the bill. There is also a big part of it that I had been professionally programming in c++ for many many years. Last, this evolved from a prototype on my laptop in miami to company. I think rust would he am excellent choice as well and I bet we'll write a bunch of rust in the not too distant future.
bashcoder 6y ago
Hi Alex - This looks a lot like what Scylladb did to unleash the potential of the Cassandra space, with a fully optimized C++ rewrite of an Apache project. Did you draw some inspiration from their efforts? As a satisfied Scylla convert, I'm looking forward to trying Redpanda.
- agallego 6y ago
  hi there! scylla is great :) we are built on the same framework seastar.io so we literally share code bases in that regard. Main departure is we are only API compatible. It was an explicit choice to use Raft vs ISR and to not use ZK, etc. but indeed, seastar is a really fun framework to build storage systems in. I know ceph is also doing a re-write of a subsystem in seastar for example.
- runT1ME 6y ago
  I'm a fan of scylla too, but if I could go back in time I'd have recommended waiting until mid 2019 to migrate. 'Fully optimized C++ rewrite's tend to take years to become battle tested.
  - bashcoder 6y ago
    In fact, we couldn't switch to Scylla prior to 2019 because their support of incremental counters wasn't yet complete.
    - agallego 6y ago
      totally. today if folks. need txns, they wouldn't be a good fit for us. What we found is about 90% of use cases are covered by the base api. For reference w/ all of the versioning there is something like 144 api calls you can make to kafka, most ppl use a small subset of those via high level clients. (java, python, librdkafka, etc)
  - agallego 6y ago
    indeed. that's why we continuously _empirically_ prove that we are in fact safe - https://vectorized.io/validating-consistency/ https://vectorized.io/validating-consistency/ there is no substitute to testing tho there are 2 levels here. 1) raft has a proof (and a great phd dissertation from diego), but what matters is if we actually implemented it correctly. so 2) is we need to continuously test it. Denis did a lot of similar work at CosmosDB (microsoft) and has spent his career working on consensus. Hopefully these eases some concerns.
timhigins 6y ago
Hey, looks really interesting. Would love to see benchmarks that are open source so we all can check it out in more detail. Seems like a lot of questions on here are just: how much better is the performance and why?, so maybe showing is better than telling. Best of luck!
- agallego 6y ago
  hi tim. absolutely, we exect ppl to test. public benchmarks coming in December. We'll probably just pick the open messaging benchmark that has been going around pulsar+kafka communities so folks can have a frame of reference. Though what is to me more interesting, is what happens when you inject failures while the benchmarks are running.
dilyevsky 6y ago
Hey Alex, this looks very interesting! From browsing the docs it appears rpk tries to mess with cgroups and some kernel settings... is there compatibility mode for running unprivileged in a container environment such as kubernetes? Thanks!
- benpope 6y ago
  The tuning isn't 100% necessary, it's there primarily for running max performance on metal, and you can pick and choose what to tune. In general it probably best to run with: --smp <n> --overprovisioned for a container with <n> CPUs. These are standard Seastar flags.
  - agallego 6y ago
    basically what ben said. we also have a container image `vectorized/redpanda:latest`
patrec 6y ago
What's end-to-end latency for a local cluster like? What's your business model? I'm a big fan of kafka as an abstract building block, but not so much the actual implementation, which is as painful to setup as a consultancy-based business model might make you suspect it would be, especially if you need reliability. The other problem is that performance kind of sucks, apart from potential latency spikes due to GC pauses I found even the average latencies for reliable end to end (in a fast local network and on decent sized hardware) not in right order of magnitude ballpark.
- Lachedup 6y ago
  Regarding biz model. -Redpanda (free) - comparable features to Kafka, no limits -Redpanda Enterprise (paid) - Additional features (security, WASM, tiered storage, support etc) -Vectorized cloud (Free and paid tiers) - Hosted in AWS+ GCP
rhodin 6y ago
What do you mean with "Resilient to Scan-The-World"?
- ryanworl 6y ago
  I'm not Alex, but I would assume he means scanning cold data will not evict all the hot data from the cache. This is a common problem with cache eviction algorithms.
  - agallego 6y ago
    indeed.
polskibus 6y ago
Hi, I need to decide in the near future on a scalable messaging solution with at-least-once or better guarantees for our own SaaS platform. I'm looking for something that can be deployed on-prem, that's not targeted only for the public cloud. Do you have any documentation on how best to deploy redpanda with Docker and/or Ansible? What are the best practices on rolling out the cluster in-house? I've seen some docs on tuning that referred to AWS types of instances, is there something that you could refer me to that is more generic and not tied to a specific cloud vendor? I'm asking partly because we must be able to offer closed on-prem installations as well as SaaS on a cloud. I'm looking for a low-ops component that will not fail me (as often as alternatives would) :)
- Lachedup 6y ago
  we are happy to help and answer any questions. Join our community slack channel or hit the contact us button on vectorized.io
manhcuongbk56 6y ago
Hi Alex, i don't see any client document. Can i use kafka java client with it?
- benpope 6y ago
  This should work fine. If you have any issues, please report as a bug.
eis 6y ago
Hey, congrats on publishing your source. It's a very interesting project indeed. I took a look around and I think with a bit expanded documentation, especially examples using the WASM transformations and maybe a some emphasis on durability and other guarantees it could grow into a great project. I'd be interested in the write amplification since you went pretty low level in your IO layer. How do you guarantee atomic writes when virtually no disk provides guarantees other than on a page level which could result in destroying already written data if a write to the same page fails - at least in theory - and so one has to resort to writing data multiple times.

It sounds like Vectorized is a streaming platform built on top of Redpanda. Is that the right way to think about it? If yes, then why not build it directly on top of Kafka instead?

agallego 6y ago
We kind of started on a different note. We wanted to solve some fundamental problems (no Zookeeper - similar to KIP-500) and most of all we wanted a single binary we can ship around, something that is easy to run. I think people LOVE the kafka _api_ but they have a hard time operating clusters at scale. So we decided to keep the same API but solve the problem of operational complexity.
- MrBuddyCasino 6y ago
  > I think people LOVE the kafka _api_ but they have a hard time operating clusters at scale. That is very true, and this is what you should emphasise. Wish you the best of luck! The actual speed of Kafka has rarely been a concern (but huge numbers of partitions are, which makes rebalancing a pain) in my experience, in fact it was mostly overkill. But operational complexity was definitely an issue!
  - agallego 6y ago
    Will do. Thank you. Do you have any thoughts here. How would you say it differently. We tried that on the landing page
    - MrBuddyCasino 6y ago
      Landing page isn't bad, when you scroll the important points are there, but you have scroll. I'm not a "landing page optimisation guru", so take this with a grain of salt, but I would change it as follows. Without scrolling, all the text thats displayed is this: "Redpanda A Kafka® API compatible streaming platform for mission-critical workloads. Try Redpanda today" It tells me what it is, which is good. It does not tell me why it is better, for that I have to scroll. Kafka is already suited for mission-critical workloads, so that is not a unique value proposition. Maybe: "Redpanda - 100% Kafka API compatible, but without the headaches. Forget Zookeeper, forget rebalancing issues. Instead, enjoy reliable message delivery, 10x faster speed and ultra-low latencies due to our thread-per-core architecture." Something like that, plus a visible "call to action" button, maybe "try it out" or "download". Could also think about a pretty graph comparing latencies or smth. People love pretty graphs.
      - agallego 6y ago
        Sweet thanks! passing this to the designer.
        agallego 6y ago
        @MrBuddyCasino - I tried incorporating the design on the site we launched 5 seconds ago. lmk what you think (alex @ vectorized.io ) is my email in case you feel inclined :D thank you tho.

AlejandroM_E 6y ago

Hey Alex, Alejandro here :wave: I was a bit surprised to see the announcement about making Redpanda open source (although very interesting for poking around!). Question: are you, as a company (Vectorized) pursuing the same business model as the Confluent one? OSS Kafka, but paid official support, cloud, connectors and registry?

agallego 6y ago
We had success with very large customers (enterprise) and also some success with finance company (looking for 0 data loss). The idea is that we can give the kafka API compat system with 10x lower tail latencies for free and still monetize the high end of the market. I think there is a big shift when you have a single binary. i.e.: no one really complains from running ngix, etc because it's easy to get up and running. so the gist we wanted to let everyone use it and reserve the right to be the only hosted redpanda provider.
- agallego 6y ago
  also i'd note one more thing which is the WASM API is going to give us an opportunity to add more enterprise value. i.e.: think GDPR compliance as long as your data streams are in JSON or some predefined format. WASM basically allows us to push computational guarantees to the storage engine itself without a separate cluster.
  - cookiesboxcar 6y ago
    what does WASM mean in this context?
    - sorenbs 6y ago
      https://webassembly.org/ https://webassembly.org/
      - cookiesboxcar 6y ago

tomp 6y ago

Hi Alex, I'm an amateur programming language enthusiast & designer, so I've a question related to this. Looks like the project is mostly written in C++ and Go. What was the reason for this choice? Have you considered other languages, like Rust, Zig or similar instead of C++? TBH not sure what an alternative to Go would be, maybe JVM AOT-compiled with NativeImage (but AFAIK that's still experimental). Did Go's GC and/or C++'s lack of GC help/impede the project? IMO memory management is one of the main differences between languages... the other is concurrency / memory model / undefined behavior, where JVM is significantly ahead of the rest (no undefined behavior), I'm not sure exactly where Go stands (there seems to be a memory model, but no mention of undefined behavior or lack thereof).

agallego 6y ago
Hi there! Sure thing. I basically was playing with an RPC framework to make my old company project fast in 2017 see github.com/smfrpc/smf I wanted to play with dpdk then. When I started this in 2019 I wanted to use a framework that was battle tested, so seastar.io fit the bill. There is also a big part of it that I had been professionally programming in c++ for many many years. Last, this evolved from a prototype on my laptop in miami to company. I think rust would he am excellent choice as well and I bet we'll write a bunch of rust in the not too distant future.

bashcoder 6y ago

Hi Alex - This looks a lot like what Scylladb did to unleash the potential of the Cassandra space, with a fully optimized C++ rewrite of an Apache project. Did you draw some inspiration from their efforts? As a satisfied Scylla convert, I'm looking forward to trying Redpanda.

agallego 6y ago
hi there! scylla is great :) we are built on the same framework seastar.io so we literally share code bases in that regard. Main departure is we are only API compatible. It was an explicit choice to use Raft vs ISR and to not use ZK, etc. but indeed, seastar is a really fun framework to build storage systems in. I know ceph is also doing a re-write of a subsystem in seastar for example.
runT1ME 6y ago
I'm a fan of scylla too, but if I could go back in time I'd have recommended waiting until mid 2019 to migrate. 'Fully optimized C++ rewrite's tend to take years to become battle tested.
- bashcoder 6y ago
  In fact, we couldn't switch to Scylla prior to 2019 because their support of incremental counters wasn't yet complete.
  - agallego 6y ago
    totally. today if folks. need txns, they wouldn't be a good fit for us. What we found is about 90% of use cases are covered by the base api. For reference w/ all of the versioning there is something like 144 api calls you can make to kafka, most ppl use a small subset of those via high level clients. (java, python, librdkafka, etc)
- agallego 6y ago
  indeed. that's why we continuously _empirically_ prove that we are in fact safe - https://vectorized.io/validating-consistency/ https://vectorized.io/validating-consistency/ there is no substitute to testing tho there are 2 levels here. 1) raft has a proof (and a great phd dissertation from diego), but what matters is if we actually implemented it correctly. so 2) is we need to continuously test it. Denis did a lot of similar work at CosmosDB (microsoft) and has spent his career working on consensus. Hopefully these eases some concerns.

timhigins 6y ago

Hey, looks really interesting. Would love to see benchmarks that are open source so we all can check it out in more detail. Seems like a lot of questions on here are just: how much better is the performance and why?, so maybe showing is better than telling. Best of luck!

agallego 6y ago
hi tim. absolutely, we exect ppl to test. public benchmarks coming in December. We'll probably just pick the open messaging benchmark that has been going around pulsar+kafka communities so folks can have a frame of reference. Though what is to me more interesting, is what happens when you inject failures while the benchmarks are running.

dilyevsky 6y ago

Hey Alex, this looks very interesting! From browsing the docs it appears rpk tries to mess with cgroups and some kernel settings... is there compatibility mode for running unprivileged in a container environment such as kubernetes? Thanks!

benpope 6y ago
The tuning isn't 100% necessary, it's there primarily for running max performance on metal, and you can pick and choose what to tune. In general it probably best to run with: --smp <n> --overprovisioned for a container with <n> CPUs. These are standard Seastar flags.
- agallego 6y ago
  basically what ben said. we also have a container image `vectorized/redpanda:latest`

patrec 6y ago

What's end-to-end latency for a local cluster like? What's your business model? I'm a big fan of kafka as an abstract building block, but not so much the actual implementation, which is as painful to setup as a consultancy-based business model might make you suspect it would be, especially if you need reliability. The other problem is that performance kind of sucks, apart from potential latency spikes due to GC pauses I found even the average latencies for reliable end to end (in a fast local network and on decent sized hardware) not in right order of magnitude ballpark.

Lachedup 6y ago
Regarding biz model. -Redpanda (free) - comparable features to Kafka, no limits -Redpanda Enterprise (paid) - Additional features (security, WASM, tiered storage, support etc) -Vectorized cloud (Free and paid tiers) - Hosted in AWS+ GCP

rhodin 6y ago

What do you mean with "Resilient to Scan-The-World"?

ryanworl 6y ago
I'm not Alex, but I would assume he means scanning cold data will not evict all the hot data from the cache. This is a common problem with cache eviction algorithms.
- agallego 6y ago
  indeed.

polskibus 6y ago

Hi, I need to decide in the near future on a scalable messaging solution with at-least-once or better guarantees for our own SaaS platform. I'm looking for something that can be deployed on-prem, that's not targeted only for the public cloud. Do you have any documentation on how best to deploy redpanda with Docker and/or Ansible? What are the best practices on rolling out the cluster in-house? I've seen some docs on tuning that referred to AWS types of instances, is there something that you could refer me to that is more generic and not tied to a specific cloud vendor? I'm asking partly because we must be able to offer closed on-prem installations as well as SaaS on a cloud. I'm looking for a low-ops component that will not fail me (as often as alternatives would) :)

Lachedup 6y ago
we are happy to help and answer any questions. Join our community slack channel or hit the contact us button on vectorized.io

manhcuongbk56 6y ago

Hi Alex, i don't see any client document. Can i use kafka java client with it?

benpope 6y ago
This should work fine. If you have any issues, please report as a bug.

eis 6y ago

Hey, congrats on publishing your source. It's a very interesting project indeed. I took a look around and I think with a bit expanded documentation, especially examples using the WASM transformations and maybe a some emphasis on durability and other guarantees it could grow into a great project. I'd be interested in the write amplification since you went pretty low level in your IO layer. How do you guarantee atomic writes when virtually no disk provides guarantees other than on a page level which could result in destroying already written data if a write to the same page fails - at least in theory - and so one has to resort to writing data multiple times.