agallego

10 ms·

61.
by agallego 6y ago
that's correct - we actually don't manually write the kafka api, but we codegenerate it: https://github.com/vectorizedio/redpanda/blob/dev/src/v/kafk... the tl;dr is that we wrote a p
62.
by agallego 6y ago
Cool. If you want help hacking in it there is a community slack at vectorized.io/slack or github discussions work too
63.
by agallego 6y ago
Hi there! Don't be disappointed :) perf benchmarks coming. The article was about architecture. As a new project we have a lot of content to put out... Stay tuned. Very likely early next month.
64.
by agallego 6y ago
This is a hard question because everything is really dependent on your threading model. One has to start w/ the threading model. At the time I wrote the first line of code in Jan 2019, there wasn't anything that was really amenabl
65.
by agallego 6y ago
The write is a 2 stage function which works quite well for the kafka layer since it's all batch. Let me explain. 1. First, we have a core-local shared chunk cache w/ DMA aligned buffers. https://github.com/vectoriz
66.
by agallego 6y ago
agreed w/ andrew. rocksdb is pretty heavy. for streaming logs, something much much simpler yields significant performance improvements specially when tied to the IO+CPU priority scheduling.
67.
by agallego 6y ago
on seastar, you yield a lot. so loops go from for( ... : collection) {} to return seastar::do_for_each(collection, callback);
68.
by agallego 6y ago
Oh shoot! thank you... fixing the link give me 5 mins. So currently the partition allocator - https://github.com/vectorizedio/redpanda/blob/dev/src/v/clus... - is primitive. But we have a worki
69.
by agallego 6y ago
so redpanda partitions 'raft' groups per kafka partition. so in the `topic/partition` model every partition is it's own raft group (similar to multi raft in cockroachdb). So it is in fact even more important due to the r
70.
by agallego 6y ago
indeed. though, the recent zstd changes w/ different levels of compression sort of close the gap in perf that lz4 had over zstd. (if interested in this kind of detail for a new streaming storage engiene, i gave a talk last week at the
71.
by agallego 6y ago
Kafka has `zstd` encoding. Here is our version of the streaming decoder i wrote a while ago https://github.com/vectorizedio/redpanda/blob/dev/src/v/comp... that's our default for our int
72.
by agallego 6y ago
So depending on the disks, we found that different IO sizes will yield optimal settings for saturating disks. As an anecdote in clouds, the IOPS is the key principle and often you can drive higher throughput depending on your DMA block size
73.
by agallego 6y ago
Indeed. I think you have different saturation points the wider the use cases you hit. One example w/ a single-core (which btw, agreed whole heartedly for io) is checksumming + decoding. For kafka, we have multiple indexes - a time inde
74.
by agallego 6y ago
@MrBuddyCasino - I tried incorporating the design on the site we launched 5 seconds ago. lmk what you think (alex @ vectorized.io ) is my email in case you feel inclined :D thank you tho.
75.
by agallego 6y ago
basically what ben said. we also have a container image `vectorized/redpanda:latest`
76.
by agallego 6y ago
We have production users today. We hope to announce our early product partnerships in the coming months. Big Corporate law is incredibly slow.... you have no idea. haha.
77.
by agallego 6y ago
we have exactly 1 replication protocol. That's raft. We spend a ton of time ensuring it's correctness 1. https://vectorized.io/validating-consistency/ 2. https://vectorized.io/kafka-redpanda-a
78.
by agallego 6y ago
indeed.
79.
by agallego 6y ago
our acks are similar to kafka in that regard. acks=0,1 leader acknowledgement. acks=-1 we go further like denis mentioned above by 'flushing' to disk before responding which is stronger (log completeness guarantee from raft) than
80.
by agallego 6y ago
nice thinking! we should test. i don't see any api incompatibility there.
81.
by agallego 6y ago
Depends on use cases. We are working with a couple of security companies doing intrusion detection and for them, writing to disk on anomaly + notification seems to matter. Also financial services wanting to leverage open source tooling like
82.
by agallego 6y ago
1) multi language support. Ppl love go and rust. So some way to open it up to that community. 2) I like Lua a lot. Such a good tool. It just didn't have the target compatibilities of wasm + the security guarantees of wasm Afaik, there
83.
by agallego 6y ago
message batching and locking can be thought of as orthogonal pieces. In practice, yes there are advantages still of using thread per core, but perhaps not for the reasons you think. It has to do with core-local metadata materialization for
84.
by agallego 6y ago
thanks! let us know how the testing goes :D
85.
by agallego 6y ago
totally. today if folks. need txns, they wouldn't be a good fit for us. What we found is about 90% of use cases are covered by the base api. For reference w/ all of the versioning there is something like 144 api calls you can mak
86.
by agallego 6y ago
indeed. that's why we continuously _empirically_ prove that we are in fact safe - https://vectorized.io/validating-consistency/ there is no substitute to testing tho there are 2 levels here. 1) raft has a proof (a
87.
by agallego 6y ago
hi tim. absolutely, we exect ppl to test. public benchmarks coming in December. We'll probably just pick the open messaging benchmark that has been going around pulsar+kafka communities so folks can have a frame of reference. Though wh
88.
by agallego 6y ago
yup compacted topics are supported. see the spill-key-index for implementation https://github.com/vectorizedio/redpanda/blob/dev/src/v/stor... if you are interested.
89.
by agallego 6y ago
hi there! scylla is great :) we are built on the same framework seastar.io so we literally share code bases in that regard. Main departure is we are only API compatible. It was an explicit choice to use Raft vs ISR and to not use ZK, etc. b
90.
by agallego 6y ago
indeed! faust uses the regular python client. :) we try to work w/ the full ecosystem. if it doesn't it's a bug. so give it a shot and let me know. feel free to jump on slack if you want real time help too.

More ›