agallego

5 ms·

1.
by agallego 10mo ago
Hmm. Strange. DM me details. Haven’t heard of anything like that.
2.
by agallego 2y ago
What we found with RPCN (redpanda connect)/old benthos is that most systems are very slow and only cpu intensive things require manual CPU instruction optimizations like the snowflake connector we wrote ( https://docs.redpand
3.
by agallego 2y ago
It is not FUD. It is deterministic. Reproducible on your laptop. Out of all the banks I work with only a handful of use cases use rf=5. Defaults matter, because most people do not change them.
4.
by agallego 2y ago
coo cool right on.
5.
by agallego 2y ago
Redpanda cloud doesn’t limit tput. Most ppl get a bigger discount at high volumes. We have customers in 10s of GB/s. Confluent has those volumes too.
6.
by agallego 2y ago
incorrect. the intend is to have it be a project that is thriving, see the last 2 additional partnerships that landed as apach2 connectors: https://redpanda.com/blog/redpanda-connect w/ peerdb, and ockam.
7.
by agallego 2y ago
we trippled the team. added 3 meaningful connectors for CDC and zero-trust as well multi-lang SDK and kept 99% of the connectors available for ppl to make money on... as well as the core engine remaining MIT. This is about them not wanting
8.
by agallego 2y ago
you may have not read the blog post i wrote. the engine remains MIT because we had customers that had embedded this in their app and it made sense to keep that. it is 100% about not having to call it "redpanda x" https:/&#x2
9.
by agallego 2y ago
let's call it what it is. warp never reached out. they do not want to have the name "redpanda" in their UI. that's all. They can* make money on 223 out of 225 connectors. More over the engine* remains MIT.
10.
by agallego 3y ago
Hey jannesan this is untrue. We pay for ZoomInfo which gives us emails on search engines results. No one is manually filtering on your GitHub. But it should have a link to unsubscribe. Let me know if it doesn’t work and happy to remove. We
11.
by agallego 3y ago
Oh that’s cool. How do plugins/add ons work with clickhouse
12.
by agallego 3y ago
500MB for such a complete product is tiny! The largest CDN in the world ships 10GB+ binaries. 1G is common for large code bases if you link things statically. The bloat tends to come from transitive dependencies, most direct code is small i
13.
by agallego 3y ago
Curious as to how the code base evolved after forking from clickhouse
14.
by agallego 3y ago
that's a littlebit of a stretch. when you say "no shortage" - outside of redpanda what product exists that actually compete in all deployment modes? it's a misconception that redpanda is simply a better kafka. the way to
15.
by agallego 3y ago
def. that should be the case, we have iot companies pushing us on a single pthread a a few megs of ram.
16.
by agallego 3y ago
this seems truthy but isn't in practice. a lot of work, my perf optimization team @ redpanda does (yes we have a full team chasing tail latencies) is spent on CPU optimization, debouncing, amortizing costs, metadata lookups, hash table
17.
by agallego 3y ago
That benchmark was comparing apples to oranges. Redpanda fsynced to disk and kafka saved to memory with deferred writes. Here is a response https://redpanda.com/blog/why-fsync-is-needed-for-data-safet...
18.
by agallego 3y ago
this is a burner account created at the time this post was up.
19.
by agallego 3y ago
what makes you think that we haven't tested this? seastar's io engine is defaulted to io-uring... there are about 10 things here to comment on. on optimized kernels we disable block coalescing at the kernel level, second we tell t
20.
by agallego 3y ago
exactly. you improve latency and throughput at the same time. is kinda cool. by delaying the reponses just a small bit you get huge benefits
21.
by agallego 3y ago
it tends to be true. we do things at the application tier to minimize these things. for example, assume you have to write A, B, C ... in syscalls it would be write( A ), flush() write( B ), flush() write( C ), flush() so if you add a deboun
22.
by agallego 3y ago
the blog posts mentions that it is for any protocol that is non bft
23.
by agallego 3y ago
That’s right. Raft uses sync writes which is what redpanda uses. Complexity and industrial level reference impls are crucial
24.
by agallego 3y ago
Right. But that line of thinking gets you to a place where one is like “how does anything work” haha. In general this was a response to Confluent attempting to dismiss fsync() as a neat trick rather than an actual safety problem and why whe
25.
by agallego 3y ago
the opposite should be true tho. opt-in for unsafe. you are the minority if you read the docs, let's be real :) most ppl never read the full docs. of the ppl i chat w/ is more like 5%
26.
by agallego 3y ago
Following up - https://redpanda.com/blog/why-fsync-is-needed-for-data-safet... Try this on your laptop see global data loss - hint: multi-az is not enough Regardless of the replication mechanism you must fsync() your
27.
by agallego 3y ago
totally. we built a new team focused on the dev experience of k8s alone. 90seconds to prod (on a working eks cluster) with TLS, external certs, etc. That's the benchmark we're trying to hit :)
28.
by agallego 3y ago
no, because it is built into the raft protocol itself. with Acks=-1 we only acknowledge to the producer once data has 1. writen to majority 2. majority has done an fsync() i can see in the future giving people opt-out options here tho.
29.
by agallego 3y ago
repeating things does not make them true. I read the post. You can only control some failures, but happy for us to write our thoughts in blog form.
30.
by agallego 3y ago
our real storage is s3 - local disk is for staging/raft layer. how is that not cloud native. if you are referring to cloud native as k8s it is true that our k8s operator was built mostly for our cloud but we released it... the good new

More ›