richieartoul

9 ms·

61.
by richieartoul 3y ago
(WarpStream co-founder) We use S3 for data replication, availability, and durability, but not for the log ordering guarantees. For that, we have our own custom metadata store that "imposes" ordering guarantees on top of the data t
62.
by richieartoul 3y ago
(WarpStream founder) It guarantees correct ordering inside a partition for all acknowledged messages regardless of which batch they originated from. We don't synchronize clocks, the agents call out to our cloud metadata store which run
63.
by richieartoul 3y ago
(WarpStream co-founder) Yeah our P99 for producer latency is ~400ms right now.
64.
by richieartoul 3y ago
(WarpStream founder) 1. Yeah, we mention at the end of the post the P99 produce latency is ~400ms. 2. MSK still charges you for networking to produce into the cluster and consumer out of it if follower fetch is not properly configured. Also
65.
by richieartoul 3y ago
(WarpStream founder) Exactly, and we designed WarpStream's storage system to minimize the number of object storage API calls it has to make while still maintaining relatively low latency (P99 of ~1s end-to-end)
66.
by richieartoul 3y ago
(WarpStream founder) No WarpStream has partitions internally and provides the same ordering guarantees Kafka does at the partition level. We're just saying that we think for most streaming applications this is not a great programming m
67.
by richieartoul 3y ago
(WarpStream founder) WarpStream has a completely different architecture than Kafka: https://docs.warpstream.com/warpstream/background-informatio... That said it does require a lot of metadata to orchestrate all the dif
68.
by richieartoul 3y ago
(WarpStream Founder) That's fair, but we think people shouldn't have to choose between multiple AZs and costs. All cloud providers recommend that critical workloads running in cloud environments should run in more than one AZ.
69.
by richieartoul 3y ago
(WarpStream founder here) Not currently. One of the things we're looking to do next is make it so any topic can be "automatically" turned into a standard format in S3, something like Parquet/Iceberg/Deltalake so its
70.
by richieartoul 3y ago
(WarpStream founder here) Yes exactly. The cloud metadata store assigns offsets, does compaction planning/scheduling, handles service discovery, etc
71.
Kafka is dead, long live Kafka(warpstream.com)
392 points by richieartoul 3y ago|287 comments
72.
by richieartoul 3y ago
I've been using it for a few weeks now. Overall pretty happy with it. Has good default API, and can be basically arbitrarily extended as needed. We even have a custom handler implementation that allows us to assert on specific logs bei
73.
by richieartoul 3y ago
Interzone bandwidth maybe? They used to not charge for that, but they started doing so recently
74.
by richieartoul 3y ago
Seconded, I used it in one of my side projects and it’s pretty amazing to have full WASI with no CGO
75.
by richieartoul 3y ago
I don’t think there is any officially recommended way to run SQL ontop of FoundationDB. I suspect the TiDB comment you saw was someone speculating about an idea maybe (I vaguely recall some GitHub issues about it). There are some well suppo
76.
by richieartoul 3y ago
I actually have the opposite feeling where I mostly like the Protobuf ecosystem, but find the actual GRPC implementations in the languages I’ve tried (Java/Go) extremely frustrating to use in practice, especially for anything performan
77.
by richieartoul 3y ago
I think it’s the last paragraph I’m really disagreeing with. Many many application will do just fine with a single Postgres. If that is not enough even on a big box, they can scale out to a large master system like aurora. If that is not en
78.
by richieartoul 3y ago
It’s really not just about annoyance it’s about correctness and speed of iteration. Many apps that can be built trivially in a couple SQL queries cannot be implemented correctly in Scylla without reinventing half of tidb/vitess/et
79.
by richieartoul 4y ago
Tigris( https://github.com/tigrisdata/tigris ) is one of the supported FerretDB backends and Tigris is backed by FoundationDB, so you can still have the Mongo interface with the reliability and scaling of FDB if that&#x2
80.
by richieartoul 4y ago
This is cool, I’ve been working on something similar in Go: https://github.com/richardartoul/nola but with a slightly different focus on linearizability / durability and Webassembly. Let me know if you want to cha
81.
NOLA: A library for building virtual actor systems with WASM and FoundationDB(github.com)
7 points by richieartoul 4y ago|0 comments
82.
by richieartoul 4y ago
Why shouldn’t they be proud of a massive system like Cloudflare that works extremely well? As a commentor below described, it’s not luck or happenstance, it’s a natural consequence of how BGP works. Seems pretty elegant to me.
83.
Show HN: Core Count, a tool to visualize your software’s physical footprint(richardartoul.github.io)
5 points by richieartoul 4y ago|0 comments
84.
by richieartoul 4y ago
It’s really the usage of the word primitive that I’m arguing with. Java’s GC comes with a lot of additional trade offs that Go’s doesn’t. For example, the fact that the Java GC is copying and generational means that there is a LOT more over
85.
by richieartoul 4y ago
I do a lot of “high throughput” stuff at work in both Go and Java, and the Go stuff is usually faster by default. Java tends to win for really naive programs where the author didn’t bother caring about performance or allocations at all, but
86.
by richieartoul 4y ago
You can manually configure the replication factor in FoundationDB and the client is intelligent enough to load balance across them. Three data hall is R.F == 3 Three datacenter is R.F == 6 You can configure it higher manually as well, we ha
87.
by richieartoul 4y ago
Isn’t that ignoring the point that the read is running concurrently with w1 and w2? Your comment about “have the effect” only applies to transactions beginning after a write has completed, but the anomaly cockroach suffers from is for a r
88.
by richieartoul 4y ago
I’m not 100% sure, but my reading of the CockroachDB Jepsen test: https://jepsen.io/analyses/cockroachdb-beta-20160829 indicates that it does meet those two requirements, but that it still is not globally strictly seri
89.
by richieartoul 4y ago
What does scalable mean in this context? In general, Cassandra is much more efficient at writes and can handle higher volumes of write throughput assuming identical hardware, but from what I’ve seen FDB is as efficient (if not more so) for
90.
by richieartoul 4y ago
Funny enough, this actually cuts to the core of why strict serializability is so important. “Technically speaking, every read-only transaction could return the empty-set (which was the initial state of the database) and not be in violation

More ›