richieartoul

12 ms·

31.
by richieartoul 3y ago
I know the link stresses that correctness and code clarity are the primary goals of this project, but I’m curious if performance is a goal as well? Do you hope that people will run “real” workloads with this or use it to embed other softwar
32.
by richieartoul 3y ago
This is so cool. Can’t wait to go digging around through the code this weekend. Good luck with the project!
33.
by richieartoul 3y ago
I’ve built pretty much my entire career around this problem and it still feels evergreen. If you want a meaningful and rich career as a software engineer, distributed storage is a great area to focus on.
34.
by richieartoul 3y ago
You usually have to redesign the storage from the ground up around S3. Some databases do transparent data tiering with S3 which can work ok too depending on the use case.
35.
by richieartoul 3y ago
(WarpStream Cofounder) https://docs.WarpStream.com is the best document we have right now. This sounds interesting though, can you jump in our slack or shoot me an email at founders@warpstreamlabs.com ? Happy to support you any
36.
by richieartoul 3y ago
It’s getting late and I’m not 100% confident I’m sure what you’re asking, but I believe the answer to your question is yes. When you produce a message/batch you get back offsets for each message you produced. If you produce again afte
37.
by richieartoul 3y ago
(WarpStream co-founder) Yeah we’ve run into a number of people who’ve rolled their own solution in this space. The “push pointers to S3 through traditional Kafka” approach is a very practical one. Was this memq at Pinterest, or something el
38.
by richieartoul 3y ago
Our kafka protocol support is still incomplete. We’ve documented our progress here: https://docs.warpstream.com/warpstream/reference/kafka-proto... That said we’d be stoked to get this working as an additional too
39.
by richieartoul 3y ago
DMd you on twitter
40.
by richieartoul 3y ago
(WarpStream Cofounder) FWIW we’re considering a version where you can host the metadata yourself for enterprise users. For the free tier though we didn’t think it made sense since for a workload that could fit into our free tier, it didn’t
41.
by richieartoul 3y ago
(WarpStream cofounder) Yeah you have to produce in parallel and use batching, but it works well in practice. We’ve tested it up to 1GiB/s in throughout without issue
42.
by richieartoul 3y ago
(WarpStream co-founder) I know that is super frustrating, I’m sorry. We’re just really early and unsure how to begin even discussing public pricing. We are 100% committed to that free tier. We designed our entire architecture around makin
43.
by richieartoul 3y ago
Replying here instead of below because we hit depth limit. WarpStream definitely isn’t magical, it makes a very real trade off around latency. On the read side, the architecture is such that you’ll have to pay for 1 GET request for every 4
44.
by richieartoul 3y ago
Will make this possible soon, oversight on our part that the agent doesn't expose a way to let you do that right now
45.
by richieartoul 3y ago
Its stateless from the customer's perspective since they don't have to manage the consensus layer (which is only possible because we separated data from metadata). Obviously there is state somewhere since the cluster is storing da
46.
by richieartoul 3y ago
(WarpStream co-founder) FWIW we subject WarpStream to continuous chaos/fault injection in our integration tests and staging environment to verify correctness and liveness properties. I wouldn't say they're far from our mind,
47.
by richieartoul 3y ago
(WarpStream co-founder) None currently, if you need latency that low WarpStream is not a good fit and you should probably stick with something more traditional like a very well tuned Kafka cluster.
48.
by richieartoul 3y ago
(WarpStream co-founder) Yeah if you can fit your workload into it, it's pretty solid from an operational perspective since AWS manages the whole thing (except scaling up shards and stuff like that). It gets pretty pricey though for lar
49.
by richieartoul 3y ago
(WarpStream co-founder) We're thinking about making the `warpstream playground` command run an in-memory local version of control plane. Come chat with us in our Slack or Discord if you want to discuss more!
50.
by richieartoul 3y ago
(WarpStream co-founder) Check the end of our blog post, we brought receipts :)
51.
by richieartoul 3y ago
(WarpStream co-founder) I'm not sure what you mean. Message persistence is a fundamental feature of Kafka that almost everyone using it relies on, its not some esoteric feature no one uses. We're each coming from our own network b
52.
by richieartoul 3y ago
1. By payload size do you mean record size? They're ~1KiB 2. Flush interval was 100ms, the agent defaults to 250ms though I believe.
53.
by richieartoul 3y ago
(WarpStream co-founder) That does not work currently because there is no way to configure the agent with a Minio URL. That's just an oversight on our part though, I'll get that working within a few days.
54.
by richieartoul 3y ago
(WarpStream co-founder) The cache is for reads, not writes. There is no cache for writes. We built our own because it needed to behave in a very specific way to meet our cost/latency goals. Running a MinIO sidecar instance means that e
55.
by richieartoul 3y ago
(WarpStream co-founder) It requires some form of object storage (S3, GCS, Azure blob storage, etc). You could use minio if you want to host the storage yourself, although I think I need to make a few changes to make that configurable in the
56.
by richieartoul 3y ago
(WarpStream co-founder) That's fair, that statement is probably significantly colored by my own personal network / work experience.
57.
by richieartoul 3y ago
(WarpStream co-founder) Cheers! We really want to drive the incremental cost (both in terms of $ and management overhead) per GiB down as low as possible so people can start using these systems for more use-cases.
58.
by richieartoul 3y ago
(WarpStream co-founder) Nope, we provide the same durability and ordering messages as Kafka, albeit with higher latency. We never acknowledge produce requests until they've been durably persisted in object storage (P99 ~400ms), so its
59.
by richieartoul 3y ago
(WarpStream co-founder) Thanks for the kind words! I'll just add that tiered storage is not quite the same thing because it means you still have to manage local disks and replication carefully, however briefly. Please reach out with
60.
by richieartoul 3y ago
(WarpStream co-founder) WarpStream's current offering is a hybrid BYOC approach. The customer runs the agents in their cloud account, and we manage the metadata store for them remotely. This keeps all the customer data in their cloud a

More ›