huntaub

8 ms·

61.
by huntaub 2y ago
It's interesting, after my time at Amazon (8 years) -- I struggled to visualize decisions without a document, because you get so used to reviewing things in that way. However, it's EXTREMELY heavy-handed. People will give you comm
62.
by huntaub 2y ago
Interesting -- is that normally done with database updates + polling vs. something purpose-built?
63.
by huntaub 2y ago
What are some example use cases where having the ability for the database to push updates to an application would be helpful (vs. the traditional polling approach)?
64.
by huntaub 2y ago
You've totally hit the nail on the head. This is the real moat of S3, the fact that they have so much front-end throughput available from the gigantic buildout that folks can take advantage of without any capacity pre-planning.
65.
by huntaub 2y ago
Are you talking about getting metadata from many objects in the bucket simultaneously? You might be interested in S3 Metadata https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingM... .
66.
by huntaub 2y ago
I generally see object storage systems advertise 11 9s of availability. You would usually see a commercial distributed file system (obviously stuff like Ceph and Lustre will depend on your specific configuration) advertise less (to trade of
67.
by huntaub 2y ago
You're totally correct, but these products also need to be specifically designed against these failure cases (i.e. it's more than just MTTR + MTTF == durability). You (of course) can't just run deployments without validating
68.
by huntaub 2y ago
I think that this is, to Andy's point, basically about simplicity. It's not that your business necessarily needs 11 9s of durability for continuity purposes, but it sure is nice that you never have to think about the durabilit
69.
by huntaub 2y ago
For example, in AWS, you can get a similar FSx for Lustre file system for just 11% more cost, which could be worth it to avoid the management costs of running your own storage cluster.
70.
by huntaub 2y ago
This has actually brought up an interesting point. Kubernetes is nothing more than an API interface. Should someone be working on building a multi-tenant Kubernetes (so that customers don't need to manage nodes or clusters) which enfor
71.
by huntaub 2y ago
Thanks for this, it's helpful. Totally heard about O_APPEND and read() returning -EACCESS. The other ones, I agree, should be fixed in later versions of the Linux kernel/NFS client.
72.
by huntaub 2y ago
What aspects of NFS do you think break half of the important guarantees of a file system?
73.
by huntaub 2y ago
This is kind of an interesting thought that more mirrors how Docker uses OverlayFS to track changes to the entire file system. No need for new file APIs.
74.
by huntaub 2y ago
I take a different view on this. IMO the tricks that existing file systems play to get more performance (specifically around ordering and atomicity) make it extra hard for developers to reason about. Obviously, you can't do anything ab
75.
by huntaub 2y ago
Ultimately, we should be able to match the performance of FSx for Lustre. With a scale-out protocol (mirroring the Lustre protocol), it's relatively simple to achieve scaleable throughput and IOPS out to the level of 100s of GiB/s
76.
by huntaub 2y ago
Here's how I would think about this. Regatta isn't the best way to add synchronization primitives to S3, if you're already using the S3 API and able to change your code. Regatta is most useful when you need a local disk, or
77.
by huntaub 2y ago
Thank you for the note. I’d recommend checking out this section of our docs [1], where we are trying to compile some of this comparison. I haven’t called out FlexFS specifically, but I’ll work on adding that soon. We’ll also get the Privacy
78.
by huntaub 2y ago
I don’t know if I agree, for example, Postgres has this [1] to say about using NFS as the backing store. I think that part of the challenge is that there are so many implementation details that differ between NFS servers and many configurat
79.
by huntaub 2y ago
That’s true, we could just release the software open source, but that doesn’t help our customers who don’t want to run and manage their own infrastructure. Our customers tell us that the value of the product comes from it being fully mana
80.
by huntaub 2y ago
It’s not as clear, but it’s certainly something we are considering. If you’d like to use us on-prem, I’d love to hear more. Can you shoot me an email with details at hleath [at] regattastorage.com?
81.
by huntaub 2y ago
Thank you for the kind words, Geert!
82.
by huntaub 2y ago
I spent several years working in the Northeast, and I developed an appreciation (but not a skill) for sailing. In some sense, I think of Regatta as high-speed sailing on top of customer data lakes.
83.
by huntaub 2y ago
Thank you so much! It’s been amazing to see what you all have built over the years, and it’s (of course) been inspirational for me.
84.
by huntaub 2y ago
Awesome! Great to meet you, so happy that so many folks in the space are here.
85.
by huntaub 2y ago
This is true, and I think that there are consumer (or at least "run on your laptop") versions of this that could make sense. However, the technology underneath would have to look very different. For example, these protocols are
86.
by huntaub 2y ago
Hey there, that's sort of the correct way to think about it -- notably that our caching layer is high-durability, so we can keep recent writes in the cache safely. External changes to the bucket are okay! Lots of customers need to (for
87.
by huntaub 2y ago
Ultimately, we're just working on a different problem space than these protocols. That's not to say that all of the existing protocols are bad, I absolutely believe that these protocols are great. Our ultimate goal, though, is to
88.
by huntaub 2y ago
We've actually been thinking about getting Jepsen to do this, so I'm happy to hear that you also think that it would inspire confidence!
89.
by huntaub 2y ago
> That is, you aren't treating the object store as the source of truth. If your caching layer goes down after a write is ack'ed but before it's "replicated" to S3, people lose their data, right? This is exactly w
90.
by huntaub 2y ago
That's correct re: the S3 API. What we do is we "merge" multiple write requests together to minimize the cost to you and the number of requests to S3. For example, if you write a file 1,000 times in the span of a minute, we w

More ›