huntaub

10 ms·

121.
by huntaub 2y ago
For the most part, I agree with this! The storage market is big enough for everyone because of all of the different needs that customers have. That said, if we are able to build something that provides EBS-like performance with pay-as-you-g
122.
by huntaub 2y ago
One thing that we’re considering for these kinds of use cases is providing the ability for users to either (a) request that their data always stays in the hot storage or (b) provide the ability for users to ask us to pre-load their data set
123.
by huntaub 2y ago
Yes, I think this is something that I’m actually struggling with. What’s the most exciting part for users? Is it the fact that we’re building a super fast file system or is it that we have this synchronization to S3? Ultimately, there just
124.
by huntaub 2y ago
I'm not totally sure what you mean. I don't think that S3 Express One Zone offers any additional atomic semantics in the file system world.
125.
by huntaub 2y ago
Great question! Full disclosure, answer copied from a another comment: It's similar to JuiceFS, but JuiceFS writes and reads data from S3 in a proprietary block format. This means that you cannot connect JuiceFS to existing data sets i
126.
by huntaub 2y ago
Trust me, I feel the same way. The problem with these things is that you end up building a company because you get so much conviction that what you're doing is the right thing for customers, and you end up shocked that this isn'
127.
by huntaub 2y ago
I totally agree that we're continuing to see a trend of applications which are designed to work directly on S3. However, like the S3 protocol, I think that the file protocol is cemented in time as something that we will be using 100 ye
128.
by huntaub 2y ago
We are looking at launching in Azure Cloud with support for Azure Blob Storage as the backend within the next 6 months. If there's a specific use case that you have, it would be helpful to share it with me at hleath [at] regattastorage
129.
by huntaub 2y ago
That's exactly right -- I like to think that we deliver on the promise of those open-source S3 adapters. We provide enterprise-grade performance.
130.
by huntaub 2y ago
This is correct, we encrypt data in-transit to the Regatta servers (using TLS), and we encrypt any data that the Regatta servers are storing. Of course, when Regatta communicates with S3, that's also encrypted with TLS (just like using
131.
by huntaub 2y ago
Let's hope so, I'd love to help teams take storage infrastructure management off of their plate! If you're in the public sector and interested in trying out Regatta, please shoot me an email at hleath [at] regattastorage.com.
132.
by huntaub 2y ago
Yeah, I like to think of it in a similar vein. We want to empower people to create stateless workflows where they may have previously needed to think about state management. Today, Regatta is an NFS file system where the cache lives on our
133.
by huntaub 2y ago
We are finding a lot of success in the ML Ops space for exactly this reason. I also completely agree that enterprise customers want to keep their data where they can govern and audit it (often in S3). We're excited about the possibilit
134.
by huntaub 2y ago
Sorry about that! It's on our list to fix once we're done responding to comments.
135.
by huntaub 2y ago
Today, we only charge for cache usage (storage) and data transfer between Regatta and S3. If your metadata access doesn't require transfer to S3, then it doesn't cost anything! However, renames do require transfer to S3 (because
136.
by huntaub 2y ago
Hey there, thanks for the concern. There are a spectrum of teams out there. Some teams are totally comfortable building something like this and running their own storage infrastructure. Other teams want a fully managed solution to handle st
137.
by huntaub 2y ago
I agree, I plan to put up a table soon.
138.
by huntaub 2y ago
Huh, that's interesting. I wouldn't imagine that there were security problems specific to FUSE compared to any other software that you would run on your servers. Regardless, I see FUSE as the fastest path to getting our protcol in
139.
by huntaub 2y ago
These distributed storage systems solve very similar problems, depending on how you use them. Our target customers aren't looking to deploy their own infrastructure, so having a "single-click" option without having to think a
140.
by huntaub 2y ago
There isn't any reason that it shouldn't be a supported use case, depending on your exact performance needs and workflow. It's very similar to ObjectiveFS except that it operates on the data in your S3 bucket in it's n
141.
by huntaub 2y ago
Sure can, full disclosure, copied from a comment below: Thanks for the question! Mountpoint for Amazon S3 is a FUSE layer that doesn't support full POSIX semantics. For example, you can't use Mountpoint for Amazon S3 for random wr
142.
by huntaub 2y ago
The funny thing about storage is that all of the problems are the same! Ultimately, there is no problem that cannot be solved with caching, journaling, write-ahead logging, etc. I think what makes the problem space so interesting is how a m
143.
by huntaub 2y ago
This isn't a technical limitation, per se, but a time limitation in terms of getting to the place where we feel comfortable supporting those environments for the public. I still wouldn't recommend mounting it from a local environm
144.
by huntaub 2y ago
> It's pure magic when you mount a filesystem on clean machine and all your data is "just there." I totally agree! I am hoping that Regatta can power a future where teams don't need more than ~8 GiB of local storage f
145.
by huntaub 2y ago
This is exactly right, and something that we think is particularly important for applications that care about data consistency. Often times, we see that customers want to be able to quickly hand off tasks from one instance to another which
146.
by huntaub 2y ago
Hey there, I have mutual friends with some of the Nasuni folks, and I have a lot of respect for what they do. In particular, Nasuni stores data in a proprietary block format in your S3 bucket, so you can't connect it to existing data s
147.
by huntaub 2y ago
Hey, thanks for asking. It very much depends on which aspect of Regatta you're interested in using. I know of a couple of different architectures -- some folks wrote in "rclone" in the thread, I know of people using SeaweedFS
148.
by huntaub 2y ago
Yeah, that's exactly right. I had some... experiences with Spark recently, that convinced me that this is something that could really help. I also really like the idea that organizations can continue to use S3 as the source of truth fo
149.
by huntaub 2y ago
I think this is a great insight, and something that I think about often. The challenge that I see is that the scientist archetype (whether it's data science, AI researcher, or anything else) isn't really interested in doing softwa
150.
by huntaub 2y ago
It depends on what you're doing with EFS! For the most part, I would expect to be lower cost than EFS. If you're doing where individual files are primarily written or accessed from an individual instance, I would expect a signifi

More ›