huntaub

6 ms·

31.
by huntaub 10mo ago
Shot you an email about how we can potentially help you with this.
32.
by huntaub 11mo ago
The root cause here is just that managing any kind of storage service is instantly painful. The property of "not losing data" means that you are sort of required to always be doing something in order to keep it healthy.
33.
by huntaub 11mo ago
I believe this is also changing with instances that now allow you to adjust the ratio of throughput on the NIC that's dedicated to EBS vs. general network traffic (with the intention, I'm sure, that people would want more EBS
34.
by huntaub 1y ago
I’d be happy to chat more about your needs and try to help recommend a path forward. Feel free to shoot me an email at the address in my profile.
35.
Show HN: Run SQLite Directly on S3 from AWS or GCP(docs.archil.com)
2 points by huntaub 1y ago|0 comments
36.
by huntaub 1y ago
> As a secondary, I wonder if it's possible to actively use a SQLite interface against a database file on S3, assuming a single server/instance is the actual active connection. You could achieve this today using one of the many
37.
by huntaub 1y ago
S3 Mountpoint is exposing a POSIX-like file system abstraction for you to use with your file-based applications. Foyer appears to be a library that helps your application coordinate access to S3 (with a cache), for applications that don&#x2
38.
by huntaub 1y ago
Storage Gateway is an appliance that you connect multiple instances to, this appears to be a library that you use in your program to coordinate caching for that process.
39.
by huntaub 1y ago
These are, effectively, different use cases. You want to use (and pay for) Express One Zone in situations in which you need the same object reused from multiple instances repeatedly, while it looks like this on-disk or in-memory cache is
40.
by huntaub 1y ago
Yes, definitely. S3 has a time to first byte of 50-150ms (depending on how lucky you are). If you're serving from memory that goes to ~0, and if you're serving from disk, that goes to 0.2-1ms. It will depend on your needs though,
41.
by huntaub 1y ago
Woah buddy, I worked with Andy for years and this is not my experience. Moving a large product like S3 around is really, really difficult, and I've always thought highly of Andy's ability to: (a) predict where he thought the produ
42.
by huntaub 1y ago
Some quick questions that came up in the last post, that I wanted to go ahead and address: How are you different than existing products like S3 Mountpoint, S3FS, ZeroFS, ObjectiveFS, JuiceFS, and cunoFS? Archil is designed to be a general-p
43.
Show HN: Archil's one-click infinite, S3-backed local disks now available
19 points by huntaub 1y ago|2 comments
44.
by huntaub 1y ago
Basically, we are building this at Archil ( https://archil.com ). The reason these things are generally super expensive is that it’s incredibly hard to build.
45.
by huntaub 1y ago
My (limited) understanding was that the industry previously knew that it was unsafe to share GPUs between tenants, which is why the major cloud providers only sell dedicated GPUs.
46.
by huntaub 1y ago
This is actually not the case. The TLS stream ensures that the packets transferred between your machine and S3 are not corrupted, but that doesn't protect against bit-flips which could (though, obviously, shouldn't) occur from wit
47.
by huntaub 1y ago
Thanks! Under the hood, when you mount an Archil volume, you connect to a fleet of instances that we're managing with SSD drives attached, which cache reads+writes before hitting the underlying data in your S3 bucket.
48.
by huntaub 1y ago
Hey, I'm Hunter -- the founder of Archil. I'll be around in the comments to answer any questions that people have about the platform, or how things have changed since the Fall.
49.
Archil: From a file system, to a data company(archil.com)
12 points by huntaub 1y ago|3 comments
50.
Archil (YC F24) Is Hiring a Distributed Systems Engineer (In-Person, SF)
1 points by huntaub 1y ago
51.
by huntaub 1y ago
This is exactly what I wanted for our team when I was at AWS. There are so many versions of operations which are just slightly too dangerous to automate, and this provides a path to iteratively building that up. Congratulations!
52.
by huntaub 1y ago
i-series instances have direct-attached drives
53.
by huntaub 1y ago
I think that worrying that a self-hosted file system has a backdoor to exfiltrate data is an odd concern. Security concerns are (obviously) normal, but you should not be exposing these kinds of services to the public internet (or giving the
54.
by huntaub 1y ago
It's a very different architecture. 3FS is storing everything on SSDs, which makes it extremely expensive but also low latency (think ~100-300us for access). JuiceFS stores data in S3, which is extremely cheap but very high latency (~2
55.
by huntaub 1y ago
My guess is going to be that performance is pretty comparable, but it looks like Seaweed contains a lot more management features (such as tiered storage) which you may or may not be using.
56.
by huntaub 1y ago
Yes, I think this is probably true. I've worked with a lot of different hedge funds who have a similar problem -- lots of shared data that they need in a file system so that they can do backtesting of strategies with things like kdb+.
57.
by huntaub 1y ago
Well, for active data, the idea is that the replication within the system is enough to keep the data alive from instance failure (assuming that you're doing the proper maintenance and repairing hosts pretty quickly after failure). Back
58.
by huntaub 1y ago
I think that's a pretty odd concern to have. What would you imagine that looks like? If you're running these kinds of things securely, you should be locking down the network access to the hosts (they don't need outbound inter
59.
by huntaub 1y ago
IMO this is the problem with all storage clusters that you run yourself, not just Ceph. Ultimately, keeping data alive through instance failures is just a lot of maintenance that needs to happen (even with automation).
60.
by huntaub 1y ago
I think that the author is spot on, there are a couple of dimensions in which you should evaluate these systems: theoretical limits, efficiency, and practical limits. From a theoretical point of view, like others have pointed out, parallel

More ›