3 ms·

Is this something thats a deficiency with S3, or is there not a more purpose built offering?

• KaiserPro 3y ago
  S3, as you know, is a key-based object store, with a hack that allows directories ('/' is just part of the key name, and is filtered) So its less of a deficiency, more of a tradeoff they went with to get performance/uptime. Enumerating the keys on an S3 bucket is pretty slow, so it makes any kind of listing operation for a follow on system slow. Having a metadata cache is sensible option, especially if you have complete control of a bucket and are able to be canonical. (ie everything talks to your DB to get keynames) But! what you can't do write to the middle of a file, you need to upload the whole thing again. This is not a problem for a lot of workflows, but for POSIX filesystems thats going to cause problems. You can seek to the middle of a file, write to it, the client will upload it to S3. What happens if another system modifies that file when you are uploading? Sounds like a locking nightmare.
  • snerbles 3y ago
    In the case of JuiceFS the file is split into chunks and those are the objects uploaded to S3 - treating it like a high latency block device. The downside is that metadata here isn't just a cache, it is necessary to operate in this fashion and must be backed up.