3 ms·

S3's metadata speed is horrifically slow. Part of the reason fake filesystems on S3 also perform badly is that checking anything to do with where a file is loca

S3's metadata speed is horrifically slow. Part of the reason fake filesystems on S3 also perform badly is that checking anything to do with where a file is located, takes >250ms. Now if you think about how many files are in a git repo, and how many are touched when you commit, you can see the problem. I'm not sure JuiceFS is the answer, given that the metadata is stored in memory. but it is an answer. personally you're better off with their managed lustre offering

Geisterde 3y ago
Is this something thats a deficiency with S3, or is there not a more purpose built offering?
- KaiserPro 3y ago
  S3, as you know, is a key-based object store, with a hack that allows directories ('/' is just part of the key name, and is filtered) So its less of a deficiency, more of a tradeoff they went with to get performance/uptime. Enumerating the keys on an S3 bucket is pretty slow, so it makes any kind of listing operation for a follow on system slow. Having a metadata cache is sensible option, especially if you have complete control of a bucket and are able to be canonical. (ie everything talks to your DB to get keynames) But! what you can't do write to the middle of a file, you need to upload the whole thing again. This is not a problem for a lot of workflows, but for POSIX filesystems thats going to cause problems. You can seek to the middle of a file, write to it, the client will upload it to S3. What happens if another system modifies that file when you are uploading? Sounds like a locking nightmare.
  - snerbles 3y ago
    In the case of JuiceFS the file is split into chunks and those are the objects uploaded to S3 - treating it like a high latency block device. The downside is that metadata here isn't just a cache, it is necessary to operate in this fashion and must be backed up.

3 ms·

S3's metadata speed is horrifically slow. Part of the reason fake filesystems on S3 also perform badly is that checking anything to do with where a file is loca

by KaiserPro 3y ago

Geisterde 3y ago
Is this something thats a deficiency with S3, or is there not a more purpose built offering?
- KaiserPro 3y ago
  S3, as you know, is a key-based object store, with a hack that allows directories ('/' is just part of the key name, and is filtered) So its less of a deficiency, more of a tradeoff they went with to get performance/uptime. Enumerating the keys on an S3 bucket is pretty slow, so it makes any kind of listing operation for a follow on system slow. Having a metadata cache is sensible option, especially if you have complete control of a bucket and are able to be canonical. (ie everything talks to your DB to get keynames) But! what you can't do write to the middle of a file, you need to upload the whole thing again. This is not a problem for a lot of workflows, but for POSIX filesystems thats going to cause problems. You can seek to the middle of a file, write to it, the client will upload it to S3. What happens if another system modifies that file when you are uploading? Sounds like a locking nightmare.
  - snerbles 3y ago
    In the case of JuiceFS the file is split into chunks and those are the objects uploaded to S3 - treating it like a high latency block device. The downside is that metadata here isn't just a cache, it is necessary to operate in this fashion and must be backed up.