4 ms·

Thank you, now I understand what the special 0-byte object refers to. It represents an empty folder. Fair enough, basing folders on object names split by / is

Thank you, now I understand what the special 0-byte object refers to. It represents an empty folder. Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees.

klodolph 3y ago
> Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees. What, exactly, is inefficient about it? Think for a moment about the data structures you would use to represent a directory structure in a filesystem, and the data structures you would use to represent a key/value store. With a filesystem, if you split a string /some/dir/file.jpg into three parts, “some”, “dir”, “file.jpg”, then you are actually making a decision about the tree structure. And here’s a question—is that a balanced tree you got there? Maybe it’s completely unbalanced! That’s actually inefficient. Let’s suppose, instead, you treat the key as a plain string and stick it in a tree. You have a lot of freedom now, in how you balance the tree, since you are not forced to stick nodes in the tree at every / character. It’s just a different efficiency tradeoff. Certain operations are now much less efficient (like “rename a directory” which, on S3, is actually “copy a zillion objects). Some operations are more efficient, like “store a file” or “retrieve a file”.
- School-Cotton 3y ago
  I think what you’re describing is simply not a hierarchical file system. It’s a different thing that supports different operations and, indeed, is better or worse at different operations.
- afiori 3y ago
  I think it is fair to say that S3 (as named files) is not a filesystem and it is inefficient to use it directly as such for common filesystem use cases; the same way that you could say it for a tarball[0]. This does not make S3 a bad storage, just a bad filesystem, not everything needs to be a filesystem. Arguably is it good that S3 is not a filesystem, as it can be a leaky abstraction eg in git you cannot have two tags name "v2" and "v2/feature-1" as you cannot have both a file and a folder with the same name. For something more closely related to URLs than filenames forcing a filesystem abstraction is a limitation as "/some/url", "/some/url/", and "/some/url/some-default-name-decided-by-the-webserver" can be different.[1] [0] where a different tradeoff is that searching a file by name is slower but reading many small files can be faster. [1] maybe they should be the same, but enforcing it is a bad idea
- riehwvfbk 3y ago
  Renaming a folder is inefficient.
inkyoto 3y ago
> […] what the special 0-byte object refers to. It represents an empty folder. Alas, no. It represents a tag, e.g. «folder/», that points to a zero byte object. You can then upload two files, e.g. «folder/file1.txt» and «folder/file2.txt», delete the «folder/», being a tag, and still have the «folder/file1.txt» and «folder/file2.txt» file intact in the S3 bucket. Deleting «folder/» in a traditional file system, on the other hand, will also delete «file1.txt» and «file2.txt» in it.
- dchest 3y ago
  It's a matter of a client UI implementation. You can't delete a non-empty folder with POSIX API on common filesystems or FTP too. However, there are file managers, FTP clients, and S3 clients that will do that for you by deleting individual files.
  - _flux 3y ago
    But if the S3 semantics are not helping you, e.g. with multiple clients doing copy/move/delete operations in the hierarchy you could still end up with files that are not in "directories". So essentially an S3 file manager must be able to handle the situation where there are files without a "directory"—and that I assume is also the most common case as well for S3. Might just not have the "directories" in the first place.
    - klodolph 3y ago
      I have personally never seen the 0-byte files people keep talking about here. In every S3 bucket I’ve ever looked at, the “directories” don’t exist at all. If you have a dir/file1.txt and dir/file2.txt, there is NO such object as dir. Not even a placeholder.
      - _flux 3y ago
        Yeah, this post was the first one I had even heard of them.
- 3y ago
gjvc 3y ago
"folders" do not exist in S3 -- why do you keep insisting that they do? They appear to exist because the key is split on the slash character for navigation in the web front-end. This gives the familiar appearance of a filesystem, but the implementation is at a much higher level.

klodolph 3y ago

> Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees. What, exactly, is inefficient about it? Think for a moment about the data structures you would use to represent a directory structure in a filesystem, and the data structures you would use to represent a key/value store. With a filesystem, if you split a string /some/dir/file.jpg into three parts, “some”, “dir”, “file.jpg”, then you are actually making a decision about the tree structure. And here’s a question—is that a balanced tree you got there? Maybe it’s completely unbalanced! That’s actually inefficient. Let’s suppose, instead, you treat the key as a plain string and stick it in a tree. You have a lot of freedom now, in how you balance the tree, since you are not forced to stick nodes in the tree at every / character. It’s just a different efficiency tradeoff. Certain operations are now much less efficient (like “rename a directory” which, on S3, is actually “copy a zillion objects). Some operations are more efficient, like “store a file” or “retrieve a file”.

School-Cotton 3y ago
I think what you’re describing is simply not a hierarchical file system. It’s a different thing that supports different operations and, indeed, is better or worse at different operations.
afiori 3y ago
I think it is fair to say that S3 (as named files) is not a filesystem and it is inefficient to use it directly as such for common filesystem use cases; the same way that you could say it for a tarball[0]. This does not make S3 a bad storage, just a bad filesystem, not everything needs to be a filesystem. Arguably is it good that S3 is not a filesystem, as it can be a leaky abstraction eg in git you cannot have two tags name "v2" and "v2/feature-1" as you cannot have both a file and a folder with the same name. For something more closely related to URLs than filenames forcing a filesystem abstraction is a limitation as "/some/url", "/some/url/", and "/some/url/some-default-name-decided-by-the-webserver" can be different.[1] [0] where a different tradeoff is that searching a file by name is slower but reading many small files can be faster. [1] maybe they should be the same, but enforcing it is a bad idea
riehwvfbk 3y ago
Renaming a folder is inefficient.

inkyoto 3y ago

> […] what the special 0-byte object refers to. It represents an empty folder. Alas, no. It represents a tag, e.g. «folder/», that points to a zero byte object. You can then upload two files, e.g. «folder/file1.txt» and «folder/file2.txt», delete the «folder/», being a tag, and still have the «folder/file1.txt» and «folder/file2.txt» file intact in the S3 bucket. Deleting «folder/» in a traditional file system, on the other hand, will also delete «file1.txt» and «file2.txt» in it.

dchest 3y ago
It's a matter of a client UI implementation. You can't delete a non-empty folder with POSIX API on common filesystems or FTP too. However, there are file managers, FTP clients, and S3 clients that will do that for you by deleting individual files.
- _flux 3y ago
  But if the S3 semantics are not helping you, e.g. with multiple clients doing copy/move/delete operations in the hierarchy you could still end up with files that are not in "directories". So essentially an S3 file manager must be able to handle the situation where there are files without a "directory"—and that I assume is also the most common case as well for S3. Might just not have the "directories" in the first place.
  - klodolph 3y ago
    I have personally never seen the 0-byte files people keep talking about here. In every S3 bucket I’ve ever looked at, the “directories” don’t exist at all. If you have a dir/file1.txt and dir/file2.txt, there is NO such object as dir. Not even a placeholder.
    - _flux 3y ago
      Yeah, this post was the first one I had even heard of them.
3y ago

gjvc 3y ago

"folders" do not exist in S3 -- why do you keep insisting that they do? They appear to exist because the key is split on the slash character for navigation in the web front-end. This gives the familiar appearance of a filesystem, but the implementation is at a much higher level.