6 ms·
My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3,
by leetrout 3y ago
My big pet peeve is AWS adding buttons in the UI to make "folders".
It is also a fiction! There are no folders in S3.
> When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to support the idea of folders.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-...
- riehwvfbk 3y agoIs that really so different from how folders work on other systems? A directory inode is just an inode.
- daynthelife 3y agoThe payload still contains a list of other inodes though
- klodolph 3y agoYes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | parent | name | data | +-------+--------+----------+-------+ | 1 | (null) | some | (dir) | | 2 | 1 | dir | (dir) | | 3 | 2 | file.jpg | jpeg | +-------+--------+----------+-------+ In S3 (and other object stores), the table is like this: +-------------------+------+ | key | data | +-------------------+------+ | some/dir/file.jpg | jpeg | +-------------------+------+ The kind of queries you can do is completely different. There are no inodes in S3. There is just a mapping from keys to objects. There’s an index on these keys, so you can do queries—but the / character is NOT SPECIAL and does not actually have any significance to the S3 storage system and API. The / character only has significance in the UI. You can, if you want, use a completely different character to separate “components” in S3, rather than using /, because / is not special. If you want something like “some:dir:file.jpg” or “some.dir.file.jpg” you can do that. Again, because / is not special.
- fiddlerwoaroof 3y agoExcept, S3 does let you query by prefix and so the keys have more structure than the second diagram implies: they’re not just random keys, the API implies that common prefixes indicate related objects.
- klodolph 3y agoThat’s kind of stretching the idea of “more structure” to the breaking point, I think. The key is just a string. There is no entry for directories. > the API implies that common prefixes indicate related objects. That’s something users do. The API doesn’t imply anything is related. And prefixes can be anything, not just directories. If you have /some/dir/file.jpg, then you can query using /some/dir/ as a prefix (like a directory!) or you can query using /so as a prefix, or /some/dir/fil as a prefix. It’s just a string. It only looks like a directory when you, the user, decide to interpret the / in the file key as a directory separator. You could just as easily use any other character.
- hiyer 3y agoOne operation where this difference is significant is renaming a "folder". In UNIX (and even UNIX-y distributed filesystems like HDFS) a rename operation at "folder" level is O(1) as it only involves metadata changes. In S3, renaming a "folder" is O(number of files).
- erik_seaberg 3y agoYou can create a simulated directory, and write a bunch of files in it, but you can't atomically rename it--behind the scenes each file needs to be copied from old name to new.
- 8organicbits 3y agoAnother challenge is directory flattening. On a file system "a/b" and "a//b" are usually considered the same path. But on S3 the slash isn't a directory separator, so the paths are distinct. You need to be extra careful when building paths not to include double slashes. Many tools end up handling this by showing a folder named "a" containing a folder named "" (empty string). This confuses users quite a bit. It's more than the inodes, it's how the tooling handles the abstraction.
- hnlmorg 3y agoCoincidentally I ran into an issue just like this a week ago. A customer facing application failed because there was an object named “/foo/bar” (emphasis on the leading slash). This created a prefix named “/“ which confused the hell out of the application.
- ithkuil 3y agoIn S3 each file is identified with a full path. Not only you cannot rename a single file, but you also cannot rename a "folder" (because that would imply a bulk rename on a large number of children of that "folder") This is the fundamental difference between a first class folder and just a convention on prefixes of full path names. If you don't allow renames, it doesn't really make sense to have each "folder" store the list of the children. You can instead have a giant ordered map (some kind of b-tree) that allows you for efficient lookup and scanning neighbouring nodes.
- lukeh 3y agoUMich LDAP server, upon which many were based, stored entrys’ hierarchical (distinguished) names with each entry, which I always found a bit weird. AD, eDirectory, and the OpenLDAP HDB backend don’t have this problem.
- solumunus 3y agoWhat exactly do you think a folder is? It’s just an abstraction for organising data.
- winwang 3y agoI'm having a lot of fun imagining this being said to a kid who's trying to buy some folders for school.
- klodolph 3y agoS3 doesn’t have that abstraction. The console UI shows folders but they don’t actually exist in S3. They’re made up by the UI.
- 3weeksearlier 3y agoIt sounds like they have that abstraction in the UI. But if the CLI and API don't have it too, that's weird.
- klodolph 3y agoYeah, the UI and CLI show you “folders”. It’s a client-side thing that doesn’t exist in the actual service. Behind the scenes, the clients are making specific types of queries on the object keys. You can’t examine when a folder was created (it doesn’t exist in the first place), you can’t rename a folder (it doesn’t exist), you can’t delete a folder (again, it doesn’t exist).
- throwitaway222 3y agoThat's just an implementation detail of well known filesystems.
- klodolph 3y agoI don’t think that’s a defensible standpoint. Folders are an important part of the way most people use filesystems.
- nostrebored 3y agoWeird that it says folders now. I remember it being very strictly called a prefix when I was at AWS.
- paranoidrobot 3y agoI think it's just the Web console, It's still prefix in the APIs and CLI. https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...
- Izkata 3y agoThe web console even collapses them like folders on slashes, further obfuscating how it actually works. I remember having to explain to coworkers why it was so slow to load a large bucket.
- klodolph 3y agoI see you getting downvotes, but you’re speaking the honest truth, here.
- halayli 3y agoI don't know why you are being downvoted, what you said is true and confuses many newcomers.
- highwaylights 3y agoThis! I’m fine with it, I actually appreciate the logic and simplicity behind it, but the amount of times I’ve tried to explain why “folders” on S3 keep disappearing while people stare at me like I’m an idiot is really frustrating. (When you remove the last file in a “folder” on S3, the “folder” disappears, because that pattern no longer appears in the bucket k/v dictionary so there’s no reason to show it as it never existed in the first place).
- wkat4242 3y agoHmm well there's no folders but if you interact with the object the URL does become nested. So in a sense it does behave exactly like a folder for all intents and purposes when dealing with it that way. It depends what API you use I guess. I use S3 just as a web bucket of files (I know it's not the best way to do that but it's what I could easily obtain through our company's processes). But in this case it makes a lot of sense though I try to avoid making folders. But other people using the same hosting do use them.
- raverbashing 3y agoExcept stuff like s3 cli has all these weird names for normal filesystem items and you have to bang your head to try to figure it out what it all means (also don't get me started on the whole s3api thing)
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]