4 ms·

The case sensitivity one is easy, here's a thing that's more likely to be entirely unintuitive: S3 paths are fake. Yes, it accepts uploads to "/builds/1/instal

The case sensitivity one is easy, here's a thing that's more likely to be entirely unintuitive: S3 paths are fake. Yes, it accepts uploads to "/builds/1/installer.exe", and yes, you can list what's in /builds, but all of that is a simulation. What you actually did was to upload a file literally named '/builds/1/installer.exe' with the '/' as part of the name. So, "/builds/1//installer.exe" and "/builds//1/installer.exe" are also possible to upload and entirely different files. Because it's the name of a key, there's no actual directories.

hnlmorg 2y ago
Yeah, the prefix thing is a source of so many bugs. I get why AWS took that approach and it’s actually a really smart approach but it still catches so many developers out. Just this year our production system was hit by a weird bug that took 5 people to find. Turned out the issue was an object literally just named “/“ and the software was trying to treat it as a path rather than a file.
- BirAdam 2y ago
  Happened at my job yesterday.
reddalo 2y ago
You're right, unless you use the new S3 "Directory buckets" [1], which make the entire thing even more confusing! [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/direct...
- temac 2y ago
  and which also according to the command docs i read recently add random limitations to maybe half of the S3 API I'm not sure reusing (parts of) the protocol was a good idea given unrelated half of it was already legacy and discouraged (e.g. the insane permission model with policies and acls) or even when not, let's say... weird and messy.
cnity 2y ago
Aside from resolution of paths to some canonical version (e.g. collapsing redundant /'s in your example), what actually is an "actual directory" other than a prefix?
- mock-possum 2y ago
  An actual directory is a node in a tree. It may be the child of a parent and it may have siblings - its children may in turn be parents with children of their own.
- ncruces 2y ago
  An actual directory knows its direct children. An actual directory A with 2 child directories A/B and A/C, each of which with 1 million files, doesn't need to go through millions of entries in an index to figure out the names of B and C. It's also not possible for A and A/ to exist independently, and for A to be a PDF, A/ an MP4 and A/B a JPG under the A/ ”directory". All of which are possible, simultaneously, with S3.
  - marcosdumay 2y ago
    Just to add, "directory" is something you use to look-up information. The thing is called "directory" exactly because you go there to look what its children are.
mdaniel 2y ago
also, don't overlook that "/" is only the default path delimiting character; one is free to use your favorite other character if you need a filename with a "/" in it: https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#:~:text=A%20delimiter%20is%20a%20character%20that%20you%20use%20to%20group%20keys https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...

hnlmorg 2y ago

Yeah, the prefix thing is a source of so many bugs. I get why AWS took that approach and it’s actually a really smart approach but it still catches so many developers out. Just this year our production system was hit by a weird bug that took 5 people to find. Turned out the issue was an object literally just named “/“ and the software was trying to treat it as a path rather than a file.

BirAdam 2y ago
Happened at my job yesterday.

reddalo 2y ago

You're right, unless you use the new S3 "Directory buckets" [1], which make the entire thing even more confusing! [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/direct...

temac 2y ago
and which also according to the command docs i read recently add random limitations to maybe half of the S3 API I'm not sure reusing (parts of) the protocol was a good idea given unrelated half of it was already legacy and discouraged (e.g. the insane permission model with policies and acls) or even when not, let's say... weird and messy.

cnity 2y ago

Aside from resolution of paths to some canonical version (e.g. collapsing redundant /'s in your example), what actually is an "actual directory" other than a prefix?

mock-possum 2y ago
An actual directory is a node in a tree. It may be the child of a parent and it may have siblings - its children may in turn be parents with children of their own.
ncruces 2y ago
An actual directory knows its direct children. An actual directory A with 2 child directories A/B and A/C, each of which with 1 million files, doesn't need to go through millions of entries in an index to figure out the names of B and C. It's also not possible for A and A/ to exist independently, and for A to be a PDF, A/ an MP4 and A/B a JPG under the A/ ”directory". All of which are possible, simultaneously, with S3.
- marcosdumay 2y ago
  Just to add, "directory" is something you use to look-up information. The thing is called "directory" exactly because you go there to look what its children are.

mdaniel 2y ago

also, don't overlook that "/" is only the default path delimiting character; one is free to use your favorite other character if you need a filename with a "/" in it: https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#:~:text=A%20delimiter%20is%20a%20character%20that%20you%20use%20to%20group%20keys https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...