13 ms·

Things you wish you didn't need to know about S3

336 points by miles 2y ago

nathants 2y ago
a fun read! i just built a live streaming platform[1]. chat, m3u8, ts, all objects. list operations used to concat chat objects. works perfectly, object storage is such a great api. it uses r2 as the only data store, which is a delightfully simple s3-alike. the only thing i miss are some of the advanced listv2 features. too anyone not enjoying s3 complexity, go try r2. 1. https://nathants.com/live https://nathants.com/live
est31 2y ago
Regarding the deletion point, note that you cannot delete S3 buckets that are non-empty, so in order to actually delete data you have to first manually delete them. Of course, if any action is allowed then that is as well. But still, it's not a single request away for any non-trivial bucket.
yencabulator 2y ago
Object lock until 2099 that can only be cancelled by deleting the AWS account is *nasty*.
yencabulator 2y ago
The S3 API, and most of AWS, is a kludgy legacy mess. Is there any chance of getting a new, industry-standard not just industry-adopted, simpler but more usable[1] common API? We already have some client libraries that try to paper over the differences (https://gocloud.dev/ https://gocloud.dev/, https://crates.io/crates/object_store https://crates.io/crates/object_store), but wouldn't it be nice to have just one wire protocol? [1]: E.g. standardize create-if-not-exist, even if S3 doesn't implement that.
- mdaniel 2y ago
  My life experience has been that this line of thinking results in two separate outcomes: "the great thing about standards is that there are so many to choose from" <https://xkcd.com/927/ https://xkcd.com/927/>, and that publishing a standard does absolutely zero for getting buy-in, with that last one made worse by any short-sighted fools who publish a bunch of words without a reference impl or ideally a TCK. As this whole thread has shown, for 10 people there will be 15 understandings of any given sentence, which is very very bad when trying to get two computers to agree on something
unethical_ban 2y ago
>Schrodiner’s cat is the one that’s both alive and de-lifed at the same time, right? It was alive and dead at the same time. Don't use censor-appeoved language outside Tiktok!
coolgoose 2y ago
A lot of them are interesting points, but I am not sure I agree with the complaint the file system is case sensitive. That's how it should be and I am annoyed at macos for not having it.
- yencabulator 2y ago
  It would be incredibly weird not to be able to store urlsafe-base64 encoded paths on S3.
- ansgri 2y ago
  TIL. I've switched to Mac about a year ago and it's sufficiently Linux-like to take such things for granted. I wonder what other surprises are going to bite me in the future.
  - aldanor 2y ago
    Sufficiently BSD-like :)
- sholladay 2y ago
  macOS is case preserving, though. To me, it’s the best of both worlds. You can stylize your file names and they will be respected, but you don’t have to remember how they are stylized when you are searching for or processing them because search is case insensitive.
  - smt88 2y ago
    Windows is also case-insensitive but case-preserving
  - self_awareness 2y ago
    Maybe macOS is case-preserving, but it's not encoding-preserving. If you create a file using a composed UTF-8 "A", the filesystem layer will decompose the string to another form, and create the filename using that decomposed form "B". Of course, "A" and "B" when compared can be completely different (even when compared using case insensitivity enabled), yet will point to the same file. More info here: https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/ https://eclecticlight.co/2021/05/08/explainer-unicode-normal...
willsoon 2y ago
My wife - she's a future developer, I'm a lawyer - has had enough of me explaining some of these details in the form of concerns. I am somewhat satisfied to know that my perplexities are not imaginary.
- trallnag 2y ago
  What's a future developer?
  - cynicalsecurity 2y ago
    A developer that is learning programming or a specific technology? Isn't it self-explanatory or what problem did you have understanding it?
    - krisoft 2y ago
      > Isn't it self-explanatory No. > what problem did you have understanding it? It is simply not a phrase I commonly read. When I google it I find a visa consultancy under the name, and not much else. Curiously your comment is also among the results. The problem is that the phrase "<something> developer" is used to describe what the person develops. A "real-estate developer" invests in real estate as a business. A "web developer" develops web applications, a "game developer" develops games, and so on and so on. So reading the word I immediately thought they mean someone who is developing the future? Like idk Douglas Engelbart or Tim Berners-Lee or someone like that. If you want to write that someone is learning to become a developer I would recommend the much less confusing "developer in training" phrase, or even better if you just write "they are learning to become a developer".
      - willsoon 2y ago
        Bad English case. It just is. I pay a lot of attention when I write English, but you can always tell when someone isn't a native English speaker in about two words. Je suis désolé.
isoprophlex 2y ago
That bit on failed multipart uploads invisibly sticking around (and incurring storage cost, unless you explicitly specify some lifecycle magic)... just ugh. And I thought one of the S-es was for 'simple'.
- teitoklien 2y ago
  That S for simple stands for simple ways to skyrocket your expenses
- e____g 2y ago
  Yes, that sucks. Blame ahenry@, then GM for S3. My proposal was that parts of incomplete uploads would stick around for only 24 hours after the most recent activity on the upload, and you wouldn't be charged for storage during that time. ahenry@ vetoed that.
  - vbezhenar 2y ago
    Why would you propose something that makes company earn less money? I'm sure that at Amazon scale, this misfeature earned millions of dollars.
    - spacebanana7 2y ago
      Customer relationships. I recall a Bezos quote along the lines of "It's better to lose a refund than to lose a customer".
- Hamuko 2y ago
  Yeah, I've definitely treaded on the storage cost landmine. Thankfully it was just some cents in my case, but it's really infuriating how badly the console exposes the information.
- scosman 2y ago
  “Simple” was coined when the alternative was managing a fleet of servers with disks . Time changes everything.
tedunangst 2y ago
[flagged]
- kthielen 2y ago
  A career change, or a job change?
nh2 2y ago
How about the fact that S3 is not suitable for web serving due to high latencies (in standard storage classes)? Many people think you can just host the resources for your websites, such as images or fonts, straight on S3. But that can make for a shitty experience: > applications can achieve consistent small object latencies (and first-byte-out latencies for larger objects) of roughly 100–200 milliseconds. From: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi...
- metadat 2y ago
  Put a memcached instance in front and you're good.
- lulznews 2y ago
  Pretty much everyone knows this …
- teitoklien 2y ago
  Most folks use S3 as a source for AWS cloudfront for serving content You can even use cloudfront signed cookies to give specific users cdn access to only specific content owned by them on S3. How cool is that.
- WatchDog 2y ago
  Typically you would use cloudfront with S3 if you want to use it for serving web assets. It will cache frequently accessed assets, and in addition to reducing latency may reduce cost quite a bit.
- mcqueenjordan 2y ago
  s3 is not optimized to directly serve websites, but to durably store and retrieve ~unlimited data.
- weird-eye-issue 2y ago
  This has been well known for over a decade
reddalo 2y ago
I can't trust myself using S3 (or any other AWS service). Nothing is straightforward, there are too many things going on, too much documentation that I should read, and even then (as OP shows) I may be accidentally and unknowingly expose everything to the world. I think I'll stick to actually simple services, such as Hetzner Storage Boxes or DigitalOcean Spaces.
- chamomeal 2y ago
  I like digital ocean spaces, but it has its own annoying quirks. Like I recently found out yif you pipe a video file larger than a few MB, it’ll drop the https:// https:// from the returned Location. So on every file upload I have to check if the location starts with https, and add it on if it’s not there. Of course the S3 node client GitHub issue says “sounds like a digital ocean bug”, and the digital ocean forums say “sounds like an S3 node client bug” lol
- slig 2y ago
  The way that DO handles secrets should scare anyone. Did you know that if you use their Container Registry and set it up so that your K8S has automatically access to it, their service will create a secret that has full access to your Spaces?
  - marcosdumay 2y ago
    Hum... Kubernetes is not on the GP's list...
    - slig 2y ago
      Fair enough, but not having scoped secrets is a red flag.
- maximinus_thrax 2y ago
  > Nothing is straightforward, there are too many things going on, too much documentation that I should read, and even then (as OP shows) I may be accidentally and unknowingly expose everything to the world. I took a break from cloud development for a couple of years (working mostly on client stuff) and just recently got back. I am shocked at the amount of complexity built over the years along with the cognitive load required for someone to build an ironclad solution in the public cloud. So many features and quirks which were originally designed to help some fringe scenario are now part of the regular protocol, so that the business makes sure nobody is turned away.
CaliforniaKarl 2y ago
> S3 buckets are the S3 API > … a relatively small part of the API requires HTTP requests to be sent to generic S3 endpoints (such as s3.us-east-2.amazonaws.com), while the vast majority of requests must be sent to the URL of a target bucket. I believe this is talking about virtual-hosted style and path-style methods for accessing the S3 API. From what I can see [0], at least for the REST API, the entire API works either with virtual-hosted style (where the bucket name is in the host part of the URL) and path-style (where the bucket name is in the path part of the URL). Amazon has been wanting folks to move over to the virtual-hosted style for a long time, but (as of 3+ years ago) the deprecation of path-style has been delayed[1]. This deprecation of path-style requests has been extremely important for products implementing the S3 API. For example… * MinIO uses path-style requests by default, requiring you set a configuration variable[2] (and set up DNS appropriately) to handle the virtual-hosted style. * Wasabi supports both path-style and virtual-hosted style, but "Wasabi recommends using path-style requests as shown in all examples in this guide (for example, http://s3.wasabisys.com/my-bucket/my-object http://s3.wasabisys.com/my-bucket/my-object) because the path-style offers the greatest flexibility in bucket names, avoiding domain name issues."[3]. Now here's the really annoying part: The REST API examples show virtual-hosting style, but path style works too! For example, take the GetBucketTagging example. Let's say you have bucket "karl123456" in region US-West-2. The example would have you do this: GET /?tagging HTTP/1.1 Host: karl123456.s3.amazonaws.com But instead, you can do this: GET /karl123456?tagging HTTP/1.1 Host: s3.us-west-2.amazonaws.com !! How do I know this? I tried it! I constructed a `curl` command do to the path-style request, and it worked! (I didn't use "karl123456", though.) So hopefully that helps resolve at least one of your S3 annoyances :-) [0]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAPI.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAP... [1]: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/ https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-... [2]: https://min.io/docs/minio/linux/reference/minio-server/settings/core.html#envvar.MINIO_DOMAIN https://min.io/docs/minio/linux/reference/minio-server/setti... [3]: https://docs.wasabi.com/docs/rest-api-introduction https://docs.wasabi.com/docs/rest-api-introduction
prpl 2y ago
Not one thing about different API limits for operations according to key prefixes, and how you need to contact support if you need partitioning and provide them the prefixes, huh?
pachico 2y ago
I have the feeling that the entire case (in)-sensitive discussions are usually too much English-centric. Allow me to iterate: I have the feeling way too many language discussions, especially in IT, are too much English-centric.
- reddalo 2y ago
  Speaking of non-English cultures, do Japanese case insensitive systems differentiate between hiragana and katakana? Because, in some ways, the two syllabaries remind me of uppercase and lowercase alphabets.
  - thaumasiotes 2y ago
    They're more like distinguishing between "o" and “ℴ”. Which is where the European idea of "capital letters" originates, but not how we think about them today.
  - presentation 2y ago
    A lot of the time forms will specifically ask for hiragana or katakana, or specify full width or half width characters. But basically it’s a mess there too
- teitoklien 2y ago
  > too much English-centric. Pretty glad about it considering how much more simpler ASCII was to work with compared to Unicode. I say it as a non native english speaker, programming has so many concepts and stuff already, its best not to make it more complex by adding a 101 different languages to account for. Unicode and Timezone, the two things that try to bring more languages and cultures to be accounted for while programming and look what happens, it creates the most amount of pain for everyone including non native english programmers. I dont want to write computer programs in my non-english native tongue, if that means i’ll have to start accounting for every major language while im programming. Its fine that IT discussions are so English-centric. Diversity is more complexity, and no one owns the english language, its just a tool used by people to communicate, thanks to that universal language, I can express my thoughts to most people in India, China, Japan, South America, etc due to having 1 common language. They all own the english language too, the moment they decided to speak in it. No need to bring diversity politics in IT. Best to keep it technical.
dale_glass 2y ago
The case sensitivity one is easy, here's a thing that's more likely to be entirely unintuitive: S3 paths are fake. Yes, it accepts uploads to "/builds/1/installer.exe", and yes, you can list what's in /builds, but all of that is a simulation. What you actually did was to upload a file literally named '/builds/1/installer.exe' with the '/' as part of the name. So, "/builds/1//installer.exe" and "/builds//1/installer.exe" are also possible to upload and entirely different files. Because it's the name of a key, there's no actual directories.
- hnlmorg 2y ago
  Yeah, the prefix thing is a source of so many bugs. I get why AWS took that approach and it’s actually a really smart approach but it still catches so many developers out. Just this year our production system was hit by a weird bug that took 5 people to find. Turned out the issue was an object literally just named “/“ and the software was trying to treat it as a path rather than a file.
  - BirAdam 2y ago
    Happened at my job yesterday.
- reddalo 2y ago
  You're right, unless you use the new S3 "Directory buckets" [1], which make the entire thing even more confusing! [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/direct...
  - temac 2y ago
    and which also according to the command docs i read recently add random limitations to maybe half of the S3 API I'm not sure reusing (parts of) the protocol was a good idea given unrelated half of it was already legacy and discouraged (e.g. the insane permission model with policies and acls) or even when not, let's say... weird and messy.
- 2y ago
xarope 2y ago
> There are only losers in this game, but at least we’ve all got a participation ribbon to comfort us in moments of angst. I sense the frustration...
swiftcoder 2y ago
> S3 isn’t the only service that works this way, Hosted Cognito UI endpoints do something similar (https://[your-user-pool-domain]/login https://[your-user-pool-domain]/login). Basically every new service that expects to deal with a lot of traffic should work this way (we did this also for AWS IoT). It's a hell of a lot easier to deal with load balancing and request routing if you can segment resources starting at the DNS level...
phrotoma 2y ago
Here's another fun one that took a colleague and I several days of analysis to diagnose. S3 will silently drop all requests once a single TCP connection has sent 100 HTTP requests. https://github.com/aws/aws-sdk-go/issues/2825 https://github.com/aws/aws-sdk-go/issues/2825
- seabrookmx 2y ago
  It doesn't silently drop, it sends a header indicating it's closed the TCP connection. This is a pretty common pattern whereby you want keep-alive.for performance reasons, but you don't want clients running _too long_ creating hot spots on your load balancers.
  - phrotoma 2y ago
    What header? Our client was envoy and it's pretty standards compliant, and it just kept trying to use the connection. Edit: I see that it is `connection: close` I wonder if that is new behaviour or if envoy did not honour it at the time we encountered the issue. Thanks for the info!
  - temac 2y ago
    Instead of closing the connection it sends a message stating that it is closed? Wow.
aao 2y ago
Related: https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1 https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-c...
- lotsofspots 2y ago
  About a week after that went viral - https://aws.amazon.com/about-aws/whats-new/2024/05/amazon-s3-no-charge-http-error-codes/ https://aws.amazon.com/about-aws/whats-new/2024/05/amazon-s3...
croes 2y ago
>Things you wish you didn't need to know about S3 >A time travel paradox in the title is a good place to start a blog post, don’t you think? Where is the paradox?
buggythebug 2y ago
Audi S3?
scosman 2y ago
Here is a good one: deleting billions of objects can be expensive if you call delete APIs. However you can set a wildcard or bucket wide object expiry of time=now for free. You’ll immediately stop being charged for storage, and AWS will manage making sure everything is deleted.
- est31 2y ago
  > You’ll immediately stop being charged for storage The effect of lifecycle rules is not immediate: they get applied in a once-per-day batch job, so the removal is not immediate.
  - electroly 2y ago
    That's true but OP clearly knows that already. You stop getting charged for storage as soon as the object is marked for expiration, not when the object is finally removed at AWS's leisure. You can check the expiration status of objects in the metadata panel of the S3 web console. https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy... > There may be a delay between the expiration date and the date at which Amazon S3 removes an object. You are not charged for expiration or the storage time associated with an object that has expired.
- ericpauley 2y ago
  Nit: the delete call is free, it’s the list call to get the objects that costs money. In theory if you know what objects you have from another source it’s free.
- bushbaba 2y ago
  Because AWS gets to choose when the actual deletes happen. The metadata is marked as object deleted but AWS can process the delete off peak. It also avoids the s3 api server being hammered with qps
  - yencabulator 2y ago
    An explicit delete operation can mark metadata the same way. The real difference is likely more along the lines of LSM compaction, with expiry they likely use a moral equivalent of https://github.com/facebook/rocksdb/wiki/Compaction-Filter https://github.com/facebook/rocksdb/wiki/Compaction-Filter to actually do the deletion.
julik 2y ago
A few more: * Multipart uploads cannot be performed from multiple machines having instance credentials (as the principal will be different and they don't have access to each other's multipart uploads). You need an actual IAM user if you want to assemble a multipart upload from multiple machines. * LIST requests are not only slow, but also very expensive if done in large numbers. There are workarounds ("bucket inventory") but they are neither convenient nor cheap * Bucket creation is not read-after-write consistent, because it uses DNS under the hood. So it is possible that you can't access a bucket right after creating it, or that you can't delete a bucket you just created until you waited enough for the changes to propagate. See https://github.com/julik/talks/blob/master/euruko-2019-no-such-bucket/no-such-bucket.pdf https://github.com/julik/talks/blob/master/euruko-2019-no-su... * You can create an object called "foo" and an object called "foo/bar". This will make the data in your bucket unportable into a filesystem structure (it will be a file clobbering a directory) * S3 is case-sensitive, meaning that you can create objects which will unportable into a filesystem structure (Rails file storage assumed a case-sensitive storage system, which made it break badly on macOS - this was fixed by always using lowercase identifiers) * Most S3 configurations will allow GETs, but will not allow HEADs. Apparently this is their way to prevent probing for object existence, I am not sure. Either way - cache-honoring flows involving, say, a HEAD request to determine how large an object is will not work (with presigned URLs for sure!). You have to work around this doing a GET with a Range: of "very small" (say, the first byte only) * If you do a lot of operations using pre-signed URLs, it is likely you can speed up the generation of these URLs by a factor of 10x-40x (see https://github.com/WeTransfer/wt_s3_signer https://github.com/WeTransfer/wt_s3_signer) * You still pay for storage of unfinished multipart uploads. If you are not careful and, say, these uploads can be initiated by users, you will be paying for storing them - there is a setting for deleting unfinished MP uploads automatically after some time. Do enable it if you don't want to have a bad time. These just off the top of my head :-) Paradoxically, S3 used to be revolutionaly and still is, onl multiple levels, a great products. But: plenty features, plenty caveats.
- dugmartin 2y ago
  The one that caught me a couple of weeks ago is multipart uploads have a minimum initial chunk size of 5 MiBs (https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts...). I built a streaming CSV post-processing pipeline in Elixir that uses Stream.transform (https://hexdocs.pm/elixir/Stream.html#transform/3 https://hexdocs.pm/elixir/Stream.html#transform/3) to modify and inject columns. The Elixir AWS and CSV modules handle streaming data in but the AWS module throws an error (from S3) if you stream "out" that totals less than 5 MiBs as is uses multi-part uploads which made me sad.
perpil 2y ago
A workaround for some of the limits of presigned urls like not being able to specify a max file size is to front your uploads using CloudFront OAC and CloudFront Functions. It costs more (.02/GB) but you can run a little JavaScript code to validate/augment headers between your user and S3 and you don't need to expose your bucket name. https://speedrun.nobackspacecrew.com/blog/2024/05/22/using-cloudfront-as-a-lightweight-proxy.html https://speedrun.nobackspacecrew.com/blog/2024/05/22/using-c...
jimbobthrowawy 2y ago
Those uploader decides rules are wild. Does that mean someone with a website poorly configured enough can have user content uploaded to, and later served from amazon glacier? (assuming a sufficiently motivated user)
- wmfiv 2y ago
  It does. But if you're concerend about this (and many of the other items mentioned), you can control access to those features using IAM. https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazons3.html https://docs.aws.amazon.com/service-authorization/latest/ref... The condition keys specifically are here and you can see keys to control access to storage class, tagging, etc. https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazons3.html#amazons3-policy-keys https://docs.aws.amazon.com/service-authorization/latest/ref...
gymbeaux 2y ago
No mention of how AWS/S3 approximates the size of a file to save CPU cycles. It used to drive me up a wall seeing S3 show a file size as slightly different than what it was. If I recall correctly, S3 uses 1000 rather than 1024 to convert bytes to KB and MB and GB. This saves CPU cycles but results in “rounding errors” with the file’s reported size. It’s discussed here, although they’re talking about it being “CLI vs console” which may be true? https://stackoverflow.com/questions/57201659/s3-bucket-size-differs-while-using-console-and-cli https://stackoverflow.com/questions/57201659/s3-bucket-size-...
- abadpoli 2y ago
  This almost certainly has nothing to do with “saving CPU cycles” and is most likely just that whoever created the Cloudwatch console used the same rounding that is used for all other metrics in CW, rather than the proper calculation for disk size, and it was a small enough issue that it was never caught until it was too late to change it because changing it would disrupt the customers that have gotten used to it.
  - presentation 2y ago
    If anything a bit shift could do the 1024 division faster
- DougBTX 2y ago
  S3 doesn't format the size for human display, the object's Size is returned in bytes: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.h...
  - gymbeaux 2y ago
    What I meant to say was S3 uses 2^20 to convert from bytes to kilobytes rather than 10^6 (e.g. what Windows uses). https://repost.aws/knowledge-center/s3-console-metric-discrepancy https://repost.aws/knowledge-center/s3-console-metric-discre... > 3 console and Storage Lens use base 2 conversion (/1024) to report storage metrics, and CloudWatch by default uses base 10 conversion (/1000)
anonymouse008 2y ago
I’ve become old. If you look at these things in disgust (ACL vs Policies, Delete bucket with s3:*, etc), you’re missing the point of (deterministic ?) software. It does what it is written to do, faithfully, and error out when not. When it doesn’t do as written or as documented, then yes… go full bore.
- temac 2y ago
  The doc is huge, and the principle of least astonishment is often not respected. Also third parties providers support a random subset of it given the protocol has nothing simple anymore (or maybe never had)

336 points by miles 2y ago

a fun read! i just built a live streaming platform[1]. chat, m3u8, ts, all objects. list operations used to concat chat objects. works perfectly, object storage is such a great api. it uses r2 as the only data store, which is a delightfully simple s3-alike. the only thing i miss are some of the advanced listv2 features. too anyone not enjoying s3 complexity, go try r2. 1. https://nathants.com/live https://nathants.com/live

est31 2y ago

Regarding the deletion point, note that you cannot delete S3 buckets that are non-empty, so in order to actually delete data you have to first manually delete them. Of course, if any action is allowed then that is as well. But still, it's not a single request away for any non-trivial bucket.

yencabulator 2y ago

Object lock until 2099 that can only be cancelled by deleting the AWS account is *nasty*.

yencabulator 2y ago

The S3 API, and most of AWS, is a kludgy legacy mess. Is there any chance of getting a new, industry-standard not just industry-adopted, simpler but more usable[1] common API? We already have some client libraries that try to paper over the differences (https://gocloud.dev/ https://gocloud.dev/, https://crates.io/crates/object_store https://crates.io/crates/object_store), but wouldn't it be nice to have just one wire protocol? [1]: E.g. standardize create-if-not-exist, even if S3 doesn't implement that.

mdaniel 2y ago
My life experience has been that this line of thinking results in two separate outcomes: "the great thing about standards is that there are so many to choose from" <https://xkcd.com/927/ https://xkcd.com/927/>, and that publishing a standard does absolutely zero for getting buy-in, with that last one made worse by any short-sighted fools who publish a bunch of words without a reference impl or ideally a TCK. As this whole thread has shown, for 10 people there will be 15 understandings of any given sentence, which is very very bad when trying to get two computers to agree on something

unethical_ban 2y ago

>Schrodiner’s cat is the one that’s both alive and de-lifed at the same time, right? It was alive and dead at the same time. Don't use censor-appeoved language outside Tiktok!

coolgoose 2y ago

A lot of them are interesting points, but I am not sure I agree with the complaint the file system is case sensitive. That's how it should be and I am annoyed at macos for not having it.

yencabulator 2y ago
It would be incredibly weird not to be able to store urlsafe-base64 encoded paths on S3.
ansgri 2y ago
TIL. I've switched to Mac about a year ago and it's sufficiently Linux-like to take such things for granted. I wonder what other surprises are going to bite me in the future.
- aldanor 2y ago
  Sufficiently BSD-like :)
sholladay 2y ago
macOS is case preserving, though. To me, it’s the best of both worlds. You can stylize your file names and they will be respected, but you don’t have to remember how they are stylized when you are searching for or processing them because search is case insensitive.
- smt88 2y ago
  Windows is also case-insensitive but case-preserving
- self_awareness 2y ago
  Maybe macOS is case-preserving, but it's not encoding-preserving. If you create a file using a composed UTF-8 "A", the filesystem layer will decompose the string to another form, and create the filename using that decomposed form "B". Of course, "A" and "B" when compared can be completely different (even when compared using case insensitivity enabled), yet will point to the same file. More info here: https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/ https://eclecticlight.co/2021/05/08/explainer-unicode-normal...

willsoon 2y ago

My wife - she's a future developer, I'm a lawyer - has had enough of me explaining some of these details in the form of concerns. I am somewhat satisfied to know that my perplexities are not imaginary.

trallnag 2y ago
What's a future developer?
- cynicalsecurity 2y ago
  A developer that is learning programming or a specific technology? Isn't it self-explanatory or what problem did you have understanding it?
  - krisoft 2y ago
    > Isn't it self-explanatory No. > what problem did you have understanding it? It is simply not a phrase I commonly read. When I google it I find a visa consultancy under the name, and not much else. Curiously your comment is also among the results. The problem is that the phrase "<something> developer" is used to describe what the person develops. A "real-estate developer" invests in real estate as a business. A "web developer" develops web applications, a "game developer" develops games, and so on and so on. So reading the word I immediately thought they mean someone who is developing the future? Like idk Douglas Engelbart or Tim Berners-Lee or someone like that. If you want to write that someone is learning to become a developer I would recommend the much less confusing "developer in training" phrase, or even better if you just write "they are learning to become a developer".
    - willsoon 2y ago
      Bad English case. It just is. I pay a lot of attention when I write English, but you can always tell when someone isn't a native English speaker in about two words. Je suis désolé.

isoprophlex 2y ago

That bit on failed multipart uploads invisibly sticking around (and incurring storage cost, unless you explicitly specify some lifecycle magic)... just ugh. And I thought one of the S-es was for 'simple'.

teitoklien 2y ago
That S for simple stands for simple ways to skyrocket your expenses
e____g 2y ago
Yes, that sucks. Blame ahenry@, then GM for S3. My proposal was that parts of incomplete uploads would stick around for only 24 hours after the most recent activity on the upload, and you wouldn't be charged for storage during that time. ahenry@ vetoed that.
- vbezhenar 2y ago
  Why would you propose something that makes company earn less money? I'm sure that at Amazon scale, this misfeature earned millions of dollars.
  - spacebanana7 2y ago
    Customer relationships. I recall a Bezos quote along the lines of "It's better to lose a refund than to lose a customer".
Hamuko 2y ago
Yeah, I've definitely treaded on the storage cost landmine. Thankfully it was just some cents in my case, but it's really infuriating how badly the console exposes the information.
scosman 2y ago
“Simple” was coined when the alternative was managing a fleet of servers with disks . Time changes everything.

tedunangst 2y ago

[flagged]

kthielen 2y ago
A career change, or a job change?

nh2 2y ago

How about the fact that S3 is not suitable for web serving due to high latencies (in standard storage classes)? Many people think you can just host the resources for your websites, such as images or fonts, straight on S3. But that can make for a shitty experience: > applications can achieve consistent small object latencies (and first-byte-out latencies for larger objects) of roughly 100–200 milliseconds. From: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi...

metadat 2y ago
Put a memcached instance in front and you're good.
lulznews 2y ago
Pretty much everyone knows this …
teitoklien 2y ago
Most folks use S3 as a source for AWS cloudfront for serving content You can even use cloudfront signed cookies to give specific users cdn access to only specific content owned by them on S3. How cool is that.
WatchDog 2y ago
Typically you would use cloudfront with S3 if you want to use it for serving web assets. It will cache frequently accessed assets, and in addition to reducing latency may reduce cost quite a bit.
mcqueenjordan 2y ago
s3 is not optimized to directly serve websites, but to durably store and retrieve ~unlimited data.
weird-eye-issue 2y ago
This has been well known for over a decade

reddalo 2y ago

I can't trust myself using S3 (or any other AWS service). Nothing is straightforward, there are too many things going on, too much documentation that I should read, and even then (as OP shows) I may be accidentally and unknowingly expose everything to the world. I think I'll stick to actually simple services, such as Hetzner Storage Boxes or DigitalOcean Spaces.

chamomeal 2y ago
I like digital ocean spaces, but it has its own annoying quirks. Like I recently found out yif you pipe a video file larger than a few MB, it’ll drop the https:// https:// from the returned Location. So on every file upload I have to check if the location starts with https, and add it on if it’s not there. Of course the S3 node client GitHub issue says “sounds like a digital ocean bug”, and the digital ocean forums say “sounds like an S3 node client bug” lol
slig 2y ago
The way that DO handles secrets should scare anyone. Did you know that if you use their Container Registry and set it up so that your K8S has automatically access to it, their service will create a secret that has full access to your Spaces?
- marcosdumay 2y ago
  Hum... Kubernetes is not on the GP's list...
  - slig 2y ago
    Fair enough, but not having scoped secrets is a red flag.
maximinus_thrax 2y ago
> Nothing is straightforward, there are too many things going on, too much documentation that I should read, and even then (as OP shows) I may be accidentally and unknowingly expose everything to the world. I took a break from cloud development for a couple of years (working mostly on client stuff) and just recently got back. I am shocked at the amount of complexity built over the years along with the cognitive load required for someone to build an ironclad solution in the public cloud. So many features and quirks which were originally designed to help some fringe scenario are now part of the regular protocol, so that the business makes sure nobody is turned away.

CaliforniaKarl 2y ago

> S3 buckets are the S3 API > … a relatively small part of the API requires HTTP requests to be sent to generic S3 endpoints (such as s3.us-east-2.amazonaws.com), while the vast majority of requests must be sent to the URL of a target bucket. I believe this is talking about virtual-hosted style and path-style methods for accessing the S3 API. From what I can see [0], at least for the REST API, the entire API works either with virtual-hosted style (where the bucket name is in the host part of the URL) and path-style (where the bucket name is in the path part of the URL). Amazon has been wanting folks to move over to the virtual-hosted style for a long time, but (as of 3+ years ago) the deprecation of path-style has been delayed[1]. This deprecation of path-style requests has been extremely important for products implementing the S3 API. For example… * MinIO uses path-style requests by default, requiring you set a configuration variable[2] (and set up DNS appropriately) to handle the virtual-hosted style. * Wasabi supports both path-style and virtual-hosted style, but "Wasabi recommends using path-style requests as shown in all examples in this guide (for example, http://s3.wasabisys.com/my-bucket/my-object http://s3.wasabisys.com/my-bucket/my-object) because the path-style offers the greatest flexibility in bucket names, avoiding domain name issues."[3]. Now here's the really annoying part: The REST API examples show virtual-hosting style, but path style works too! For example, take the GetBucketTagging example. Let's say you have bucket "karl123456" in region US-West-2. The example would have you do this: GET /?tagging HTTP/1.1 Host: karl123456.s3.amazonaws.com But instead, you can do this: GET /karl123456?tagging HTTP/1.1 Host: s3.us-west-2.amazonaws.com !! How do I know this? I tried it! I constructed a `curl` command do to the path-style request, and it worked! (I didn't use "karl123456", though.) So hopefully that helps resolve at least one of your S3 annoyances :-) [0]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAPI.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAP... [1]: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/ https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-... [2]: https://min.io/docs/minio/linux/reference/minio-server/settings/core.html#envvar.MINIO_DOMAIN https://min.io/docs/minio/linux/reference/minio-server/setti... [3]: https://docs.wasabi.com/docs/rest-api-introduction https://docs.wasabi.com/docs/rest-api-introduction

prpl 2y ago

Not one thing about different API limits for operations according to key prefixes, and how you need to contact support if you need partitioning and provide them the prefixes, huh?

pachico 2y ago

I have the feeling that the entire case (in)-sensitive discussions are usually too much English-centric. Allow me to iterate: I have the feeling way too many language discussions, especially in IT, are too much English-centric.

reddalo 2y ago
Speaking of non-English cultures, do Japanese case insensitive systems differentiate between hiragana and katakana? Because, in some ways, the two syllabaries remind me of uppercase and lowercase alphabets.
- thaumasiotes 2y ago
  They're more like distinguishing between "o" and “ℴ”. Which is where the European idea of "capital letters" originates, but not how we think about them today.
- presentation 2y ago
  A lot of the time forms will specifically ask for hiragana or katakana, or specify full width or half width characters. But basically it’s a mess there too
teitoklien 2y ago
> too much English-centric. Pretty glad about it considering how much more simpler ASCII was to work with compared to Unicode. I say it as a non native english speaker, programming has so many concepts and stuff already, its best not to make it more complex by adding a 101 different languages to account for. Unicode and Timezone, the two things that try to bring more languages and cultures to be accounted for while programming and look what happens, it creates the most amount of pain for everyone including non native english programmers. I dont want to write computer programs in my non-english native tongue, if that means i’ll have to start accounting for every major language while im programming. Its fine that IT discussions are so English-centric. Diversity is more complexity, and no one owns the english language, its just a tool used by people to communicate, thanks to that universal language, I can express my thoughts to most people in India, China, Japan, South America, etc due to having 1 common language. They all own the english language too, the moment they decided to speak in it. No need to bring diversity politics in IT. Best to keep it technical.

dale_glass 2y ago

The case sensitivity one is easy, here's a thing that's more likely to be entirely unintuitive: S3 paths are fake. Yes, it accepts uploads to "/builds/1/installer.exe", and yes, you can list what's in /builds, but all of that is a simulation. What you actually did was to upload a file literally named '/builds/1/installer.exe' with the '/' as part of the name. So, "/builds/1//installer.exe" and "/builds//1/installer.exe" are also possible to upload and entirely different files. Because it's the name of a key, there's no actual directories.

hnlmorg 2y ago
Yeah, the prefix thing is a source of so many bugs. I get why AWS took that approach and it’s actually a really smart approach but it still catches so many developers out. Just this year our production system was hit by a weird bug that took 5 people to find. Turned out the issue was an object literally just named “/“ and the software was trying to treat it as a path rather than a file.
- BirAdam 2y ago
  Happened at my job yesterday.
reddalo 2y ago
You're right, unless you use the new S3 "Directory buckets" [1], which make the entire thing even more confusing! [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/direct...
- temac 2y ago
  and which also according to the command docs i read recently add random limitations to maybe half of the S3 API I'm not sure reusing (parts of) the protocol was a good idea given unrelated half of it was already legacy and discouraged (e.g. the insane permission model with policies and acls) or even when not, let's say... weird and messy.
2y ago

xarope 2y ago

> There are only losers in this game, but at least we’ve all got a participation ribbon to comfort us in moments of angst. I sense the frustration...

swiftcoder 2y ago

> S3 isn’t the only service that works this way, Hosted Cognito UI endpoints do something similar (https://[your-user-pool-domain]/login https://[your-user-pool-domain]/login). Basically every new service that expects to deal with a lot of traffic should work this way (we did this also for AWS IoT). It's a hell of a lot easier to deal with load balancing and request routing if you can segment resources starting at the DNS level...

phrotoma 2y ago

Here's another fun one that took a colleague and I several days of analysis to diagnose. S3 will silently drop all requests once a single TCP connection has sent 100 HTTP requests. https://github.com/aws/aws-sdk-go/issues/2825 https://github.com/aws/aws-sdk-go/issues/2825

seabrookmx 2y ago
It doesn't silently drop, it sends a header indicating it's closed the TCP connection. This is a pretty common pattern whereby you want keep-alive.for performance reasons, but you don't want clients running _too long_ creating hot spots on your load balancers.
- phrotoma 2y ago
  What header? Our client was envoy and it's pretty standards compliant, and it just kept trying to use the connection. Edit: I see that it is `connection: close` I wonder if that is new behaviour or if envoy did not honour it at the time we encountered the issue. Thanks for the info!
- temac 2y ago
  Instead of closing the connection it sends a message stating that it is closed? Wow.

aao 2y ago

Related: https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1 https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-c...

lotsofspots 2y ago
About a week after that went viral - https://aws.amazon.com/about-aws/whats-new/2024/05/amazon-s3-no-charge-http-error-codes/ https://aws.amazon.com/about-aws/whats-new/2024/05/amazon-s3...

croes 2y ago

>Things you wish you didn't need to know about S3 >A time travel paradox in the title is a good place to start a blog post, don’t you think? Where is the paradox?

buggythebug 2y ago

Audi S3?

scosman 2y ago

Here is a good one: deleting billions of objects can be expensive if you call delete APIs. However you can set a wildcard or bucket wide object expiry of time=now for free. You’ll immediately stop being charged for storage, and AWS will manage making sure everything is deleted.

est31 2y ago
> You’ll immediately stop being charged for storage The effect of lifecycle rules is not immediate: they get applied in a once-per-day batch job, so the removal is not immediate.
- electroly 2y ago
  That's true but OP clearly knows that already. You stop getting charged for storage as soon as the object is marked for expiration, not when the object is finally removed at AWS's leisure. You can check the expiration status of objects in the metadata panel of the S3 web console. https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy... > There may be a delay between the expiration date and the date at which Amazon S3 removes an object. You are not charged for expiration or the storage time associated with an object that has expired.
ericpauley 2y ago
Nit: the delete call is free, it’s the list call to get the objects that costs money. In theory if you know what objects you have from another source it’s free.
bushbaba 2y ago
Because AWS gets to choose when the actual deletes happen. The metadata is marked as object deleted but AWS can process the delete off peak. It also avoids the s3 api server being hammered with qps
- yencabulator 2y ago
  An explicit delete operation can mark metadata the same way. The real difference is likely more along the lines of LSM compaction, with expiry they likely use a moral equivalent of https://github.com/facebook/rocksdb/wiki/Compaction-Filter https://github.com/facebook/rocksdb/wiki/Compaction-Filter to actually do the deletion.

julik 2y ago

A few more: * Multipart uploads cannot be performed from multiple machines having instance credentials (as the principal will be different and they don't have access to each other's multipart uploads). You need an actual IAM user if you want to assemble a multipart upload from multiple machines. * LIST requests are not only slow, but also very expensive if done in large numbers. There are workarounds ("bucket inventory") but they are neither convenient nor cheap * Bucket creation is not read-after-write consistent, because it uses DNS under the hood. So it is possible that you can't access a bucket right after creating it, or that you can't delete a bucket you just created until you waited enough for the changes to propagate. See https://github.com/julik/talks/blob/master/euruko-2019-no-such-bucket/no-such-bucket.pdf https://github.com/julik/talks/blob/master/euruko-2019-no-su... * You can create an object called "foo" and an object called "foo/bar". This will make the data in your bucket unportable into a filesystem structure (it will be a file clobbering a directory) * S3 is case-sensitive, meaning that you can create objects which will unportable into a filesystem structure (Rails file storage assumed a case-sensitive storage system, which made it break badly on macOS - this was fixed by always using lowercase identifiers) * Most S3 configurations will allow GETs, but will not allow HEADs. Apparently this is their way to prevent probing for object existence, I am not sure. Either way - cache-honoring flows involving, say, a HEAD request to determine how large an object is will not work (with presigned URLs for sure!). You have to work around this doing a GET with a Range: of "very small" (say, the first byte only) * If you do a lot of operations using pre-signed URLs, it is likely you can speed up the generation of these URLs by a factor of 10x-40x (see https://github.com/WeTransfer/wt_s3_signer https://github.com/WeTransfer/wt_s3_signer) * You still pay for storage of unfinished multipart uploads. If you are not careful and, say, these uploads can be initiated by users, you will be paying for storing them - there is a setting for deleting unfinished MP uploads automatically after some time. Do enable it if you don't want to have a bad time. These just off the top of my head :-) Paradoxically, S3 used to be revolutionaly and still is, onl multiple levels, a great products. But: plenty features, plenty caveats.

dugmartin 2y ago
The one that caught me a couple of weeks ago is multipart uploads have a minimum initial chunk size of 5 MiBs (https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts...). I built a streaming CSV post-processing pipeline in Elixir that uses Stream.transform (https://hexdocs.pm/elixir/Stream.html#transform/3 https://hexdocs.pm/elixir/Stream.html#transform/3) to modify and inject columns. The Elixir AWS and CSV modules handle streaming data in but the AWS module throws an error (from S3) if you stream "out" that totals less than 5 MiBs as is uses multi-part uploads which made me sad.

perpil 2y ago

A workaround for some of the limits of presigned urls like not being able to specify a max file size is to front your uploads using CloudFront OAC and CloudFront Functions. It costs more (.02/GB) but you can run a little JavaScript code to validate/augment headers between your user and S3 and you don't need to expose your bucket name. https://speedrun.nobackspacecrew.com/blog/2024/05/22/using-cloudfront-as-a-lightweight-proxy.html https://speedrun.nobackspacecrew.com/blog/2024/05/22/using-c...

jimbobthrowawy 2y ago

Those uploader decides rules are wild. Does that mean someone with a website poorly configured enough can have user content uploaded to, and later served from amazon glacier? (assuming a sufficiently motivated user)

wmfiv 2y ago
It does. But if you're concerend about this (and many of the other items mentioned), you can control access to those features using IAM. https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazons3.html https://docs.aws.amazon.com/service-authorization/latest/ref... The condition keys specifically are here and you can see keys to control access to storage class, tagging, etc. https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazons3.html#amazons3-policy-keys https://docs.aws.amazon.com/service-authorization/latest/ref...

gymbeaux 2y ago

No mention of how AWS/S3 approximates the size of a file to save CPU cycles. It used to drive me up a wall seeing S3 show a file size as slightly different than what it was. If I recall correctly, S3 uses 1000 rather than 1024 to convert bytes to KB and MB and GB. This saves CPU cycles but results in “rounding errors” with the file’s reported size. It’s discussed here, although they’re talking about it being “CLI vs console” which may be true? https://stackoverflow.com/questions/57201659/s3-bucket-size-differs-while-using-console-and-cli https://stackoverflow.com/questions/57201659/s3-bucket-size-...

abadpoli 2y ago
This almost certainly has nothing to do with “saving CPU cycles” and is most likely just that whoever created the Cloudwatch console used the same rounding that is used for all other metrics in CW, rather than the proper calculation for disk size, and it was a small enough issue that it was never caught until it was too late to change it because changing it would disrupt the customers that have gotten used to it.
- presentation 2y ago
  If anything a bit shift could do the 1024 division faster
DougBTX 2y ago
S3 doesn't format the size for human display, the object's Size is returned in bytes: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.h...
- gymbeaux 2y ago
  What I meant to say was S3 uses 2^20 to convert from bytes to kilobytes rather than 10^6 (e.g. what Windows uses). https://repost.aws/knowledge-center/s3-console-metric-discrepancy https://repost.aws/knowledge-center/s3-console-metric-discre... > 3 console and Storage Lens use base 2 conversion (/1024) to report storage metrics, and CloudWatch by default uses base 10 conversion (/1000)

anonymouse008 2y ago

I’ve become old. If you look at these things in disgust (ACL vs Policies, Delete bucket with s3:*, etc), you’re missing the point of (deterministic ?) software. It does what it is written to do, faithfully, and error out when not. When it doesn’t do as written or as documented, then yes… go full bore.

temac 2y ago
The doc is huge, and the principle of least astonishment is often not respected. Also third parties providers support a random subset of it given the protocol has nothing simple anymore (or maybe never had)