11 ms·

In S3 simplicity is table stakes

200 points by riv991 2y ago

rook1 2y ago
"I think one thing that we’ve learned from the work on Tables is that it’s these _properties of storage_ that really define S3 much more than the object API itself." Between the above philosophy, S3 Tables, and Express One Zone (SSD perf), it makes me really curious about what other storage modalities S3 moves towards supporting going forward.
neom 2y ago
little history: When we were getting ready to do an API at DigitalOcean I got asked "uhm... how should it feel?" I thought about that for about 11 seconds and said "if all our APIs feel as good as S3, it should be fine" - it's a good API.
- flessner 2y ago
  The API, absolutely. It's only sad that the SDKs are often on the heavy side, I remember that the npm package used to be multiple megabytes as it bundled large parts of the core AWS SDK. Nowadays, I believe it's still around half a megabyte.
  - malfist 2y ago
    I don't know if the npm is the same way, but the java sdk now has options by service. So you can include just s3 instead of all of aws
  - rglover 2y ago
    The footprint of the JS SDK is much better as they split all of it into service-specific packages, but the SDK APIs are a bit confusing (everything is a class that has to be instantiated—even config).
greatpostman 2y ago
I can guarantee you, nothing is simple about S3. I bet engineers spend months on a single configuration change to the underlying subsystems.
- jacksnipe 2y ago
  That’s what makes it a good abstraction.
- ecshafer 2y ago
  If you read the article, they say that exactly. Its by Dr Werner Vogels, they know exactly what goes into S3, since they are a principal engineer on the project.
  - apetresc 2y ago
    It’s Vogels’ blog but this is a guest post by Andy Warfield.
    - ecshafer 2y ago
      My mistake, I stand corrected.
ginko 2y ago
Would have been good if they mentioned they meant Amazon S3. It took me a while to figure out what this was about. Initially I thought this was about S3 standby mode.
- deleted 2y ago
  [deleted]
- PantaloonFlames 2y ago
  All things distributed - the source of the headline - is a blog written by Werner Vogels, CTO of Amazon.
  - ginko 2y ago
    Only people in the web dev niche would know this.
waiwai933 2y ago
> I’ve seen other examples where customers guess at new APIs they hope that S3 will launch, and have scripts that run in the background probing them for years! When we launch new features that introduce new REST verbs, we typically have a dashboard to report the call frequency of requests to it, and it’s often the case that the team is surprised that the dashboard starts posting traffic as soon as it’s up, even before the feature launches, and they discover that it’s exactly these customer probes, guessing at a new feature. This surprises me; has anyone done something similar and benefitted from it? It's the sort of thing where I feel like you'd maybe get a result 1% of the time if that, and then only years later when everyone has moved on from the problem they were facing at the time...
- easton 2y ago
  Maybe this is a faster way of getting AWS feature requests heard. I'm going to write a script that keeps trying to call ecs:MakeFargateCacheImages.
  - ajb 2y ago
    It could also be hackers, as when a new service launches is exactly when it will be most buggy. And the contents of S3 are a big payoff.
_1tem 2y ago
I have a feeling that economies of scale have a point of diminishing returns. At what point does it become more costly and complicated to store your data on S3 versus just maintaining a server with RAID disks somewhere? S3 is an engineering marvel, but it's an insanely complicated backend architecture just to store some files.
- Tanjreeve 2y ago
  Probably never. The complexity is borne by Amazon. Even before any of the development begins if you want a RAID setup with some sort of decent availability you've already multiplied your server costs by the number of replicas you'd need. It's a Sisyphean task that also has little value for most people. Much like twitter it's conceptually simple but it's a hard problem to solve at any scale beyond a toy.
- sjsdaiuasgdia 2y ago
  That's going to depend a lot on what your needs are, particularly in terms of redundancy and durability. S3 takes care of a lot of that for you. One server with a RAID array can survive, usually, 1 or maybe 2 drive failures. The remaining drives in the array will have to do more work when a failed drive is replaced and data is copied to the new array member. This sometimes leads to additional failures before replacement completes, because all the drives in the array are probably all the same model bought at the same time and thus have similar manufacturing quality and materials. This is part of why it's generally said that RAID != backup. You can make a local backup to something like another server with its own storage, external drives, or tape storage. Capacity, recovery time, and cost varies a lot across the available options here. Now you're protected against the original server failing, but you're not protected against location-based impacts - power/network outages, weather damage/flooding, fire, etc. You can make a remote backup. That can be in a location you own / control, or you can pay someone else to use their storage. Each layer of redundancy adds cost and complexity. AWS says they can guarantee 99.999999999% durability and 99.99% availability. You can absolutely design your own system that meets those thresholds, but that is far beyond what one server with a RAID array can do.
  - 2y ago
_fat_santa 2y ago
S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system. One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 10,000 years. Since then anytime I talk about S3's durability I bring up that example and it always seems to convey the point for a layperson. And it's "simplicity" is truly elegant. When I first started using S3 I thought of it as a dumb storage location but as I learned more I realized that it had some wild features that they all managed to hide properly so when you start it looks like a simple system but you can gradually get deeper and deeper until your S3 bucket is doing some pretty sophisticated stuff. Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors.
- victorp13 2y ago
  > Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors. Counter-point: You know that you're the dominant player. See: .psd, .pdf, .xslx. Not particularly good file types, yet widely supported by competitor products.
  - pavlov 2y ago
    Photoshop, PDF and Excel are all products that were truly much better than their competitors at the time of their introduction. Every file format accumulates cruft over thirty years, especially when you have hundreds of millions of users and you have to expand the product for use cases the original developers never imagined. But that doesn’t mean the success wasn’t justified.
    - jalk 2y ago
      PDF is not a product. I get what you are day but I can’t say that I’ve ever liked Adobe Acrobat
conorjh 2y ago
tf is that title supposed to mean?
- dgfitz 2y ago
  Somehow gambling is directly correlated to using S3? That’s the best I got.
- MillironX 2y ago
  Table stakes is a poker term meaning the absolute minimum amount you are allowed to bet. So the title translates to "In S3, simplicity is the bare minimum" or "In S3, simplicity is so important that if we didn't have it, we might as well not even have S3."
  - MatthewCampbell 2y ago
    It's a risky idiom in general because it's often used to prevent debate. "Every existing product has feature X, so feature X is table stakes." "Why are we testing whether we really need this feature, it's table stakes!" My observation has been that "table stakes" features are often the best ones to reject. (Not so in the case of this title, though)
deleted 2y ago
[deleted]
alberth 2y ago
S3 is the simplest CRUD app you could create. It's essentially just the 4 functions of C.R.U.D done to a file. Most problems in tech are not that simple. Note: not knocking the service. just pointing out not all things are so inherently basic (and valuable at the same time).
- riv991 2y ago
  Isn't that the most impressive part? That the abstraction makes it seem so simple
- bdcravens 2y ago
  That's the public interface. The underlying architecture is where the power is.
- cruffle_duffle 2y ago
  That’s when you really know you hid all the complexity well. When people call your globally replicated data store with granular permissions, sophisticated data retention policies, versioning, and manage to have, what, seven (ten?) nines or something, “simple”. No problem. I’m sure ChatGPT could cook up a replacement in a weekend. Like Dropbox it’s just rsync with some scripts that glue it together. How hard could it possibly be? I mean people serve entire websites right out of s3 buckets. Using it as a crude CDN of sorts. It’s a modern marvel.
  - sebastiansm 2y ago
    I could build a netflix in a weekend.
- golly_ned 2y ago
  A file system is simple. Open, read, close. Most tech problems are not that simple. How hard could a filesystem be?
  - dekhn 2y ago
    Locking, checks after unclean shutdown, sparse files, high performance, reliabilty.... are all things that make filesystems harder.
robertwebb 2y ago
[dead]
arnath 2y ago
I found out last year that you can actually run a full SPA using S3 and a CDN. It’s kind of a nuts platform
- ellisv 2y ago
  I use S3+Cloudfront for static sites and Cloudflare workers if it needed. It's always crazy to me that people will run a could be static site on Netlify/Vercel/etc.
  - Cthulhu_ 2y ago
    We've used Netlify at previous projects, we used it because it was easy. No AWS accounts or knowledge needed, just push to master, let the CI build (it was a Gatsby site) and it was live.
    - ellisv 2y ago
      I think Netlify is great but to me it's overkill if you just have a static site. I understand that Netlify is much simpler to get started with and setting up an AWS account is somewhat more complex. If you have several sites, it's worth spending the time to learn.
- jorams 2y ago
  Since everything you need to run "a full SPA" is to serve some static files over an internet connection I'm not sure how that tells you anything interesting about the platform. It's basically the simplest thing a web server can do.
user9999999999 2y ago
if only metadata could be queried without processing a csv output file first, imagine storing thumbnails in there even! copied objects had actual events, not something you have to dig cloudtrail for, you could get last update time from a bucket to make caching easier
- huntaub 2y ago
  Are you talking about getting metadata from many objects in the bucket simultaneously? You might be interested in S3 Metadata https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingM....
- aeyes 2y ago
  When S3 Tables launched they made the Metadata available using this technology. So you can query it like an Apache Iceberg table. https://aws.amazon.com/blogs/aws/introducing-queryable-object-metadata-for-amazon-s3-buckets-preview/ https://aws.amazon.com/blogs/aws/introducing-queryable-objec...
- user9999999999 2y ago
  whoops. I was wrong! you can store base64 encoded metadata in the object then run a HEAD request to get it, but its limited to 2kb. also, you ~can~ query the metadata, but its latency is more suited to batch processing and not lambdas
chasd00 2y ago
S3 was one of the first offerings coming out of AWS right? It’s pretty legendary and a great concept to begin with. You can tell by how much sense it makes and then trying to wrap your ahead around the web dev world pre-S3.
- Cthulhu_ 2y ago
  ? The web dev world pre-S3 was pretty much the same, but you stored your files on a regular server (and set up your own redundancy and backup strategy). Not that much different to be honest from an end user's point of view.
  - kevindamm 2y ago
    At a lot of places there wasn't even a redundancy nor backup strategy, so it really was just as simple as registering with a hosting company and ssh+ftp (or cPanel or something like that for what amounted to the managed solutions of the time). I agree, things before S3 weren't really that different. LAMP stacks everywhere, and developer skills were very portable between different deployments of these LAMP stacks. Single machines didn't scale as much then, but for most small-medium sites they really didn't need to.
ipsento606 2y ago
Lots of comments here talking about how great S3 is. Anyone willing to give a cliff notes about what's good about it? I've been running various sites and apps for a decade, but have never touched S3 because the bandwidth costs are 1, sometimes 2 orders of magnitude more expensive than other static hosting solutions.
- waiwai933 2y ago
  S3 is great for being able to stick files somewhere and not have to think about any of the surrounding infrastructure on an ongoing basis [1]. You don't have to worry about keeping a RAID server, swapping out disks when one fails, etc. For static hosting, it's fine, but as you say, it's not necessarily the cheapest, though you can bring the cost down by sticking a CDN (Cloudflare/CloudFront) in front of it. There are other use cases where it really shines though. [1]: I say ongoing basis because you will need to figure out your security controls, etc. at the beginning so it's not totally no-thought.
- jjice 2y ago
  S3 has often fallen into a "catch all" solution for me whenever I need to store data large enough that I don't want to keep it in a database (RDBMS or Redis). Need to save a file somewhere? Dump it in S3. It's generally affordable (obviously dependent on scale and use), fast, easy, and super configurable. Being able to expose something to the outside, or with a presigned URL is a huge advantage as well. Off the top of my head, I think of application storage generally in this tier ordering (just off the top of my head based on the past few years of software development, no real deep thought here): 1. General application data that needs to be read, written, and related - RDBMS 2. Application data that needs to be read and written fast, no relations - Redis 3. Application data that is mostly stored and read - S3 Replace any of those with an equivalent storage layer.
- ak217 2y ago
  One underappreciated feature of S3 - that allowed it to excel in workloads like the Tables feature described in the article - is that it's able to function as the world's highest throughout network filesystem. And you don't have to do anything to configure it (as the article points out). By storing data on S3, you get to access the full cross-sectional bandwidth of EC2, which is colossal. For effectively all workloads, you will max out your network connection before S3's. This enables workloads that can't scale anywhere else. Things like data pipelines generating unplanned hundred-terabit-per-second traffic spikes with hotspots that would crash any filesystem cluster I've ever seen. And you don't have to pay a lot for it- once you're done using the bandwidth, you can archive the data elsewhere or delete it.
neerajk 2y ago
[flagged]
bob1029 2y ago
I really enjoy using S3 to serve arbitrary blobs. It perfectly solves the problem space for my use cases. I avoid getting tangled in authentication mess by simply naming my files using type4 GUIDs and dumping them in public buckets. The file name is effectively the authentication token and expiration policies are used to deal with the edges. This has been useful for problems like emailing customers gigantic reports and transferring build artifacts between systems. Having a stable URL that "just works" everywhere easily pays for the S3 bill in terms of time & frustration saved.
- bityard 2y ago
  My favorite use case for S3 API-compatible solutions: I often run into systems that generate lots of arbitrary data that only have temporary importance. A common example might be intermediate build artifacts, or testing ephemera (browser screenshots, etc). Things that are needed for X number of months and then just need to disappear. Yeah, we can dump those to a filesystem. But then we have to ask which filesystem? What should the directory layout be? If there are millions or billions of objects, walking the whole tree gets expensive. Do we write a script to clean everything up? Run it via cron or some other job runner? With S3, you just write your artifact to S3 with a TTL and it gets deleted automagically when it should. No cron jobs, no walking the whole tree. And you can set up other lifecycle options if you need it moved to other (cheaper) storage later on, backups, versioning, and whatever else. For on-prem, you have Minio, Garage, or SeaweedFS. These are pretty nice to deploy the servers however you need for the level of reliability/durability you require.
bentobean 2y ago
It's funny—S3 started as a "simple" storage service, and now it's handling entire table abstractions. Reminds me how SQL was declared dead every few years, yet here we are, building ever more complex data solutions on top of supposedly simple foundations.
- imiric 2y ago
  I instinctively distrust any software or protocol that implies it is "simple" in its name: SNMP, SMTP, TFTP, SQS, etc. They're usually the cause of an equal or more amount of headaches than alternatives. Maybe such solutions are a reaction to previous more "complex" solutions, and they do indeed start simple, but inevitably get swallowed by the complexity monster with age.
  - great_wubwub 2y ago
    TFTP is probably the exception to that rule. All the other protocols started out easily enough and added more and more cruft. TFTP stayed the way it's always been - minimalist, terrifyingly awful at most things, handy for a few corner cases. If you know when to use it and when to use something like SCP, you're golden. If TFTP had gone the way of SNMP, we'd have 'tftp <src> <dest> --proto tcp --tls --retries 8 --log-type json' or some horrendous mess like that.
    - yencabulator 2y ago
      TFTP's usefulness in the modern day is strictly for things that don't have a TCP stack. Anything with a TCP stack is better off with HTTP. That doesn't leave much on the table except legacy & inertia.
paulddraper 2y ago
> In S3 simplicity is table stakes The S3 API is not "simple." Authentication being a good part of that.
brikym 2y ago
I'm curious about S3 tables. Azure has had tables in their storage account product for years. What are the differences?
diroussel 2y ago
It's great that they added iceberg support I guess, but it's a shame that they also removed S3 Select. S3 Select wasn't perfect. For instance the performance was no where near as good as using DuckDB to scan a parquet file, since duck is smart, and S3 Select does a full table scan. But S3 Select is nearly way cheaper that the new iceberg support. So if your needs are only for reading one parquet snapshot, we no need to do updates, then this change is not welcome. Great article though, and I was pleased to see this at the end: > We’ve invested in a collaboration with DuckDB to accelerate Iceberg support in Duck,
StratusBen 2y ago
For those interested in S3 Tables which is referenced in this blog post, we literally just published this overview on what they are and cost considerations of them that people might find interesting: https://www.vantage.sh/blog/amazon-s3-tables https://www.vantage.sh/blog/amazon-s3-tables
- 1a527dd5 2y ago
  https://www.vantage.sh/blog/amazon-s3-tables#s3-tables-cost https://www.vantage.sh/blog/amazon-s3-tables#s3-tables-cost I can't make head or tails of the beginning of this sentence:- > Pricing for S3 Tables is all and all not bad. Otherwise lovely article!
  - shawabawa3 2y ago
    "all and all" is a typo for "all in all" which means "overall", or "taking everything into consideration" So they are saying the pricing is not bad considering everything it does
- dangoodmanUT 2y ago
  > There are 1 million PUT requests and 10 million GET requests that month > + 1,000,000 GET requests x ($0.004 / 1,000 requests) = $9
CobrastanJorji 2y ago
> When we moved S3 to a strong consistency model, the customer reception was stronger than any of us expected. This feels like one of those Apple-like stories about inventing and discovering an amazing, brand new feature that delighted customers but not mentioning the motivating factor of the competing products that already had it. A more honest sentence might have been "After years of customers complaining that the other major cloud storage providers had strong consistency models, customers were relieved when we finally joined the party."
- progbits 2y ago
  I mainly use GCP but keep hearing how great AWS is in comparison. Imagine my surprise when porting some GCS code to S3 last year and realizing there is no way to get consistency guarantees without external lock service.
  - klysm 2y ago
    Distributed locks are a lie
  - snoman 2y ago
    That would surprise me too considering read-after-write consistency came to S3 like 5 years ago?
    - jchw 2y ago
      IIRC (it has been a while) the difference is that on Amazon it can only be consistent within a region whereas on GCS I believe even multi-region buckets offer strong consistency.
      - snoman 2y ago
        Ahh. That makes sense!
  - thiht 2y ago
    > I mainly use GCP but keep hearing how great AWS is in comparison. Where do you keep hearing this? Having used both, AWS is trash in comparison. It's way to complicated to do anything simple. At work I wish we could migrate to GCP (or just something that's not AWS, really).
lizknope 2y ago
Clicked on the article thinking it was about S3 Graphics, the company that made the graphics chip in my first PC. Now I see it's some amazon cloud storage thing.
Kab1r 2y ago
> S3 launched as the first public AWS service. Didn't SQS launch publicly earlier than S3?
- deanCommie 2y ago
  SQS went into beta first, S3 went "GA" first. AWS typically considers the "GA" milestone as the "public launch" date, which is silly because the criteria for what is good enough for GA has changed over the years.
CaffeineLD50 2y ago
I wish they'd comment on how their service keeps leaking massive amounts of data that isn't supposed to. Didn't Microsoft blame their customers for not running updates and using AV? Gosh I guess MS did a fine job. Customers were to blame. Just like those s3 buckets. Customers ignored the warnings ...
- snoman 2y ago
  Not sure what you’re getting at. An S3 bucket out-of-the-box is secure and can’t leak/be accessed publicly. There’s legitimate use cases for making a bucket public. What are you advocating for?
  - CaffeineLD50 2y ago
    A good security model doesn't let stupid people do stupid things with warnings. Stupid people ignore warnings. A good model protects stupid people from themselves. A password model that says "you MUST have a strong password" protects stupid people from their own stupidity. A model that says you can use bad passwords if you click through warnings is a shite model. Stupid people ignore warnings. Seriously.
    - immibis 2y ago
      So a good operating system doesn't let you install apps except from the app store?
cgio 2y ago
Lakehouse is an architecture defined to overcome the limitations associated with an immutable object store. It is already in my eyes introducing unnecessary complexity (i.e. at which point do I just need to transition to a proper database, even for larger scales (that cannot be accommodated by a database?), when is a tiered data architecture with stream materialisation snapshots actually simpler to reason about and more economic etc.) I would hope that S3 could introduce a change in the operation of said fundamental building block (immutable object), rather than just slap an existing downstream abstraction. That's not what I call design for simplicity. As an external observer, I would think that's internal amazon moat management with some co-branding strategy.
mannyv 2y ago
The true hero in AWS is its authentication and accounting infrastructure. Most people don't even think about it. But authenticating trillions of operations per second is why AWS works. And the accounting and billing. Anyone that does authentication knows how hard it is. At AWS' scale it's well, the pinnacle of distributed systems.
doktorhladnjak 2y ago
Is there a more Amazon word than "table stakes"?

200 points by riv991 2y ago

"I think one thing that we’ve learned from the work on Tables is that it’s these _properties of storage_ that really define S3 much more than the object API itself." Between the above philosophy, S3 Tables, and Express One Zone (SSD perf), it makes me really curious about what other storage modalities S3 moves towards supporting going forward.

neom 2y ago

little history: When we were getting ready to do an API at DigitalOcean I got asked "uhm... how should it feel?" I thought about that for about 11 seconds and said "if all our APIs feel as good as S3, it should be fine" - it's a good API.

flessner 2y ago
The API, absolutely. It's only sad that the SDKs are often on the heavy side, I remember that the npm package used to be multiple megabytes as it bundled large parts of the core AWS SDK. Nowadays, I believe it's still around half a megabyte.
- malfist 2y ago
  I don't know if the npm is the same way, but the java sdk now has options by service. So you can include just s3 instead of all of aws
- rglover 2y ago
  The footprint of the JS SDK is much better as they split all of it into service-specific packages, but the SDK APIs are a bit confusing (everything is a class that has to be instantiated—even config).

greatpostman 2y ago

I can guarantee you, nothing is simple about S3. I bet engineers spend months on a single configuration change to the underlying subsystems.

jacksnipe 2y ago
That’s what makes it a good abstraction.
ecshafer 2y ago
If you read the article, they say that exactly. Its by Dr Werner Vogels, they know exactly what goes into S3, since they are a principal engineer on the project.
- apetresc 2y ago
  It’s Vogels’ blog but this is a guest post by Andy Warfield.
  - ecshafer 2y ago
    My mistake, I stand corrected.

ginko 2y ago

Would have been good if they mentioned they meant Amazon S3. It took me a while to figure out what this was about. Initially I thought this was about S3 standby mode.

deleted 2y ago
[deleted]
PantaloonFlames 2y ago
All things distributed - the source of the headline - is a blog written by Werner Vogels, CTO of Amazon.
- ginko 2y ago
  Only people in the web dev niche would know this.

waiwai933 2y ago

> I’ve seen other examples where customers guess at new APIs they hope that S3 will launch, and have scripts that run in the background probing them for years! When we launch new features that introduce new REST verbs, we typically have a dashboard to report the call frequency of requests to it, and it’s often the case that the team is surprised that the dashboard starts posting traffic as soon as it’s up, even before the feature launches, and they discover that it’s exactly these customer probes, guessing at a new feature. This surprises me; has anyone done something similar and benefitted from it? It's the sort of thing where I feel like you'd maybe get a result 1% of the time if that, and then only years later when everyone has moved on from the problem they were facing at the time...

easton 2y ago
Maybe this is a faster way of getting AWS feature requests heard. I'm going to write a script that keeps trying to call ecs:MakeFargateCacheImages.
- ajb 2y ago
  It could also be hackers, as when a new service launches is exactly when it will be most buggy. And the contents of S3 are a big payoff.

_1tem 2y ago

I have a feeling that economies of scale have a point of diminishing returns. At what point does it become more costly and complicated to store your data on S3 versus just maintaining a server with RAID disks somewhere? S3 is an engineering marvel, but it's an insanely complicated backend architecture just to store some files.

Tanjreeve 2y ago
Probably never. The complexity is borne by Amazon. Even before any of the development begins if you want a RAID setup with some sort of decent availability you've already multiplied your server costs by the number of replicas you'd need. It's a Sisyphean task that also has little value for most people. Much like twitter it's conceptually simple but it's a hard problem to solve at any scale beyond a toy.
sjsdaiuasgdia 2y ago
That's going to depend a lot on what your needs are, particularly in terms of redundancy and durability. S3 takes care of a lot of that for you. One server with a RAID array can survive, usually, 1 or maybe 2 drive failures. The remaining drives in the array will have to do more work when a failed drive is replaced and data is copied to the new array member. This sometimes leads to additional failures before replacement completes, because all the drives in the array are probably all the same model bought at the same time and thus have similar manufacturing quality and materials. This is part of why it's generally said that RAID != backup. You can make a local backup to something like another server with its own storage, external drives, or tape storage. Capacity, recovery time, and cost varies a lot across the available options here. Now you're protected against the original server failing, but you're not protected against location-based impacts - power/network outages, weather damage/flooding, fire, etc. You can make a remote backup. That can be in a location you own / control, or you can pay someone else to use their storage. Each layer of redundancy adds cost and complexity. AWS says they can guarantee 99.999999999% durability and 99.99% availability. You can absolutely design your own system that meets those thresholds, but that is far beyond what one server with a RAID array can do.
- 2y ago

_fat_santa 2y ago

S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system. One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 10,000 years. Since then anytime I talk about S3's durability I bring up that example and it always seems to convey the point for a layperson. And it's "simplicity" is truly elegant. When I first started using S3 I thought of it as a dumb storage location but as I learned more I realized that it had some wild features that they all managed to hide properly so when you start it looks like a simple system but you can gradually get deeper and deeper until your S3 bucket is doing some pretty sophisticated stuff. Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors.

victorp13 2y ago
> Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors. Counter-point: You know that you're the dominant player. See: .psd, .pdf, .xslx. Not particularly good file types, yet widely supported by competitor products.
- pavlov 2y ago
  Photoshop, PDF and Excel are all products that were truly much better than their competitors at the time of their introduction. Every file format accumulates cruft over thirty years, especially when you have hundreds of millions of users and you have to expand the product for use cases the original developers never imagined. But that doesn’t mean the success wasn’t justified.
  - jalk 2y ago
    PDF is not a product. I get what you are day but I can’t say that I’ve ever liked Adobe Acrobat

conorjh 2y ago

tf is that title supposed to mean?

dgfitz 2y ago
Somehow gambling is directly correlated to using S3? That’s the best I got.
MillironX 2y ago
Table stakes is a poker term meaning the absolute minimum amount you are allowed to bet. So the title translates to "In S3, simplicity is the bare minimum" or "In S3, simplicity is so important that if we didn't have it, we might as well not even have S3."
- MatthewCampbell 2y ago
  It's a risky idiom in general because it's often used to prevent debate. "Every existing product has feature X, so feature X is table stakes." "Why are we testing whether we really need this feature, it's table stakes!" My observation has been that "table stakes" features are often the best ones to reject. (Not so in the case of this title, though)

deleted 2y ago

[deleted]

alberth 2y ago

S3 is the simplest CRUD app you could create. It's essentially just the 4 functions of C.R.U.D done to a file. Most problems in tech are not that simple. Note: not knocking the service. just pointing out not all things are so inherently basic (and valuable at the same time).

riv991 2y ago
Isn't that the most impressive part? That the abstraction makes it seem so simple
bdcravens 2y ago
That's the public interface. The underlying architecture is where the power is.
cruffle_duffle 2y ago
That’s when you really know you hid all the complexity well. When people call your globally replicated data store with granular permissions, sophisticated data retention policies, versioning, and manage to have, what, seven (ten?) nines or something, “simple”. No problem. I’m sure ChatGPT could cook up a replacement in a weekend. Like Dropbox it’s just rsync with some scripts that glue it together. How hard could it possibly be? I mean people serve entire websites right out of s3 buckets. Using it as a crude CDN of sorts. It’s a modern marvel.
- sebastiansm 2y ago
  I could build a netflix in a weekend.
golly_ned 2y ago
A file system is simple. Open, read, close. Most tech problems are not that simple. How hard could a filesystem be?
- dekhn 2y ago
  Locking, checks after unclean shutdown, sparse files, high performance, reliabilty.... are all things that make filesystems harder.

robertwebb 2y ago

[dead]

arnath 2y ago

I found out last year that you can actually run a full SPA using S3 and a CDN. It’s kind of a nuts platform

ellisv 2y ago
I use S3+Cloudfront for static sites and Cloudflare workers if it needed. It's always crazy to me that people will run a could be static site on Netlify/Vercel/etc.
- Cthulhu_ 2y ago
  We've used Netlify at previous projects, we used it because it was easy. No AWS accounts or knowledge needed, just push to master, let the CI build (it was a Gatsby site) and it was live.
  - ellisv 2y ago
    I think Netlify is great but to me it's overkill if you just have a static site. I understand that Netlify is much simpler to get started with and setting up an AWS account is somewhat more complex. If you have several sites, it's worth spending the time to learn.
jorams 2y ago
Since everything you need to run "a full SPA" is to serve some static files over an internet connection I'm not sure how that tells you anything interesting about the platform. It's basically the simplest thing a web server can do.

user9999999999 2y ago

if only metadata could be queried without processing a csv output file first, imagine storing thumbnails in there even! copied objects had actual events, not something you have to dig cloudtrail for, you could get last update time from a bucket to make caching easier

huntaub 2y ago
Are you talking about getting metadata from many objects in the bucket simultaneously? You might be interested in S3 Metadata https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingM....
aeyes 2y ago
When S3 Tables launched they made the Metadata available using this technology. So you can query it like an Apache Iceberg table. https://aws.amazon.com/blogs/aws/introducing-queryable-object-metadata-for-amazon-s3-buckets-preview/ https://aws.amazon.com/blogs/aws/introducing-queryable-objec...
user9999999999 2y ago
whoops. I was wrong! you can store base64 encoded metadata in the object then run a HEAD request to get it, but its limited to 2kb. also, you ~can~ query the metadata, but its latency is more suited to batch processing and not lambdas

chasd00 2y ago

S3 was one of the first offerings coming out of AWS right? It’s pretty legendary and a great concept to begin with. You can tell by how much sense it makes and then trying to wrap your ahead around the web dev world pre-S3.

Cthulhu_ 2y ago
? The web dev world pre-S3 was pretty much the same, but you stored your files on a regular server (and set up your own redundancy and backup strategy). Not that much different to be honest from an end user's point of view.
- kevindamm 2y ago
  At a lot of places there wasn't even a redundancy nor backup strategy, so it really was just as simple as registering with a hosting company and ssh+ftp (or cPanel or something like that for what amounted to the managed solutions of the time). I agree, things before S3 weren't really that different. LAMP stacks everywhere, and developer skills were very portable between different deployments of these LAMP stacks. Single machines didn't scale as much then, but for most small-medium sites they really didn't need to.

ipsento606 2y ago

Lots of comments here talking about how great S3 is. Anyone willing to give a cliff notes about what's good about it? I've been running various sites and apps for a decade, but have never touched S3 because the bandwidth costs are 1, sometimes 2 orders of magnitude more expensive than other static hosting solutions.

waiwai933 2y ago
S3 is great for being able to stick files somewhere and not have to think about any of the surrounding infrastructure on an ongoing basis [1]. You don't have to worry about keeping a RAID server, swapping out disks when one fails, etc. For static hosting, it's fine, but as you say, it's not necessarily the cheapest, though you can bring the cost down by sticking a CDN (Cloudflare/CloudFront) in front of it. There are other use cases where it really shines though. [1]: I say ongoing basis because you will need to figure out your security controls, etc. at the beginning so it's not totally no-thought.
jjice 2y ago
S3 has often fallen into a "catch all" solution for me whenever I need to store data large enough that I don't want to keep it in a database (RDBMS or Redis). Need to save a file somewhere? Dump it in S3. It's generally affordable (obviously dependent on scale and use), fast, easy, and super configurable. Being able to expose something to the outside, or with a presigned URL is a huge advantage as well. Off the top of my head, I think of application storage generally in this tier ordering (just off the top of my head based on the past few years of software development, no real deep thought here): 1. General application data that needs to be read, written, and related - RDBMS 2. Application data that needs to be read and written fast, no relations - Redis 3. Application data that is mostly stored and read - S3 Replace any of those with an equivalent storage layer.
ak217 2y ago
One underappreciated feature of S3 - that allowed it to excel in workloads like the Tables feature described in the article - is that it's able to function as the world's highest throughout network filesystem. And you don't have to do anything to configure it (as the article points out). By storing data on S3, you get to access the full cross-sectional bandwidth of EC2, which is colossal. For effectively all workloads, you will max out your network connection before S3's. This enables workloads that can't scale anywhere else. Things like data pipelines generating unplanned hundred-terabit-per-second traffic spikes with hotspots that would crash any filesystem cluster I've ever seen. And you don't have to pay a lot for it- once you're done using the bandwidth, you can archive the data elsewhere or delete it.

neerajk 2y ago

[flagged]

bob1029 2y ago

I really enjoy using S3 to serve arbitrary blobs. It perfectly solves the problem space for my use cases. I avoid getting tangled in authentication mess by simply naming my files using type4 GUIDs and dumping them in public buckets. The file name is effectively the authentication token and expiration policies are used to deal with the edges. This has been useful for problems like emailing customers gigantic reports and transferring build artifacts between systems. Having a stable URL that "just works" everywhere easily pays for the S3 bill in terms of time & frustration saved.

bityard 2y ago
My favorite use case for S3 API-compatible solutions: I often run into systems that generate lots of arbitrary data that only have temporary importance. A common example might be intermediate build artifacts, or testing ephemera (browser screenshots, etc). Things that are needed for X number of months and then just need to disappear. Yeah, we can dump those to a filesystem. But then we have to ask which filesystem? What should the directory layout be? If there are millions or billions of objects, walking the whole tree gets expensive. Do we write a script to clean everything up? Run it via cron or some other job runner? With S3, you just write your artifact to S3 with a TTL and it gets deleted automagically when it should. No cron jobs, no walking the whole tree. And you can set up other lifecycle options if you need it moved to other (cheaper) storage later on, backups, versioning, and whatever else. For on-prem, you have Minio, Garage, or SeaweedFS. These are pretty nice to deploy the servers however you need for the level of reliability/durability you require.

bentobean 2y ago

It's funny—S3 started as a "simple" storage service, and now it's handling entire table abstractions. Reminds me how SQL was declared dead every few years, yet here we are, building ever more complex data solutions on top of supposedly simple foundations.

imiric 2y ago
I instinctively distrust any software or protocol that implies it is "simple" in its name: SNMP, SMTP, TFTP, SQS, etc. They're usually the cause of an equal or more amount of headaches than alternatives. Maybe such solutions are a reaction to previous more "complex" solutions, and they do indeed start simple, but inevitably get swallowed by the complexity monster with age.
- great_wubwub 2y ago
  TFTP is probably the exception to that rule. All the other protocols started out easily enough and added more and more cruft. TFTP stayed the way it's always been - minimalist, terrifyingly awful at most things, handy for a few corner cases. If you know when to use it and when to use something like SCP, you're golden. If TFTP had gone the way of SNMP, we'd have 'tftp <src> <dest> --proto tcp --tls --retries 8 --log-type json' or some horrendous mess like that.
  - yencabulator 2y ago
    TFTP's usefulness in the modern day is strictly for things that don't have a TCP stack. Anything with a TCP stack is better off with HTTP. That doesn't leave much on the table except legacy & inertia.

paulddraper 2y ago

> In S3 simplicity is table stakes The S3 API is not "simple." Authentication being a good part of that.

brikym 2y ago

I'm curious about S3 tables. Azure has had tables in their storage account product for years. What are the differences?

diroussel 2y ago

It's great that they added iceberg support I guess, but it's a shame that they also removed S3 Select. S3 Select wasn't perfect. For instance the performance was no where near as good as using DuckDB to scan a parquet file, since duck is smart, and S3 Select does a full table scan. But S3 Select is nearly way cheaper that the new iceberg support. So if your needs are only for reading one parquet snapshot, we no need to do updates, then this change is not welcome. Great article though, and I was pleased to see this at the end: > We’ve invested in a collaboration with DuckDB to accelerate Iceberg support in Duck,

StratusBen 2y ago

For those interested in S3 Tables which is referenced in this blog post, we literally just published this overview on what they are and cost considerations of them that people might find interesting: https://www.vantage.sh/blog/amazon-s3-tables https://www.vantage.sh/blog/amazon-s3-tables

1a527dd5 2y ago
https://www.vantage.sh/blog/amazon-s3-tables#s3-tables-cost https://www.vantage.sh/blog/amazon-s3-tables#s3-tables-cost I can't make head or tails of the beginning of this sentence:- > Pricing for S3 Tables is all and all not bad. Otherwise lovely article!
- shawabawa3 2y ago
  "all and all" is a typo for "all in all" which means "overall", or "taking everything into consideration" So they are saying the pricing is not bad considering everything it does
dangoodmanUT 2y ago
> There are 1 million PUT requests and 10 million GET requests that month > + 1,000,000 GET requests x ($0.004 / 1,000 requests) = $9

CobrastanJorji 2y ago

> When we moved S3 to a strong consistency model, the customer reception was stronger than any of us expected. This feels like one of those Apple-like stories about inventing and discovering an amazing, brand new feature that delighted customers but not mentioning the motivating factor of the competing products that already had it. A more honest sentence might have been "After years of customers complaining that the other major cloud storage providers had strong consistency models, customers were relieved when we finally joined the party."

progbits 2y ago
I mainly use GCP but keep hearing how great AWS is in comparison. Imagine my surprise when porting some GCS code to S3 last year and realizing there is no way to get consistency guarantees without external lock service.
- klysm 2y ago
  Distributed locks are a lie
- snoman 2y ago
  That would surprise me too considering read-after-write consistency came to S3 like 5 years ago?
  - jchw 2y ago
    IIRC (it has been a while) the difference is that on Amazon it can only be consistent within a region whereas on GCS I believe even multi-region buckets offer strong consistency.
    - snoman 2y ago
      Ahh. That makes sense!
- thiht 2y ago
  > I mainly use GCP but keep hearing how great AWS is in comparison. Where do you keep hearing this? Having used both, AWS is trash in comparison. It's way to complicated to do anything simple. At work I wish we could migrate to GCP (or just something that's not AWS, really).

lizknope 2y ago

Clicked on the article thinking it was about S3 Graphics, the company that made the graphics chip in my first PC. Now I see it's some amazon cloud storage thing.

Kab1r 2y ago

> S3 launched as the first public AWS service. Didn't SQS launch publicly earlier than S3?

deanCommie 2y ago
SQS went into beta first, S3 went "GA" first. AWS typically considers the "GA" milestone as the "public launch" date, which is silly because the criteria for what is good enough for GA has changed over the years.

CaffeineLD50 2y ago

I wish they'd comment on how their service keeps leaking massive amounts of data that isn't supposed to. Didn't Microsoft blame their customers for not running updates and using AV? Gosh I guess MS did a fine job. Customers were to blame. Just like those s3 buckets. Customers ignored the warnings ...

snoman 2y ago
Not sure what you’re getting at. An S3 bucket out-of-the-box is secure and can’t leak/be accessed publicly. There’s legitimate use cases for making a bucket public. What are you advocating for?
- CaffeineLD50 2y ago
  A good security model doesn't let stupid people do stupid things with warnings. Stupid people ignore warnings. A good model protects stupid people from themselves. A password model that says "you MUST have a strong password" protects stupid people from their own stupidity. A model that says you can use bad passwords if you click through warnings is a shite model. Stupid people ignore warnings. Seriously.
  - immibis 2y ago
    So a good operating system doesn't let you install apps except from the app store?

cgio 2y ago

Lakehouse is an architecture defined to overcome the limitations associated with an immutable object store. It is already in my eyes introducing unnecessary complexity (i.e. at which point do I just need to transition to a proper database, even for larger scales (that cannot be accommodated by a database?), when is a tiered data architecture with stream materialisation snapshots actually simpler to reason about and more economic etc.) I would hope that S3 could introduce a change in the operation of said fundamental building block (immutable object), rather than just slap an existing downstream abstraction. That's not what I call design for simplicity. As an external observer, I would think that's internal amazon moat management with some co-branding strategy.

mannyv 2y ago

The true hero in AWS is its authentication and accounting infrastructure. Most people don't even think about it. But authenticating trillions of operations per second is why AWS works. And the accounting and billing. Anyone that does authentication knows how hard it is. At AWS' scale it's well, the pinnacle of distributed systems.

doktorhladnjak 2y ago

Is there a more Amazon word than "table stakes"?