4 ms·

The limitations of S3 (and all the cloud "file systems") are quite astonishing when you consider you're paying for it as a premium service. Try to imagine your

by zmmmmm 3y ago

The limitations of S3 (and all the cloud "file systems") are quite astonishing when you consider you're paying for it as a premium service. Try to imagine your astonishment if a traditional storage vendor showed up and told you that their very expensive premium file system they had just sold you: - can't store log files because it can't append anything to an existing files - can't copy files more than 5GB - can't rename or move a file When challenged on how you are supposed to make all your applications work with limitations like that, they glibly told you "oh you're supposed to rewrite them all".

throwaway290 3y ago
It's for building things on top. If you want to rename/move/copy data, implement a layer that maps objects to "filenames" or any metadata you like (or use some lib). If you want to write logs, implement append and rotation. But I for example don't and won't need any of that and if it helps keep the API simpler and more reliable then I benefit. being a conventional filesystem for S3 would be either a very leaky abstraction or completely different product
School-Cotton 3y ago
Amazon doesn’t market S3 as a replacement for file systems, that’s why EBS exists. Also, is S3 really “very expensive”? Relative to what?
- vbezhenar 3y ago
  S3 usually is the cheapest storage, not only for Amazon, but for other clouds. I don’t understand why.
  - ForHackernews 3y ago
    This is not true in my experience https://www.backblaze.com/cloud-storage/pricing https://www.backblaze.com/cloud-storage/pricing
    - kiwijamo 3y ago
      That Backblaze page (not surprisingly) compares their prices to a fairly expensive S3 pricing tier and makes other assumptions in Blackblaze's favour. For some use cases B2 is more expensive e.g. one copy of my backups goes to AWS Deep Glacier which is really cheap.
      - deleted 3y ago
        [deleted]
Cthulhu_ 3y ago
They're not filesystems though, they're object storage or key/value storage if you will. It's intended to store the log files for long term once they're full. You can rename / move a file, but it involves copying and deleting the original; I don't understand why they don't have a shortcut for that, but it probably makes sense that the user of the service is aware of the process instead of hiding it. I'm not sure about the 5GB limit, it's probably documented somewhere as to why that is; possibly, like tweets, having an upper limit helps them optimize things. Anyway there too there's tools, you can do multipart somethings and there's this official blogpost on the subject: https://aws.amazon.com/blogs/storage/copying-objects-greater-than-5-gb-with-amazon-s3-batch-operations/ https://aws.amazon.com/blogs/storage/copying-objects-greater... Interesting to note maybe in the context of the post; copy, rename, moving large files, all that could be abstracted away, but that would hide the underlying logic - which might lead to inefficient usage of the service - and worse, make users think it's just a filesystem and use it accordingly, but it's not intended or designed for that use case.
- gray_-_wolf 3y ago
  Current limit is 5TB. The 5GB is for a single upload, you can hover do multipart upload to get up to the maximum size of 5TB. https://aws.amazon.com/s3/faqs/ https://aws.amazon.com/s3/faqs/
ozim 3y ago
These “file systems” are not file systems and I don’t understand why people expect them to be. Some people are creating tools that make those services easier to synch with file systems but that is not intended use anyway.
inkyoto 3y ago
S3 is an object storage, not a file system. The file system in AWS is called EFS. S3 is not positioned as a substitute for file systems, either.
deleted 3y ago
[deleted]
pjc50 3y ago
It's not a filesystem, but it has better semantics for distributed operation because of it. Nobody talks about the locking semantics of S3 because it's at the blob level; that rules out whole categories of problems. And that's also why you can't append. If you had multiple readers while appending, and appending to multiple replicas, guaranteeing that each reader would see a consistent only-forwards read of the append is extremely hard. So simply ban people from doing that and force them to use a different system designed for the purpose of logging. Microservices. S3 is for blobs. If you want something that isn't a blob, use a different microservice.

by zmmmmm 3y ago

throwaway290 3y ago

It's for building things on top. If you want to rename/move/copy data, implement a layer that maps objects to "filenames" or any metadata you like (or use some lib). If you want to write logs, implement append and rotation. But I for example don't and won't need any of that and if it helps keep the API simpler and more reliable then I benefit. being a conventional filesystem for S3 would be either a very leaky abstraction or completely different product

School-Cotton 3y ago

Amazon doesn’t market S3 as a replacement for file systems, that’s why EBS exists. Also, is S3 really “very expensive”? Relative to what?

vbezhenar 3y ago
S3 usually is the cheapest storage, not only for Amazon, but for other clouds. I don’t understand why.
- ForHackernews 3y ago
  This is not true in my experience https://www.backblaze.com/cloud-storage/pricing https://www.backblaze.com/cloud-storage/pricing
  - kiwijamo 3y ago
    That Backblaze page (not surprisingly) compares their prices to a fairly expensive S3 pricing tier and makes other assumptions in Blackblaze's favour. For some use cases B2 is more expensive e.g. one copy of my backups goes to AWS Deep Glacier which is really cheap.
    - deleted 3y ago
      [deleted]

Cthulhu_ 3y ago

They're not filesystems though, they're object storage or key/value storage if you will. It's intended to store the log files for long term once they're full. You can rename / move a file, but it involves copying and deleting the original; I don't understand why they don't have a shortcut for that, but it probably makes sense that the user of the service is aware of the process instead of hiding it. I'm not sure about the 5GB limit, it's probably documented somewhere as to why that is; possibly, like tweets, having an upper limit helps them optimize things. Anyway there too there's tools, you can do multipart somethings and there's this official blogpost on the subject: https://aws.amazon.com/blogs/storage/copying-objects-greater-than-5-gb-with-amazon-s3-batch-operations/ https://aws.amazon.com/blogs/storage/copying-objects-greater... Interesting to note maybe in the context of the post; copy, rename, moving large files, all that could be abstracted away, but that would hide the underlying logic - which might lead to inefficient usage of the service - and worse, make users think it's just a filesystem and use it accordingly, but it's not intended or designed for that use case.

gray_-_wolf 3y ago
Current limit is 5TB. The 5GB is for a single upload, you can hover do multipart upload to get up to the maximum size of 5TB. https://aws.amazon.com/s3/faqs/ https://aws.amazon.com/s3/faqs/

ozim 3y ago

These “file systems” are not file systems and I don’t understand why people expect them to be. Some people are creating tools that make those services easier to synch with file systems but that is not intended use anyway.

inkyoto 3y ago

S3 is an object storage, not a file system. The file system in AWS is called EFS. S3 is not positioned as a substitute for file systems, either.

deleted 3y ago

[deleted]

pjc50 3y ago

It's not a filesystem, but it has better semantics for distributed operation because of it. Nobody talks about the locking semantics of S3 because it's at the blob level; that rules out whole categories of problems. And that's also why you can't append. If you had multiple readers while appending, and appending to multiple replicas, guaranteeing that each reader would see a consistent only-forwards read of the append is extremely hard. So simply ban people from doing that and force them to use a different system designed for the purpose of logging. Microservices. S3 is for blobs. If you want something that isn't a blob, use a different microservice.