6 ms·

Just use the damn file system. Why does everyone have to put HTTP between everything?

by sdcfgy 17d ago

xienze 17d ago
Because you might be running something that only talks S3 and want to point it at something local.
Tostino 17d ago
Because you want to mock the tools you are going to use in production locally or in CI. Really good reasons to not totally change what you are doing in your app between environments.
- drdexebtjl 17d ago
  > Really good reasons to not totally change what you are doing in your app between environments. Mocks do that. If you want to not totally change what you are doing in your app between environments, point it at an actual S3 bucket in all environments.
wink 17d ago
Because this is about having a local standin for S3, which (in a production deploy) often solves a different problem than a (local) filesystem. This post does not even mention the S3 API as a replacement for a filesystem.
Mashimo 17d ago
Because if you ship a tool with s3 support, the customer can decide if he wants to store in the cloud, or spin up a docker container locally.
- drdexebtjl 17d ago
  The same argument can be made for the other side: the customer can also trivially mount the S3 bucket as a filesystem, if you decide to ship the tool with filesystem support instead.
  - Mashimo 17d ago
    I did not know you could do that. But is that really something you do in an enterprise application? What if you write a file with the same name from two servers to the bucket? What about latency? What if the connection / mount fails, how would the application even notice? Never tried it, I don't think it fits our needs.
bartekpacia 17d ago
I like how you got 4 different comments all starting with "Because..." hah!
jonenst 17d ago
Because AFAIK the filesystem is at the same time a huge API and designed for a different use case. I can think of the following examples: - very flat structures: storing hundred of thousands of files in a single directory will fail - designed for local disks and NFS is a leaky abstraction: - running a system using on locks will fail - cache behaviors work fine for the "humans browsing files" usecase but not so much for other cases probably many more reasons
- sdcfgy 17d ago
  My point is to push the storage abstraction up a level. Then plug S3 or file system into it. I wouldn’t say this if I hadn’t done this on the PB scale. On a file system! With billions of files! People are blind to anything other than shitty cloud APIs now. It’s depressing.
nijave 17d ago
I can pay Amazon pennies on the dollar to saturate 100Gbps with arbitrarily high concurrency Sometimes I also want a smaller version for when I don't need that performance but don't want multiple implementations of storage backends Edit: before someone says NVMe, highly available across 3 fault domains

Just use the damn file system. Why does everyone have to put HTTP between everything?

by sdcfgy 17d ago

Just use the damn file system. Why does everyone have to put HTTP between everything?

xienze 17d ago

Because you might be running something that only talks S3 and want to point it at something local.

Tostino 17d ago

Because you want to mock the tools you are going to use in production locally or in CI. Really good reasons to not totally change what you are doing in your app between environments.

drdexebtjl 17d ago
> Really good reasons to not totally change what you are doing in your app between environments. Mocks do that. If you want to not totally change what you are doing in your app between environments, point it at an actual S3 bucket in all environments.

wink 17d ago

Because this is about having a local standin for S3, which (in a production deploy) often solves a different problem than a (local) filesystem. This post does not even mention the S3 API as a replacement for a filesystem.

Mashimo 17d ago

Because if you ship a tool with s3 support, the customer can decide if he wants to store in the cloud, or spin up a docker container locally.

drdexebtjl 17d ago
The same argument can be made for the other side: the customer can also trivially mount the S3 bucket as a filesystem, if you decide to ship the tool with filesystem support instead.
- Mashimo 17d ago
  I did not know you could do that. But is that really something you do in an enterprise application? What if you write a file with the same name from two servers to the bucket? What about latency? What if the connection / mount fails, how would the application even notice? Never tried it, I don't think it fits our needs.

bartekpacia 17d ago

I like how you got 4 different comments all starting with "Because..." hah!

jonenst 17d ago

Because AFAIK the filesystem is at the same time a huge API and designed for a different use case. I can think of the following examples: - very flat structures: storing hundred of thousands of files in a single directory will fail - designed for local disks and NFS is a leaky abstraction: - running a system using on locks will fail - cache behaviors work fine for the "humans browsing files" usecase but not so much for other cases probably many more reasons

sdcfgy 17d ago
My point is to push the storage abstraction up a level. Then plug S3 or file system into it. I wouldn’t say this if I hadn’t done this on the PB scale. On a file system! With billions of files! People are blind to anything other than shitty cloud APIs now. It’s depressing.

nijave 17d ago

I can pay Amazon pennies on the dollar to saturate 100Gbps with arbitrarily high concurrency Sometimes I also want a smaller version for when I don't need that performance but don't want multiple implementations of storage backends Edit: before someone says NVMe, highly available across 3 fault domains