3 ms·

Apart from Minio, we tried Garage and Ceph. I think there's definitely a need for something that interfaces using S3 API but is just a simple file system undern

by wvh 8mo ago

Apart from Minio, we tried Garage and Ceph. I think there's definitely a need for something that interfaces using S3 API but is just a simple file system underneath, for local, testing and small scale deployments. Not sure that exists? Of course a lot of stuff is being bolted onto S3 and it's not as simple as it initially claimed to be.

memset 8mo ago
For testing, consider https://github.com/localstack/localstack https://github.com/localstack/localstack
- 9dev 8mo ago
  WAY too much. I just need a tiny service that translates common S3 ops into filesystem ops and back.
seddonm1 8mo ago
What about s3 stored in SQLite? https://github.com/seddonm1/s3ite https://github.com/seddonm1/s3ite This was written to store many thousands of images for machine learning
magicalhippo 8mo ago
From what I can gather, S3Proxy[1] can do this, but relies on a Java library that's no longer maintained[2], so not really much better. I too think it would be great with a simple project that can serve S3 from filesystem, for local deployments that doesn't need balls to the walls performance. [1]: https://github.com/gaul/s3proxy https://github.com/gaul/s3proxy [2]: https://jclouds.apache.org/ https://jclouds.apache.org/
hobofan 8mo ago
SeaweedFS's new `weed mini` command[0] does a great job at that. Previously our most flakey tests in CI were due to MinIO sometimes not starting up properly, but with `weed mini` that was completely resolved. [0]: https://github.com/seaweedfs/seaweedfs/wiki/Quick-Start-with-weed-mini https://github.com/seaweedfs/seaweedfs/wiki/Quick-Start-with...
PunchyHamster 8mo ago
Minio started like that but they migrated away from it. It's just hard to keep it up once you start implementing advanced S3 features (versioning/legal hold, metadata etc.) and storage features (replication/erasure coding)
- yencabulator 8mo ago
  There's plenty of middle ground between "just expose underlying FS as objects, can't support many S3 operations well" and "it's a single node not a distributed storage system". For my personal use, asynchronous remote backup is plenty for durability, a Bee-Link ME Mini with 6x4TB NVMe is plenty of space for any bucket I care to have, and I'd love to have an S3 server that doesn't even attempt to be distributed.
egorfine 8mo ago
> for local, testing and small scale deployments Yes I'm looking for exactly that and unfortunately haven't found a solution. Tried garage, but they require running a proxy for CORS, which makes signed browser uploads a practical impossibility for the development machine. I had no idea that such a simple popular scenario is in fact too exotic.
status_quo69 8mo ago
I've been looking into rclone which can serve s3 in a basic way https://rclone.org/commands/rclone_serve_s3/ https://rclone.org/commands/rclone_serve_s3/
amluto 8mo ago
Try versitygw. I’m considering it for a production deployment, too. There’s much to be said for a system that doesn’t lock your data in to its custom storage format.
dark-star 8mo ago
The problem with that approach is that S3 object names are not compatible with POSIX file names. They can contain characters that are not valid on a filesystem, or have special meaning (like "/") A simple litmus test I like to do with S3 storages is to create two objects, one called "foo" and one called "foo/bar". If the S3 uses a filesystem as backend, only the first of those can be created

by wvh 8mo ago

memset 8mo ago

For testing, consider https://github.com/localstack/localstack https://github.com/localstack/localstack

9dev 8mo ago
WAY too much. I just need a tiny service that translates common S3 ops into filesystem ops and back.

seddonm1 8mo ago

What about s3 stored in SQLite? https://github.com/seddonm1/s3ite https://github.com/seddonm1/s3ite This was written to store many thousands of images for machine learning

magicalhippo 8mo ago

From what I can gather, S3Proxy[1] can do this, but relies on a Java library that's no longer maintained[2], so not really much better. I too think it would be great with a simple project that can serve S3 from filesystem, for local deployments that doesn't need balls to the walls performance. [1]: https://github.com/gaul/s3proxy https://github.com/gaul/s3proxy [2]: https://jclouds.apache.org/ https://jclouds.apache.org/

hobofan 8mo ago

SeaweedFS's new `weed mini` command[0] does a great job at that. Previously our most flakey tests in CI were due to MinIO sometimes not starting up properly, but with `weed mini` that was completely resolved. [0]: https://github.com/seaweedfs/seaweedfs/wiki/Quick-Start-with-weed-mini https://github.com/seaweedfs/seaweedfs/wiki/Quick-Start-with...

PunchyHamster 8mo ago

Minio started like that but they migrated away from it. It's just hard to keep it up once you start implementing advanced S3 features (versioning/legal hold, metadata etc.) and storage features (replication/erasure coding)

yencabulator 8mo ago
There's plenty of middle ground between "just expose underlying FS as objects, can't support many S3 operations well" and "it's a single node not a distributed storage system". For my personal use, asynchronous remote backup is plenty for durability, a Bee-Link ME Mini with 6x4TB NVMe is plenty of space for any bucket I care to have, and I'd love to have an S3 server that doesn't even attempt to be distributed.

egorfine 8mo ago

> for local, testing and small scale deployments Yes I'm looking for exactly that and unfortunately haven't found a solution. Tried garage, but they require running a proxy for CORS, which makes signed browser uploads a practical impossibility for the development machine. I had no idea that such a simple popular scenario is in fact too exotic.

status_quo69 8mo ago

I've been looking into rclone which can serve s3 in a basic way https://rclone.org/commands/rclone_serve_s3/ https://rclone.org/commands/rclone_serve_s3/

amluto 8mo ago

Try versitygw. I’m considering it for a production deployment, too. There’s much to be said for a system that doesn’t lock your data in to its custom storage format.

dark-star 8mo ago

The problem with that approach is that S3 object names are not compatible with POSIX file names. They can contain characters that are not valid on a filesystem, or have special meaning (like "/") A simple litmus test I like to do with S3 storages is to create two objects, one called "foo" and one called "foo/bar". If the S3 uses a filesystem as backend, only the first of those can be created