3 ms·

Is there any open source S3 compatible software? I know Riak Cloud Storage (http://docs.basho.com/riak/cs/2.1.1/ http://docs.basho.com/riak/cs/2.1.1/), but I th

Is there any open source S3 compatible software? I know Riak Cloud Storage (http://docs.basho.com/riak/cs/2.1.1/ http://docs.basho.com/riak/cs/2.1.1/), but I think it’s not maintained anymore.

cyberferret 9y ago
I believe minio.io provides open source S3 compatible storage on your own servers.
- unixhero 9y ago
  True
unixhero 9y ago
The rclone project supports S3 https://rclone.org/s3/ https://rclone.org/s3/
tobias3 9y ago
OpenStack Swift or Ceph with Ceph Object Gateway. Don't use minio, it's a toy for testing.
- icebraining 9y ago
  I've never used it in production; in what ways is minio lacking compared to Swift and Ceph?
  - tobias3 9y ago
    S3 is an abstraction that gives you a limited amount of operations with limited guarantees (e.g. eventual consistency, can only replace whole files). What you get in return is easier scalability and performance. If you use an S3 API to store files (like minio does) you give up power and gain nothing. So you are better off using NFS, samba, webdav, ftp, etc. Additionally minio doesn't seem to sync files to the file system, so you can't be sure a file is actually stored after a PUT operation (AWS S3 and swift have eventual consistency and Ceph has stronger guarantees).
    - icebraining 9y ago
      That's curious, because the OpenStack Swift docs say "Objects are stored as binary files on the filesystem with metadata stored in the file’s extended attributes (xattrs)". As far as I know, the advantage comes from exposing a course-grained API over the network, which is generally more efficient and reliable than doing many small operations. It would be hard to implement Minio's distributed mode using the filesystem API.
- striking 9y ago
  What's wrong with minio? It does its job and does it well. I did backups with it once upon a time, in production, and it never failed.
- notacoward 9y ago
  That's libelously untrue. You might think Ceph or Swift are better, that's fine, but it's no toy. I've seen a few toy S3 implementations. I even called out Zettar on my blog back in the day. If it's not scalable or highly available it's a toy. If it's unstable it's a toy. If it's super-duper slow it's a toy. Minio is none of these things, plus it has features like erasure coding and encryption that are mature enough to be backed by real support. Best ever? I'm not saying that, but it's no toy. Disclaimer: I have no affiliation with Minio. In fact, they're very nearly a competitor with the project I work on. However, Minio is full of ex-Gluster people and I consider many of them friends.
chrisper 9y ago
Rclone.org
gizzlon 9y ago
https://minio.io/ https://minio.io/
notacoward 9y ago
Minio and Ceph both fit this description, with the latter also offering block and file access. Swift is close, but a slightly different native API (there's an adapter but I'm not sure of its status). Scality has open-sourced their implementation as Zenko. Also LeoFS (which despite the name is not a filesystem).
retrack 9y ago
There is pithos [1] from the Exoscale [2] and powers the production system. A new version of the open source project with API signature v4 and other updates is due out soon. [1] http://pithos.io/ http://pithos.io/ [2] https://www.exoscale.com/object-storage/ https://www.exoscale.com/object-storage/
doublerebel 9y ago
Joyent's Manta has an S3 adapter, all open-source.

Is there any open source S3 compatible software? I know Riak Cloud Storage (http://docs.basho.com/riak/cs/2.1.1/ http://docs.basho.com/riak/cs/2.1.1/), but I think it’s not maintained anymore.

cyberferret 9y ago

I believe minio.io provides open source S3 compatible storage on your own servers.

unixhero 9y ago
True

unixhero 9y ago

The rclone project supports S3 https://rclone.org/s3/ https://rclone.org/s3/

tobias3 9y ago

OpenStack Swift or Ceph with Ceph Object Gateway. Don't use minio, it's a toy for testing.

icebraining 9y ago
I've never used it in production; in what ways is minio lacking compared to Swift and Ceph?
- tobias3 9y ago
  S3 is an abstraction that gives you a limited amount of operations with limited guarantees (e.g. eventual consistency, can only replace whole files). What you get in return is easier scalability and performance. If you use an S3 API to store files (like minio does) you give up power and gain nothing. So you are better off using NFS, samba, webdav, ftp, etc. Additionally minio doesn't seem to sync files to the file system, so you can't be sure a file is actually stored after a PUT operation (AWS S3 and swift have eventual consistency and Ceph has stronger guarantees).
  - icebraining 9y ago
    That's curious, because the OpenStack Swift docs say "Objects are stored as binary files on the filesystem with metadata stored in the file’s extended attributes (xattrs)". As far as I know, the advantage comes from exposing a course-grained API over the network, which is generally more efficient and reliable than doing many small operations. It would be hard to implement Minio's distributed mode using the filesystem API.
striking 9y ago
What's wrong with minio? It does its job and does it well. I did backups with it once upon a time, in production, and it never failed.
notacoward 9y ago
That's libelously untrue. You might think Ceph or Swift are better, that's fine, but it's no toy. I've seen a few toy S3 implementations. I even called out Zettar on my blog back in the day. If it's not scalable or highly available it's a toy. If it's unstable it's a toy. If it's super-duper slow it's a toy. Minio is none of these things, plus it has features like erasure coding and encryption that are mature enough to be backed by real support. Best ever? I'm not saying that, but it's no toy. Disclaimer: I have no affiliation with Minio. In fact, they're very nearly a competitor with the project I work on. However, Minio is full of ex-Gluster people and I consider many of them friends.

chrisper 9y ago

Rclone.org

gizzlon 9y ago

https://minio.io/ https://minio.io/

notacoward 9y ago

Minio and Ceph both fit this description, with the latter also offering block and file access. Swift is close, but a slightly different native API (there's an adapter but I'm not sure of its status). Scality has open-sourced their implementation as Zenko. Also LeoFS (which despite the name is not a filesystem).

retrack 9y ago

There is pithos [1] from the Exoscale [2] and powers the production system. A new version of the open source project with API signature v4 and other updates is due out soon. [1] http://pithos.io/ http://pithos.io/ [2] https://www.exoscale.com/object-storage/ https://www.exoscale.com/object-storage/

doublerebel 9y ago

Joyent's Manta has an S3 adapter, all open-source.