3 ms·

Switching over to erasure based storage such as Minio could bring down the cost even further. https://www.minio.io/ https://www.minio.io/

by acd 10y ago

fnord123 10y ago
Minio is an interesting new project but they seem to be favouring "features" over stability. For now it would be better to go with a proven system like WD Active Archive or Scality.
Veratyr 10y ago
They're already on Ceph-based Cloud Object Store, which uses 8/3 erasure coding: https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-object-store-object-storage-at https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-ob... Erasure coding isn't something new, unknown or unique to Minio, it's built into Ceph, GlusterFS and Openstack Swift, the largest distributed object stores.
londons_explore 10y ago
I want to see a distributed erasure coding system. For example, the data is distributed across 10 datacenters, and every object is available in one datacenter without requiring slow costly network roundtrips, but if that datacenter becomes unavailable it can be recovered from a combination of the other 9.
- rakoo 10y ago
  I'm not aware of any company doing that (I know backblaze does that but at the pod level, not at the datacenter level... because they only have one) You may be interested in Tahoe-LAFS though (https://tahoe-lafs.org/trac/tahoe-lafs https://tahoe-lafs.org/trac/tahoe-lafs). It has many good things in it, one of them is that all files get erasure-encoded so that k nodes out of n are needed to restore the file. When you set a node to be a storage provider (such as S3, GCS, ...), then you effectively have erasure encoding over providers: If S3 is down, you can still retrieve your data from the rest of the providers. Some people have actually tried to do it, as described in the first section in (https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TipsTricks https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TipsTricks).

Switching over to erasure based storage such as Minio could bring down the cost even further. https://www.minio.io/ https://www.minio.io/

by acd 10y ago

Switching over to erasure based storage such as Minio could bring down the cost even further. https://www.minio.io/ https://www.minio.io/

fnord123 10y ago

Minio is an interesting new project but they seem to be favouring "features" over stability. For now it would be better to go with a proven system like WD Active Archive or Scality.

Veratyr 10y ago

They're already on Ceph-based Cloud Object Store, which uses 8/3 erasure coding: https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-object-store-object-storage-at https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-ob... Erasure coding isn't something new, unknown or unique to Minio, it's built into Ceph, GlusterFS and Openstack Swift, the largest distributed object stores.

londons_explore 10y ago

I want to see a distributed erasure coding system. For example, the data is distributed across 10 datacenters, and every object is available in one datacenter without requiring slow costly network roundtrips, but if that datacenter becomes unavailable it can be recovered from a combination of the other 9.

rakoo 10y ago
I'm not aware of any company doing that (I know backblaze does that but at the pod level, not at the datacenter level... because they only have one) You may be interested in Tahoe-LAFS though (https://tahoe-lafs.org/trac/tahoe-lafs https://tahoe-lafs.org/trac/tahoe-lafs). It has many good things in it, one of them is that all files get erasure-encoded so that k nodes out of n are needed to restore the file. When you set a node to be a storage provider (such as S3, GCS, ...), then you effectively have erasure encoding over providers: If S3 is down, you can still retrieve your data from the rest of the providers. Some people have actually tried to do it, as described in the first section in (https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TipsTricks https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TipsTricks).