4 ms·

Does anyone have large (PB-scale) deployments of Minio on premises? We have a need to store hundreds of terabytes of data, and we don't need a filesystem per se

Does anyone have large (PB-scale) deployments of Minio on premises? We have a need to store hundreds of terabytes of data, and we don't need a filesystem per se, so I was wondering how robust it is.

alrs 10y ago
Physically impossible to run a PB-scale Minio cluster: it's limited to 16 hard drives. See my comment above.
- general_ai 10y ago
  Good to know. Looks like more of a PR initiative then. Some of my servers have more drives than that. :-)
  - unlocksmith 10y ago
    Minio shards data across 16 drives. If you have more drives, you would run multiple instances for each set of 16 drives. I would generally recommend against denser storage servers. When it goes down, all the drives go offline with it.
y4m4b4 10y ago
With 16 x 8TB drives, we are talking about 64TB usable space after erasure coding. Minio is designed to serve the needs of a single tenant - 64TB is usually much bigger than a single tenant's needs. However you often see 100s of thousands of such tenants in a cloud like environment. Making a single large PB sized volume where the disks and nodes are managed like a blackbox by the filesystem is quite scary to us at Minio. Any crash means we blew up all the tenants at once. 1000s of individual Minio tenants means, we know when we add the million'th minio instance, it is not any more complex than the first instance of Minio. Provisioning with k8s [1], mesos [2] or other external orchestration tools is better than Minio's own resource management system. When it comes to the applications, objects are just represented as URLs. Some data sitting on Amazon S3 and some on Minio makes no difference to the application. We are also planning to introduce a namespace layer to work on top to combine these individual tenants into single namespace. This would allow you to transparently add new clusters as you scale without loosing the ability to have a single namespace. We hang out at slack [3], please reach out if you have further questions. [1] - https://github.com/kubernetes/charts/tree/master/stable/minio https://github.com/kubernetes/charts/tree/master/stable/mini... [2] - https://github.com/dcos/examples/tree/master/1.8/minio https://github.com/dcos/examples/tree/master/1.8/minio [3] - https://slack.minio.io https://slack.minio.io

Does anyone have large (PB-scale) deployments of Minio on premises? We have a need to store hundreds of terabytes of data, and we don't need a filesystem per se, so I was wondering how robust it is.

alrs 10y ago

Physically impossible to run a PB-scale Minio cluster: it's limited to 16 hard drives. See my comment above.

general_ai 10y ago
Good to know. Looks like more of a PR initiative then. Some of my servers have more drives than that. :-)
- unlocksmith 10y ago
  Minio shards data across 16 drives. If you have more drives, you would run multiple instances for each set of 16 drives. I would generally recommend against denser storage servers. When it goes down, all the drives go offline with it.

y4m4b4 10y ago

With 16 x 8TB drives, we are talking about 64TB usable space after erasure coding. Minio is designed to serve the needs of a single tenant - 64TB is usually much bigger than a single tenant's needs. However you often see 100s of thousands of such tenants in a cloud like environment. Making a single large PB sized volume where the disks and nodes are managed like a blackbox by the filesystem is quite scary to us at Minio. Any crash means we blew up all the tenants at once. 1000s of individual Minio tenants means, we know when we add the million'th minio instance, it is not any more complex than the first instance of Minio. Provisioning with k8s [1], mesos [2] or other external orchestration tools is better than Minio's own resource management system. When it comes to the applications, objects are just represented as URLs. Some data sitting on Amazon S3 and some on Minio makes no difference to the application. We are also planning to introduce a namespace layer to work on top to combine these individual tenants into single namespace. This would allow you to transparently add new clusters as you scale without loosing the ability to have a single namespace. We hang out at slack [3], please reach out if you have further questions. [1] - https://github.com/kubernetes/charts/tree/master/stable/minio https://github.com/kubernetes/charts/tree/master/stable/mini... [2] - https://github.com/dcos/examples/tree/master/1.8/minio https://github.com/dcos/examples/tree/master/1.8/minio [3] - https://slack.minio.io https://slack.minio.io