4 ms·
What is the primary use case for storage this fast — distributed training? Even for that is storage throughput a common bottleneck above say 10GiB/s?
by upbeat_general 5y ago
What is the primary use case for storage this fast — distributed training? Even for that is storage throughput a common bottleneck above say 10GiB/s?
- manquer 5y agoif you are running S3 compatible service in your stack, at reasonable scale you will end serving 10s-100s req/sec in parallel with even 1000s of concurrent users that could overwhelm a system if r/w are not fast enough. Not everyone can use s3 directly,for various reasons mix of legacy/ existing NAS systems or compliance/ regulatory requirements, not on AWS stack but most libraries/clients support S3 APIs even if you use GCP/Azure you could have S3 compatible API with minIO.
- nijave 5y agoIt's also used in big data processing stacks. I think it's currently common to dump all your data in object storage in a structured format and use that as the data store with a query engine on top. Even for something like just querying logs, you can churn through a few TB pretty quick by searching in parallel
- deleted 5y ago[deleted]