4 ms·

Sorry for two posts (the other one was unrelated). If anyone has experience with this I have a few questions regarding a particular use case. How does somethin

Sorry for two posts (the other one was unrelated). If anyone has experience with this I have a few questions regarding a particular use case. How does something like this behave with really large files. Video files in 100s of gigabytes, for example. I'm asking because if one could set up a resilient online (online as in available) storage with fat pipes like this it could be used as a platform to build a centralized video hub for editing. It's another question how much sense would it make over a filesystem though.

klodolph 10y ago
I think these days we should by default think of storing blobs of data (like video files) in storage systems like S3 or the alternatives, and that ordinary filesystems should be thought of as a special case where you want to attach storage to an individual computer. Edit: I'm going to elaborate, because people are calling me naïve. Full disclosure: I work at a cloud provider on a storage team. For most people and applications, you simply don't get good value for your money by using filesystems and hard drives directly. We've tried to make things more reliable and durable with backup policies, RAID, and ZFS but the fact is all of these things come with operational and capital expenditures that compare unfavorably with common cloud storage options. There are some good technical reasons why cloud storage is better: basically technologies like RAID and ZFS are attempts to make each layer of your storage stack completely durable and available, but this approach is not competitive with the way cloud storage is typically implemented, which is to build a reliable distributed service on top of cheap hardware. Consider RAID 1, for example. This gives you N+1 redundancy at the drive level for an individual computer. This worked in the 1990s but drives are bigger and RAID failure modes suck with larger drives—it's worrying how common it is to see errors when rebuilding a degraded RAID array, and at N+1 that means that your data is lost from that computer. Essentially, with modern drive sizes (4+ TB seems pretty common these days) a RAID 1 array should always be considered N+0 instead of N+1. Cloud storage is implemented much more intelligently. If you have distributed storage, you can simply spread files across computers in different DCs and use error correction codes to increase the redundancy. You can get more nines of durability and availability for less money this way. You end up with something like 33% overhead on disk space instead of 300% overhead, and you're also off the hook for a big chunk of your capacity planning and various other operational expenditures. These days I would consider starting from "this file is in cloud storage, and we have a local cache" rather than "this file is in local storage, but we have a cloud backup". That's really all I'm saying. It also won't always be competitive. Sometimes cloud storage is more expensive than regular filesystems, depending on how you're using it. If you're a big company you can sometimes amortize the costs of doing it yourself better. That's all I mean by "default"—I'm going to put my data in cloud storage unless I have a compelling reason to store it some other way.
- mi100hael 10y ago
  That's awfully naive, especially for tasks like video editing that are significantly impacted by disk read/write speeds. Even a NAS on a gigabit network is going to be roughly 6x slower than a standard internal SATA III spinning disk.
  - Keyframe 10y ago
    I agree. Network-wise we start at 10GbE. It's a lot more complicated than simple file storage on network though. Many needs and solutions. And I mean MANY.
  - klodolph 10y ago
    I said "by default", the implication being that you'd do something else if your application needs it. But it's much easier from an operational perspective if you start with a reliable system (replicated, networked storage) and cache locally for speed, then to try and make local filesystems reliable and durable.
- orestes910 10y ago
  Can you help me understand this statement better? Why should we do that? I may sound like I'm playing dumb, but I'm really struggling to see whats compelling about this in its current state aside from the fact that its one tool as opposed to a RAID + filesystem + something to make the data available.
- alfalfasprout 10y ago
  Uhhh have you seen the internet data transfer costs for S3? That would become absurdly expensive quickly. Even with a dedicated cross connect.
  - fwessels 10y ago
    S3 data transfers costs are an issue -- that's why you can host minio yourself at any hosting company, and save significantly (multiple times) on data transfer and storage costs.
- 10y ago
krishnasrinivas 10y ago
Uses cases like these are a really good fit for Minio. i.e videos/photos ... actually any blob/file.
- Keyframe 10y ago
  Would one expect any issues with large files? Can one file span machines? For example, you have a 1TB single file, but one machine has 500GB free and the other 200GB and third 400GB or whatever (stupid example). I really think this could be useful to build something like Avid Interplay on top of.
  - y4m4b4 10y ago
    Yes definitely you can read our docs here https://docs.minio.io/docs/minio-erasure-code-quickstart-guide https://docs.minio.io/docs/minio-erasure-code-quickstart-gui... for more understanding and even hardware recommendations.
zzzcpan 10y ago
It's a bit confusing, but minio is not a resilient storage. It's just a server, kind of like webdav, but with s3 api and a capability to use multiple filesystem folders with erasure coding. As for distributed object storages, I would expect them to work great for video editing, since they can saturate any link given enough servers. But not out of the box, you would need a client designed for it, splitting files into chunks in parallel, etc.
- krishnasrinivas 10y ago
  We are working on distributed minio (resilient to server failures) on the "distributed" branch here https://github.com/minio/minio/tree/distributed https://github.com/minio/minio/tree/distributed Currently available minio is resilient to disk failures using erasure coding (similar to RAID)

klodolph 10y ago

I think these days we should by default think of storing blobs of data (like video files) in storage systems like S3 or the alternatives, and that ordinary filesystems should be thought of as a special case where you want to attach storage to an individual computer. Edit: I'm going to elaborate, because people are calling me naïve. Full disclosure: I work at a cloud provider on a storage team. For most people and applications, you simply don't get good value for your money by using filesystems and hard drives directly. We've tried to make things more reliable and durable with backup policies, RAID, and ZFS but the fact is all of these things come with operational and capital expenditures that compare unfavorably with common cloud storage options. There are some good technical reasons why cloud storage is better: basically technologies like RAID and ZFS are attempts to make each layer of your storage stack completely durable and available, but this approach is not competitive with the way cloud storage is typically implemented, which is to build a reliable distributed service on top of cheap hardware. Consider RAID 1, for example. This gives you N+1 redundancy at the drive level for an individual computer. This worked in the 1990s but drives are bigger and RAID failure modes suck with larger drives—it's worrying how common it is to see errors when rebuilding a degraded RAID array, and at N+1 that means that your data is lost from that computer. Essentially, with modern drive sizes (4+ TB seems pretty common these days) a RAID 1 array should always be considered N+0 instead of N+1. Cloud storage is implemented much more intelligently. If you have distributed storage, you can simply spread files across computers in different DCs and use error correction codes to increase the redundancy. You can get more nines of durability and availability for less money this way. You end up with something like 33% overhead on disk space instead of 300% overhead, and you're also off the hook for a big chunk of your capacity planning and various other operational expenditures. These days I would consider starting from "this file is in cloud storage, and we have a local cache" rather than "this file is in local storage, but we have a cloud backup". That's really all I'm saying. It also won't always be competitive. Sometimes cloud storage is more expensive than regular filesystems, depending on how you're using it. If you're a big company you can sometimes amortize the costs of doing it yourself better. That's all I mean by "default"—I'm going to put my data in cloud storage unless I have a compelling reason to store it some other way.

mi100hael 10y ago
That's awfully naive, especially for tasks like video editing that are significantly impacted by disk read/write speeds. Even a NAS on a gigabit network is going to be roughly 6x slower than a standard internal SATA III spinning disk.
- Keyframe 10y ago
  I agree. Network-wise we start at 10GbE. It's a lot more complicated than simple file storage on network though. Many needs and solutions. And I mean MANY.
- klodolph 10y ago
  I said "by default", the implication being that you'd do something else if your application needs it. But it's much easier from an operational perspective if you start with a reliable system (replicated, networked storage) and cache locally for speed, then to try and make local filesystems reliable and durable.
orestes910 10y ago
Can you help me understand this statement better? Why should we do that? I may sound like I'm playing dumb, but I'm really struggling to see whats compelling about this in its current state aside from the fact that its one tool as opposed to a RAID + filesystem + something to make the data available.
alfalfasprout 10y ago
Uhhh have you seen the internet data transfer costs for S3? That would become absurdly expensive quickly. Even with a dedicated cross connect.
- fwessels 10y ago
  S3 data transfers costs are an issue -- that's why you can host minio yourself at any hosting company, and save significantly (multiple times) on data transfer and storage costs.
10y ago

krishnasrinivas 10y ago

Uses cases like these are a really good fit for Minio. i.e videos/photos ... actually any blob/file.

Keyframe 10y ago
Would one expect any issues with large files? Can one file span machines? For example, you have a 1TB single file, but one machine has 500GB free and the other 200GB and third 400GB or whatever (stupid example). I really think this could be useful to build something like Avid Interplay on top of.
- y4m4b4 10y ago
  Yes definitely you can read our docs here https://docs.minio.io/docs/minio-erasure-code-quickstart-guide https://docs.minio.io/docs/minio-erasure-code-quickstart-gui... for more understanding and even hardware recommendations.

zzzcpan 10y ago

It's a bit confusing, but minio is not a resilient storage. It's just a server, kind of like webdav, but with s3 api and a capability to use multiple filesystem folders with erasure coding. As for distributed object storages, I would expect them to work great for video editing, since they can saturate any link given enough servers. But not out of the box, you would need a client designed for it, splitting files into chunks in parallel, etc.

krishnasrinivas 10y ago
We are working on distributed minio (resilient to server failures) on the "distributed" branch here https://github.com/minio/minio/tree/distributed https://github.com/minio/minio/tree/distributed Currently available minio is resilient to disk failures using erasure coding (similar to RAID)