3 ms·
> If you're doing a 20+ disk array ... Then you might want to consider ceph.
by tutfbhuf 6y ago
> If you're doing a 20+ disk array ...
Then you might want to consider ceph.
- secabeen 6y agoYou might, but you might not. I have an 140 disk array that just works. Ceph would require multiple nodes, lots of tuning, maintenance, etc.
- tutfbhuf 6y agoBut multiple nodes offers you resilience against node failures not just drive failures. And with Ceph you have the flexibility to add or remove single disks as you like. And you have multiple storage interface not just FS, but also S3 or block device if you like.
- mbreese 6y agoBut that's a lot of extra overhead too. You need monitors, OSDs, metadata servers, etc... If you're comfortable with the node being a single point of failure, you can do a lot with a single storage node connected to a JBOD and serving data over NFS. And even if the node is the unit of failure, you can attach a SAS JBOD to multiple servers if you'd like. In my position, in addition to the added cost, I just don't have the physical rack space for the extra servers required for ceph.
- tutfbhuf 6y ago> don't have the physical rack space for the extra servers required for ceph Sure, it depends but the 140 drives (mentioned by secabeen) alone will take up a huge amount of space. Maybe I just wanted to point out that there might be a point when you have maxed out the storage capability of a single node and might want to consider a distributed storage setup.
- secabeen 6y agoThey take about 14U total. (3x4U 60-drive JBOds, and a 2U server.) Big, but not huge.
- secabeen 6y agoAnd I need none of those things. This is bulk storage over CIFS. I have two systems in two different buildings, and we bought enough capacity to exceed the projected lifetime of the system. It should run till drives start reaching EOL, at which point, we'll migrate to a replacement, or shut it down.