3 ms·
There's not enough information to really answer this question. What is "high performance"? What is "commodity hardware"? How much storage do you actually need?
by batbomb 12y ago
There's not enough information to really answer this question. What is "high performance"? What is "commodity hardware"? How much storage do you actually need? What is the use of the data?
Of course, if you mainly are using Hadoop you'll probably be playing around with HDFS.
If you just have lots of Data and need to process it, maybe Ceph.
If you need POSIX/NFS, you'll probably look at some sort of clustered NFS solution (a la GPFS).
You've also got Gluster and Lustre.
My general recommendation would probably be to examine Ceph first.
Edit: I was assuming you meant some sort of networked storage system, I didn't realize it was for workstations. My answers are probably super overkill for what you need.
- CyberDildonics 12y agoI'm wondering about workstations where everyone is accessing shared directories and gigabytes of data, so lots of IOPS and throughput to replace Netapp and not suffer 50MB/s. Basically my thought is that with combinations of SSDs, PCIe SSDs and things like FreeNAS (or some BSD+ZFS configuration) this shouldn't be an extravagant luxury.
- voidlogic 12y agoYou can now get 1 TB SSDs for ~$380, put 8 of these in RAID-10 (or 5 with HW RAID) shared over a 10 GBE NIC with NFS to workstations with 1 GBE / Jumbo frames. Nothing fancy, performance should be good.
- CyberDildonics 12y agoThat's great for a couple of computers but that isn't a full solution for a couple hundred people all hammering network storage. Also the original post is about NFS, so what do you do with the commodity setup, use NFS or something else?
- voidlogic 12y ago>but that isn't a full solution for a couple hundred people all hammering network storage. Come on, I have seen 8 10k SCSI drives/RAID10 in a Windows 2000 box be used by "a couple hundred people". It wasn't great, but this setup would blow that away, esp. if the hosting machine as a decent amount of RAM (32 GB+). Yes I was suggesting NFS + jumboframes.
- suprjami 12y agoSounds like you need a scale-out filesystem like Ceph or Gluster.
- CyberDildonics 12y agoThanks, I'll look in to those two.