3 ms·

Yes. Single ZFS server attached to 3 JBODs(60 disks each). Well, single ZFS server per namespace. We have 3 similar configurations for logically separate storag

Yes. Single ZFS server attached to 3 JBODs(60 disks each). Well, single ZFS server per namespace. We have 3 similar configurations for logically separate storage needs. Yes, nodes have NVMe SSD local scratch space for fast compute needs. Large datasets reside in central storage. Smaller chunks are usually brought to local scratch to let compute jobs do their thing. Results go back to central storage again. I built it for our needs based on freely available docs and community support. The server runs FreeBSD + ZFS + NFS and nothing else. Snapshots from this primary storage gets shipped over to a secondary location that has a slow mirror(similar setup but with cheap SATA disks). Our need is only minor geographical data durability and not lower downtime DR. Experience: beats most commercial stuff we have run so far. Minimal setup of FreeBSD, ZFS, and NFS runs buttery smooth with almost no operational needs. I run upgrades on OS after it has been in community for a while, and it has only been smooth. We didn’t find a need to buy specialist support as our setup is minimal, and workload not any crazy, and availability targets not so strict. Performance wise this setup far exceeded our expectations in all our usecases. We do have 768 GiB of memory for this that ZFS happily makes use of. ARC is about 60-70% of mem most of the time. I will happily build the same again should there be a need for storage.

yjftsjthsd-h 5y ago
But how are you handling availability? If something happens to the hardware (ex. motherboard) or even just for installing OS updates (ex. FreeBSD security patches), do you lose storage for the whole cluster?
- reacharavindh 5y ago
  Yes. Our availability target is not high. Our priorities while designing this system were 1. Data durability - need to make sure scientific data is never lost, 2. cost - our budget is small, and we build what we can afford, 3. performance - extract most value out of the hardware we use, 4. Availability - maximise availability by virtue of having fewer things that can fail. However, fancy failover systems and distributed filesystems may be out of reach(at least for now). So, we make do with what we have. We have three similar storage configs(and three mirrors as slow backups). We carry spare memory sticks, Optane drives, disks, and power supply for minimising down time due to hardware failures. If something were to go wrong for which we don't have spares, yes, the storage will be offline until we can fix it. Planned updates? - we have two scheduled downtimes in a year. All patches to these machines happen at that time. These machines are strictly internal and don't have internet access except for those planned maintenance days.

yjftsjthsd-h 5y ago

But how are you handling availability? If something happens to the hardware (ex. motherboard) or even just for installing OS updates (ex. FreeBSD security patches), do you lose storage for the whole cluster?

reacharavindh 5y ago
Yes. Our availability target is not high. Our priorities while designing this system were 1. Data durability - need to make sure scientific data is never lost, 2. cost - our budget is small, and we build what we can afford, 3. performance - extract most value out of the hardware we use, 4. Availability - maximise availability by virtue of having fewer things that can fail. However, fancy failover systems and distributed filesystems may be out of reach(at least for now). So, we make do with what we have. We have three similar storage configs(and three mirrors as slow backups). We carry spare memory sticks, Optane drives, disks, and power supply for minimising down time due to hardware failures. If something were to go wrong for which we don't have spares, yes, the storage will be offline until we can fix it. Planned updates? - we have two scheduled downtimes in a year. All patches to these machines happen at that time. These machines are strictly internal and don't have internet access except for those planned maintenance days.