4 ms·

I have looked into exactly those boards and they by far seem like the best available solution. I'd be really interested in hearing about your experience with th

I have looked into exactly those boards and they by far seem like the best available solution. I'd be really interested in hearing about your experience with them Which type of case/chassis are you using for your set-up and how many nodes are you running?

3np 5y ago
I have 3x X570D4I-2T with Ryzen 3600. It's my understanding the new Zen3 should work just fine as well. * I gave up on finding a chassis that wasn't either gigantic, too tight, or crazy expensive. They run naked resting on cork-boards on a very cheap DIY shelf that I've hung some 140mm fans on. Looks a bit cyberpunk and swapping drives is super easy. You may have better luck with chassis if you're in the US. * ECC RAM supported, which is nice. I've understood you can't surveil the ECC, so I've seen it described as "semi-ECC". This is a Ryzen thing, not an ASROCK-RACK-thing. * ECC DDR4 SODIMMs are expensive and rare. If you need ECC and lots of RAM but don't care about the small form factor, a bigger board will be a lot more cost-efficient. I got my sticks from Nemix and Samsung. * Intially I had stability issues and errors in memtest. Completely disappeared after putting them behind a UPS; turned out the culprit was power fluctuations in my previous house. * They need a PSU with EPS (my old leftover 15y/o PSU in the closet only booted the IPMI, not the board itself. Silverstone SFX 450W is great and cheap) * (Specific for mITX) The CPU socket has bastard Intel fittings. I got Scythe Shoten and Noctua on there fine in the end, but it took some anxious-inducing creativity with M3 screws/nuts and DIY spacers, * IPMI and monitoring are great for initial setup and if something would go wrong. Flashing UEFI/BIOS is a breeze. * Apart from the on-board SATA I plopped in 4xM.2 PCIe bifurcator cards from 10Gtek - so besides the boot drive I can have 4xM2 NVMe drives on each board. Works great. * I'd really really like to figure out something more compact PSU-wise - either Meanwell or Pico-style - but electricity is not my strong side and despite researching quit a bit I still haven't figured out exactly what I should buy and how I would hook that up. Overall I'm happy. It's quite impressive how much IO you can get out of such a small board. But since I moved to a bigger house and the mATX model came out I'd have bought that instead if it was today. In case you haven't checked out the servethehome forums already there are some people writing about their boards there as well. Oh, and since you mentioned redundant storage: Glusterfs is great, except when it isn't. I have issues with it every now and then (files being stuck in heal without going in split-brain requiring low-level manual intervention, and fuse mounts on other nodes randomly disconnecting with "transport error" until I manually remount, despite always having min 2/3 nodes continuously online). I kind of wish I went with Ceph or LizardFS or something instead but I sunk so much time into Gluster already and have better things to do than start over with something new - and who knows if the grass would actually be greener.
- Scandiravian 5y ago
  That's a fantastic rundown - thank you so much for taking the time to write such a comprehensible explanation! I live in a rather small apartment (56 square metres), so I'm trying to find an acceptable trade-off between noise and space. I'm in Europe, so sourcing parts can unfortunately be a bit of a challenge, so I'll probably have to be creative to solve some issues (which honestly is also fun) I had found a couple of 2U racks that could hold 2 mITX boards, but the lack of cooling options made me nervous whether it would get too hot With you now writing that you would have gone with the mATX board if you were making the build today, I think I'll probably go with that instead - especially since I'd like to use ECC ram, and you mention that ECC SODIMMs are expensive and difficult to find I've been reading a lot of reviews on servethehome, but haven't gone through their forums yet. I'll make sure to check it out - thanks for the tip! I haven't yet decided whether I'll go with ceph, gluster, or something else entirely. I'm leaning towards ceph, but I've read several warnings about running only 3 nodes in a ceph cluster, so it's (once again) a matter of figuring out a good trade-off In any case - you've definitely helped me out here and I really appreciate that you took the time to write such a comprehensive explanation
- moepstar 5y ago
  Wow, that actually sounds awesome and something that could finally enable me to retire my shitty QNAP NAS and my Proxmox "Server" running on an old i7 2nd gen desktop... If you don't mind, how much power/watts does one of these draw continously?
- mastax 5y ago
  The power connectors on that board are strange - it's almost ATX12VO but not quite. Does the motherboard come with the 24-to-4-pin adapter? Assuming you need AC->DC an SFX power supply is probably your best option. It's hard to find generic power supplies that have power good, 5VSB, and remote PS_ON. The generic supplies of reasonable cost are almost certainly less efficient as well.

3np 5y ago

I have 3x X570D4I-2T with Ryzen 3600. It's my understanding the new Zen3 should work just fine as well. * I gave up on finding a chassis that wasn't either gigantic, too tight, or crazy expensive. They run naked resting on cork-boards on a very cheap DIY shelf that I've hung some 140mm fans on. Looks a bit cyberpunk and swapping drives is super easy. You may have better luck with chassis if you're in the US. * ECC RAM supported, which is nice. I've understood you can't surveil the ECC, so I've seen it described as "semi-ECC". This is a Ryzen thing, not an ASROCK-RACK-thing. * ECC DDR4 SODIMMs are expensive and rare. If you need ECC and lots of RAM but don't care about the small form factor, a bigger board will be a lot more cost-efficient. I got my sticks from Nemix and Samsung. * Intially I had stability issues and errors in memtest. Completely disappeared after putting them behind a UPS; turned out the culprit was power fluctuations in my previous house. * They need a PSU with EPS (my old leftover 15y/o PSU in the closet only booted the IPMI, not the board itself. Silverstone SFX 450W is great and cheap) * (Specific for mITX) The CPU socket has bastard Intel fittings. I got Scythe Shoten and Noctua on there fine in the end, but it took some anxious-inducing creativity with M3 screws/nuts and DIY spacers, * IPMI and monitoring are great for initial setup and if something would go wrong. Flashing UEFI/BIOS is a breeze. * Apart from the on-board SATA I plopped in 4xM.2 PCIe bifurcator cards from 10Gtek - so besides the boot drive I can have 4xM2 NVMe drives on each board. Works great. * I'd really really like to figure out something more compact PSU-wise - either Meanwell or Pico-style - but electricity is not my strong side and despite researching quit a bit I still haven't figured out exactly what I should buy and how I would hook that up. Overall I'm happy. It's quite impressive how much IO you can get out of such a small board. But since I moved to a bigger house and the mATX model came out I'd have bought that instead if it was today. In case you haven't checked out the servethehome forums already there are some people writing about their boards there as well. Oh, and since you mentioned redundant storage: Glusterfs is great, except when it isn't. I have issues with it every now and then (files being stuck in heal without going in split-brain requiring low-level manual intervention, and fuse mounts on other nodes randomly disconnecting with "transport error" until I manually remount, despite always having min 2/3 nodes continuously online). I kind of wish I went with Ceph or LizardFS or something instead but I sunk so much time into Gluster already and have better things to do than start over with something new - and who knows if the grass would actually be greener.

Scandiravian 5y ago
That's a fantastic rundown - thank you so much for taking the time to write such a comprehensible explanation! I live in a rather small apartment (56 square metres), so I'm trying to find an acceptable trade-off between noise and space. I'm in Europe, so sourcing parts can unfortunately be a bit of a challenge, so I'll probably have to be creative to solve some issues (which honestly is also fun) I had found a couple of 2U racks that could hold 2 mITX boards, but the lack of cooling options made me nervous whether it would get too hot With you now writing that you would have gone with the mATX board if you were making the build today, I think I'll probably go with that instead - especially since I'd like to use ECC ram, and you mention that ECC SODIMMs are expensive and difficult to find I've been reading a lot of reviews on servethehome, but haven't gone through their forums yet. I'll make sure to check it out - thanks for the tip! I haven't yet decided whether I'll go with ceph, gluster, or something else entirely. I'm leaning towards ceph, but I've read several warnings about running only 3 nodes in a ceph cluster, so it's (once again) a matter of figuring out a good trade-off In any case - you've definitely helped me out here and I really appreciate that you took the time to write such a comprehensive explanation
moepstar 5y ago
Wow, that actually sounds awesome and something that could finally enable me to retire my shitty QNAP NAS and my Proxmox "Server" running on an old i7 2nd gen desktop... If you don't mind, how much power/watts does one of these draw continously?
mastax 5y ago
The power connectors on that board are strange - it's almost ATX12VO but not quite. Does the motherboard come with the 24-to-4-pin adapter? Assuming you need AC->DC an SFX power supply is probably your best option. It's hard to find generic power supplies that have power good, 5VSB, and remote PS_ON. The generic supplies of reasonable cost are almost certainly less efficient as well.