4 ms·

I have a 14U rack next to my desk. It's doable, but you need to spend time to optimize the noise factor: - If you need many cores but can live with lower cloc

I have a 14U rack next to my desk. It's doable, but you need to spend time to optimize the noise factor: - If you need many cores but can live with lower clock speeds, you can underclock. - 1U servers will be noisy no matter what. 2U is already more manageable. 4Us are the best. - GPUs will be noisy, no real solution around that. High heat production, and they hinder the air flow, so the front fans will have to spin faster. I physically remove GPUs from chassis when I'm not actively using them. - There are small-size soundproof server cabinets that can easily solve this problem. They're very expensive though. - Depending on the server, you can directly control fan speed at the firmware level. This needs careful control if you don't want to destroy your hardware and/or burn down your place. - The pandemic makes all this much harder. I used to keep things running lightly during the night and full blast during the day when everyone was out of the house. I saved tons of money, compared to what I would have paid on any cloud platform. Now that we're home 24/7 it's not so worth it anymore. Also this is all worth it if you actually enjoy playing with the hardware, in that case the money you put into this brings you both fun and cheap compute. If the setup is a chore for you, don't do it, it probably won't be worth your hourly rate. Fun fact: the noise/space/aesthetics factors are referred to in the community as the "WAF" - Wife Acceptance Factor ;-)

127 6y ago
You can always water-cool the GPU, with big radiators. I have no idea about how to do it in 4U, I personally just use off the shelf consumer hardware (mini-ITX, small PSUs, short GPU, etc.) and 3D print the cases. ~10 cm height is perfectly achievable with custom cases and PCI riser cables.
- eptcyka 6y ago
  There are no riser cables that are stable for PCI-Ev4.
  - colejohnson66 6y ago
    I’ve never heard of this. Can you explain why? (I don’t use riser cables FWIW)
    - davemp 6y ago
      I’d imagine the parasitic capacitance becomes more of an issue at the higher frequencies of v4.
    - 1996 6y ago
      Higher frequencies are more susceptible to interference, especially if you have long cables. A short 2 inches riser, surrounded by a grounded metal sheet, might work.
  - MichaelBurge 6y ago
    LINKUP sells them - I have 7 for a home server. Up to 50cm for v4 and I have a couple v3 because they go up to 1 meter. https://www.amazon.com/gp/product/B0898QMCLS https://www.amazon.com/gp/product/B0898QMCLS
damagednoob 6y ago
> There are small-size soundproof server cabinets that can easily solve this problem. I would have thought anything that restricts sound would also impede air flow.
- MasterScrat 6y ago
  See eg this demo: https://youtu.be/NfTP3yhTybc?t=150 https://youtu.be/NfTP3yhTybc?t=150 I could never afford one (they're also super heavy so you have to be careful your floor can take it) but I assume the idea is to replace the airflow noise of many small fans with a few much larger ones.
weeboid 6y ago
Threadripper and 10 SSD RAID array, everything liquid cooled with one large slow fan. Throw in some LED edge RAM
Lex-2008 6y ago
> "WAF" - Wife Acceptance Factor ;-) You made my day! That's the primarily limiting factor in my "bedroom datacenter" expansion :D
- pbronez 6y ago
  Honestly it’s a brilliant term. I’ve seen it in the Audiophile, DIY Audio, and Synthesizer communities as well. While WAF capacity tends to be fairly constant over time, careful strategic planning can ensure sufficient WAF capacity to avoid significant restrictions on side project activity.
unixhero 6y ago
You must acquire a HP Z series Z620 machine for your server needs. Potentially also the Z840 and newer versions Z series (must verify noise emissions from all subsequent models above Z620). I have had a huge success with my Z620. Considering expanding to buy more nodes. It has a very low accoustic footprint. My girlfriend instantly approved :) edit: yes, get the version with the Xeon E5 series. Preferably already with the double CPU configuration. edit2: yes, and also don't get the first version that has Xeon X-series. It will not perform well enough for semi-modern homelab workloads.
- KaiserPro 6y ago
  Just to back this up, the HP z series is rack mountable, but also stackable. They can be used for structural purposes as well, its perfectly possible to make a sturdy set of bookshelves with 4x z620s and 4 scaffold planks.
- fencepost 6y ago
  There are at least 2 generations of the motherboard, make sure you get the second (z620). Also if it still has the original workstation Nvidia card, toss it and replace it. Pissed off the cat that settles on top of the machine since it was no longer so warm, but definitely cheaper to run and the new cheap card is faster too.
na85 6y ago
>Fun fact: the noise/space/aesthetics factors are referred to in the community as the "WAF" - Wife Acceptance Factor ;-) I work in Aerospace so I spend entirely too much time around pilots, amongst whom it's a common thing to refer to Wife Acceptance as Tower Clearance. E.g.: >I'm thinking about getting one of those big green eggs >Nice. Do you think Tower will let you? >I won't let her see it until it's too late Etc
lostlogin 6y ago
I’m a proponent of using Intel NUCs as headless servers. I’d be happier if the psu brick was incorporated and there was another nic, but they quiet, small and powerful in the NUC8 and NUC10 flavours. They can fit 2 drives when in the larger sized case, and so a NAS or similar is probably needed. They can be stacked, are readily available and seem very reliable.
- 1996 6y ago
  To scale, I prefer tablets with a wwan port and a shattered screen: on ebay, for less than $100 each, I can fit 2x m2 NVMe (one 2242 in the wwan port), they are small, and each come with their own UPS (battery) and standardized power connector (USB-c) while supporting KVM (AMT if you know which models to buy)
  - lostlogin 6y ago
    Nice. What are you running on them? ESXi runs nicely on a Nuc, and it’ll do a load of VMs with plenty of resources (the Nucs will take 64gb). But the price for that is probably 5-10x what you are describing.
    - 1996 6y ago
      > the price for that is probably 5-10x what you are describing. Indeed the reason why I do what I do! I can have 10x of them for the cost of 1 of your Nucs. For example, my latest purchase was $15 per tablet with 8Gb of RAM, delivered. Add 2x brand new M2 2230 drives which cost about $20 apiece, and each node costs about $55. > What are you running on them? Sharded databases and Ceph clusters mostly. Each shard uses very little power (and does not run hot) but has 2x nvme in mirror. With raid10 f2 on linux or zfs mirror on bsd you gain in read speed on top of the raid1 safety. Sometimes I add USB external storage: tablets often have 2 USBC ports, so $100 gives an extra 8Tb for warm storage. For HDD, USBC is not a restricting factor anymore. Now I am trying VMs: with a dedicated NVMe, IO performance is good, but there is not enough RAM yet. When 16Gb becomes the new normal, that will be much easier.
neilv 6y ago
I could use a more flexible/scalable term. SOAF (Significant Other Acceptance Factor)? SOA? SOFA? SOC (SO Compliance)? https://news.ycombinator.com/item?id=25067485 https://news.ycombinator.com/item?id=25067485

127 6y ago

You can always water-cool the GPU, with big radiators. I have no idea about how to do it in 4U, I personally just use off the shelf consumer hardware (mini-ITX, small PSUs, short GPU, etc.) and 3D print the cases. ~10 cm height is perfectly achievable with custom cases and PCI riser cables.

eptcyka 6y ago
There are no riser cables that are stable for PCI-Ev4.
- colejohnson66 6y ago
  I’ve never heard of this. Can you explain why? (I don’t use riser cables FWIW)
  - davemp 6y ago
    I’d imagine the parasitic capacitance becomes more of an issue at the higher frequencies of v4.
  - 1996 6y ago
    Higher frequencies are more susceptible to interference, especially if you have long cables. A short 2 inches riser, surrounded by a grounded metal sheet, might work.
- MichaelBurge 6y ago
  LINKUP sells them - I have 7 for a home server. Up to 50cm for v4 and I have a couple v3 because they go up to 1 meter. https://www.amazon.com/gp/product/B0898QMCLS https://www.amazon.com/gp/product/B0898QMCLS

damagednoob 6y ago

> There are small-size soundproof server cabinets that can easily solve this problem. I would have thought anything that restricts sound would also impede air flow.

MasterScrat 6y ago
See eg this demo: https://youtu.be/NfTP3yhTybc?t=150 https://youtu.be/NfTP3yhTybc?t=150 I could never afford one (they're also super heavy so you have to be careful your floor can take it) but I assume the idea is to replace the airflow noise of many small fans with a few much larger ones.

weeboid 6y ago

Threadripper and 10 SSD RAID array, everything liquid cooled with one large slow fan. Throw in some LED edge RAM

Lex-2008 6y ago

> "WAF" - Wife Acceptance Factor ;-) You made my day! That's the primarily limiting factor in my "bedroom datacenter" expansion :D

pbronez 6y ago
Honestly it’s a brilliant term. I’ve seen it in the Audiophile, DIY Audio, and Synthesizer communities as well. While WAF capacity tends to be fairly constant over time, careful strategic planning can ensure sufficient WAF capacity to avoid significant restrictions on side project activity.

unixhero 6y ago

You must acquire a HP Z series Z620 machine for your server needs. Potentially also the Z840 and newer versions Z series (must verify noise emissions from all subsequent models above Z620). I have had a huge success with my Z620. Considering expanding to buy more nodes. It has a very low accoustic footprint. My girlfriend instantly approved :) edit: yes, get the version with the Xeon E5 series. Preferably already with the double CPU configuration. edit2: yes, and also don't get the first version that has Xeon X-series. It will not perform well enough for semi-modern homelab workloads.

KaiserPro 6y ago
Just to back this up, the HP z series is rack mountable, but also stackable. They can be used for structural purposes as well, its perfectly possible to make a sturdy set of bookshelves with 4x z620s and 4 scaffold planks.
fencepost 6y ago
There are at least 2 generations of the motherboard, make sure you get the second (z620). Also if it still has the original workstation Nvidia card, toss it and replace it. Pissed off the cat that settles on top of the machine since it was no longer so warm, but definitely cheaper to run and the new cheap card is faster too.

na85 6y ago

>Fun fact: the noise/space/aesthetics factors are referred to in the community as the "WAF" - Wife Acceptance Factor ;-) I work in Aerospace so I spend entirely too much time around pilots, amongst whom it's a common thing to refer to Wife Acceptance as Tower Clearance. E.g.: >I'm thinking about getting one of those big green eggs >Nice. Do you think Tower will let you? >I won't let her see it until it's too late Etc

lostlogin 6y ago

I’m a proponent of using Intel NUCs as headless servers. I’d be happier if the psu brick was incorporated and there was another nic, but they quiet, small and powerful in the NUC8 and NUC10 flavours. They can fit 2 drives when in the larger sized case, and so a NAS or similar is probably needed. They can be stacked, are readily available and seem very reliable.

1996 6y ago
To scale, I prefer tablets with a wwan port and a shattered screen: on ebay, for less than $100 each, I can fit 2x m2 NVMe (one 2242 in the wwan port), they are small, and each come with their own UPS (battery) and standardized power connector (USB-c) while supporting KVM (AMT if you know which models to buy)
- lostlogin 6y ago
  Nice. What are you running on them? ESXi runs nicely on a Nuc, and it’ll do a load of VMs with plenty of resources (the Nucs will take 64gb). But the price for that is probably 5-10x what you are describing.
  - 1996 6y ago
    > the price for that is probably 5-10x what you are describing. Indeed the reason why I do what I do! I can have 10x of them for the cost of 1 of your Nucs. For example, my latest purchase was $15 per tablet with 8Gb of RAM, delivered. Add 2x brand new M2 2230 drives which cost about $20 apiece, and each node costs about $55. > What are you running on them? Sharded databases and Ceph clusters mostly. Each shard uses very little power (and does not run hot) but has 2x nvme in mirror. With raid10 f2 on linux or zfs mirror on bsd you gain in read speed on top of the raid1 safety. Sometimes I add USB external storage: tablets often have 2 USBC ports, so $100 gives an extra 8Tb for warm storage. For HDD, USBC is not a restricting factor anymore. Now I am trying VMs: with a dedicated NVMe, IO performance is good, but there is not enough RAM yet. When 16Gb becomes the new normal, that will be much easier.

neilv 6y ago

I could use a more flexible/scalable term. SOAF (Significant Other Acceptance Factor)? SOA? SOFA? SOC (SO Compliance)? https://news.ycombinator.com/item?id=25067485 https://news.ycombinator.com/item?id=25067485