5 ms·

I usually have to tweak my home rackmount equipment over IPMI to set the fan settings to idle when temps are low instead of 24/7/365 jet engine mode which the l

I usually have to tweak my home rackmount equipment over IPMI to set the fan settings to idle when temps are low instead of 24/7/365 jet engine mode which the local university datacenter seems to prefer. I imagine running the fans full blast all the time results in slightly more reliable performance and reliability, but I don't have the luxury of a soundproof HVAC controlled room to lock everything away in, so the lower noise at lower utilization matters to me. This trade-off definitely happens all over the place in enterprise gear.

justinclift 2y ago
You've been through all the stuff like setting max device power for the NVMe drives and similar yeah? https://nvmexpress.org/resource/technology-power-features/ https://nvmexpress.org/resource/technology-power-features/
- timschmidt 2y ago
  I just upgraded my main server to an https://www.supermicro.com/products/archive/motherboard/x9drw-3f https://www.supermicro.com/products/archive/motherboard/x9dr... with dual Xeon E5-2680v2s. $15 for 20c/40t and 256gb ECC from the local university surplus. Unfortunately, it's still a little too old to offer NVME. That doesn't matter too much to me, since it's connected via 6gbps SAS to several 24 bay NetApp disk shelves (also $15 ea!) full of disks. I have dealt with PCIe ASPM (https://en.wikipedia.org/wiki/Active_State_Power_Management https://en.wikipedia.org/wiki/Active_State_Power_Management) which similarly trades off between performance, power consumption, and latency. I am not hugely concerned with power consumption as I have solar panels. But noise and excess heat matter to me. My whole stack is consuming around 650W currently. Wildly removed from the days I was running off a creatively overprovisioned OpenWRT wireless router using < 20W. But I'm happy to have the increased capacity for my use cases. Gigabit synchronous fiber to the home is a dangerous drug :)
  - justsomehnguy 2y ago
    Just use PCI-E converter for NVMe. You wouldn't be able to boot from it, but it would work just fine.
    - timschmidt 2y ago
      A good suggestion, but it's a 1U server with both available PCIe slots populated between the external SAS controller and a dual 10gbE CNA. I boot from SATA SSDs currently. Could upgrade to enterprise SAS SSDs, but haven't noticed a need for my current uses. The boot drive is rarely touched, as most data terminates at the NetApp disk shelves, which have been a far more reliable way to connect unholy amounts of storage to a machine than any of the consumer direct attached storage options I've tried in the past. There's a mix of SSDs and spinning rust in the disk shelves - SSDs for web server stuff and spinning rust for low cost bulk storage.
      - justinclift 2y ago
        > with both available PCIe slots populated between the external SAS controller and a dual 10gbE CNA. Interestingly, there do appear to be PCIe cards that have both external SAS and high speed networking in one: https://forums.servethehome.com/index.php?threads/unique-oem-dual-lsi-2308-mellanox-connectx-3-vpi-combo-expansion-cards.30532/ https://forums.servethehome.com/index.php?threads/unique-oem... Looks like a bunch of mucking around is needed to get things setup, but seems like it works ok from that point onwards: • https://www.ebay.com/itm/314150370285 https://www.ebay.com/itm/314150370285 Have been considering grabbing one to try in a HP Microserver Gen 8 that runs Proxmox. Not a lot of free time at the moment though. ;)
        timschmidt 2y ago
        Neat! And affordable! Thanks! I will definitely look further into them. I got the 10gbE CNA from another university surplus find where a half dozen were being used as 16gb fibre channel adapters with SFP fibre modules. A little digging revealed that they could be switched to 10gbE CNA mode which is a lot more useful for my purposes. Enterprise gear is full of surprises :D Only potential downside I see is the included pcie switch chip. It's dividing up 8gbps between three devices which could independently consume all of that bandwidth. Probably not a huge issue for my uses, but worth considering if you plan on transferring large amounts of data over network and storage simultaneously.

justinclift 2y ago

You've been through all the stuff like setting max device power for the NVMe drives and similar yeah? https://nvmexpress.org/resource/technology-power-features/ https://nvmexpress.org/resource/technology-power-features/

timschmidt 2y ago
I just upgraded my main server to an https://www.supermicro.com/products/archive/motherboard/x9drw-3f https://www.supermicro.com/products/archive/motherboard/x9dr... with dual Xeon E5-2680v2s. $15 for 20c/40t and 256gb ECC from the local university surplus. Unfortunately, it's still a little too old to offer NVME. That doesn't matter too much to me, since it's connected via 6gbps SAS to several 24 bay NetApp disk shelves (also $15 ea!) full of disks. I have dealt with PCIe ASPM (https://en.wikipedia.org/wiki/Active_State_Power_Management https://en.wikipedia.org/wiki/Active_State_Power_Management) which similarly trades off between performance, power consumption, and latency. I am not hugely concerned with power consumption as I have solar panels. But noise and excess heat matter to me. My whole stack is consuming around 650W currently. Wildly removed from the days I was running off a creatively overprovisioned OpenWRT wireless router using < 20W. But I'm happy to have the increased capacity for my use cases. Gigabit synchronous fiber to the home is a dangerous drug :)
- justsomehnguy 2y ago
  Just use PCI-E converter for NVMe. You wouldn't be able to boot from it, but it would work just fine.
  - timschmidt 2y ago
    A good suggestion, but it's a 1U server with both available PCIe slots populated between the external SAS controller and a dual 10gbE CNA. I boot from SATA SSDs currently. Could upgrade to enterprise SAS SSDs, but haven't noticed a need for my current uses. The boot drive is rarely touched, as most data terminates at the NetApp disk shelves, which have been a far more reliable way to connect unholy amounts of storage to a machine than any of the consumer direct attached storage options I've tried in the past. There's a mix of SSDs and spinning rust in the disk shelves - SSDs for web server stuff and spinning rust for low cost bulk storage.
    - justinclift 2y ago
      > with both available PCIe slots populated between the external SAS controller and a dual 10gbE CNA. Interestingly, there do appear to be PCIe cards that have both external SAS and high speed networking in one: https://forums.servethehome.com/index.php?threads/unique-oem-dual-lsi-2308-mellanox-connectx-3-vpi-combo-expansion-cards.30532/ https://forums.servethehome.com/index.php?threads/unique-oem... Looks like a bunch of mucking around is needed to get things setup, but seems like it works ok from that point onwards: • https://www.ebay.com/itm/314150370285 https://www.ebay.com/itm/314150370285 Have been considering grabbing one to try in a HP Microserver Gen 8 that runs Proxmox. Not a lot of free time at the moment though. ;)
      - timschmidt 2y ago
        Neat! And affordable! Thanks! I will definitely look further into them. I got the 10gbE CNA from another university surplus find where a half dozen were being used as 16gb fibre channel adapters with SFP fibre modules. A little digging revealed that they could be switched to 10gbE CNA mode which is a lot more useful for my purposes. Enterprise gear is full of surprises :D Only potential downside I see is the included pcie switch chip. It's dividing up 8gbps between three devices which could independently consume all of that bandwidth. Probably not a huge issue for my uses, but worth considering if you plan on transferring large amounts of data over network and storage simultaneously.