6 ms·

7950X3D, X670E Taichi, 96GB 6400MHz CL32, 2x4TB Lexar, 4x18TB Seagate Exos X18, RX570 8G, Proxmox. Idle no VM ~60-70W. Idle TrueNAS VM drives spinning ~90-100

7950X3D, X670E Taichi, 96GB 6400MHz CL32, 2x4TB Lexar, 4x18TB Seagate Exos X18, RX570 8G, Proxmox. Idle no VM ~60-70W. Idle TrueNAS VM drives spinning ~90-100W. Idle TrueNAS & Fedora Desktop with GPU passthrough ~150W In a few weeks the 570 is replaced by 7900 xtx. The RAM adds a lot of W. 3-5W per 8GB of RAM depending on the frequency is common for DDR5. I was expecting around 50-100W for Proxmox+TrueNAS. I did not consider the power draw of the RAM when I went for 96GB.

eurekin 3y ago
What about networking? Did you go over 1gbit?
- ThatMedicIsASpy 3y ago
  It has 2.5G. There are X670E with 10G if you desire more. My home net is 1G with two MikroTik hAP ax3 which are connected with the single 2.5G poe port they have (and one powers the other).
hypercube33 3y ago
I really want the Ryzen Embedded and or Epyc 3000(?) series that has dual 10gbe on package for something like a NAS but both are super expensive or impossible to find.
- ThatMedicIsASpy 3y ago
  AsRock Rack B650D4U-2L2T/BCM, 2x10G, 2x1G, IPMI For less power consumption Ryzen 8000 is coming up (wait for Jan 8th, CES) and the APU tend to be monolithic and draw a lot less power than the chiplets.
  - tw04 3y ago
    Even that uses Broadcom 10gbe, not the embedded AMD Ethernet. It’s really strange, I can only assume there’s something fatally wrong with the AMD Ethernet.
    - AdrianB1 3y ago
      Or it just tells the customers of this kind of equipment want proven solutions instead of other (novelty) options, so the manufacturers build their products with that in mind. Stability and support are very important to most buyers.
      - tw04 3y ago
        If that were the case I’d expect to still see at least SOME products utilizing the AMD chipset, even if budget focused. I have literally not seen a single board from any MFG that utilizes the built in NIC. Heck there are Intel Xeon-d chipsets that utilize both the onboard NIC and external Broadcom to get 4x for cheap.
        formerly_proven 3y ago
        That's because neither AM4 nor AM5 include provisions for a NIC (MAC) on the SoC. Afaik the chipsets don't, either. Intel chipsets generally have a gigabit MAC, but this is rarely used (apparently it specifically requires an Intel PHY and that's more expensive than a complete NIC from say Realtek). Only the Embedded chips have MACs built in.
- j45 3y ago
  It may be possible to install, or add an external 2.5 or 10GbE device. Either way, it's awful there is not more 10 GbE connectivity available default. There's no reason it shouldn't be the next level up, we have been at 1 / 2.5 for far too long.
  - ThatMedicIsASpy 3y ago
    You can find what you desire but you always have to pay for it. ASUS ProArt X670E-Creator WIFI, 10G & 2.5G at 460€ 10G simply isn't that cheap. The cheapest 5 port switch is 220€. Upgrading my home net would be rather expensive.
    - vetinari 3y ago
      What makes is more expensive is insisting on 10GBase-T. 10G over SFP+ is not that expensive; the cheapest 4 port switch (Mikrotik CRS305) is ~130 EUR.
      - selectodude 3y ago
        Optics aren’t free either.
      - j45 3y ago
        I’m currently working through this. My fibre internet is faster than the 2.5 GbE. A UDM-Pro could be added to the mix but simply pushes the issue out. A used datacenter/server grade SFP+ switch could be the way to go but have to get up to speed on the basics of the equipment I’d need. Open to ideas and suggestions!
MrFoof 3y ago
You can go down to 50W idle, but it requires some very specific hardware choices where the ROI will never materialize, some of which aren’t available yet for Zen4. I have… * AMD Ryzen 7 PRO 5750GE * 128GB ECC DDR4-3200 * Intel XL710-QDA2 (using QSFP+ to a quad-SFP+ passive DAC breakout) * LSI 9500-16i * Eight WD 16TB HDDs (shucked) * Two 2TB SK Hynix P41 Platinum M.2 NVMe SSD * Two Samsung 3.84TB PM9A3 U.2 NVMe SSD * Two Samsung 960GB PM893 SATA SSD So that’s the gist. Has a BMC, but dual 40GbE and can sustain about 55GbE over the network (in certain scenarios, 30-35GbE for almost all), running TrueNAS scale purely as a storage appliance for video editing, a Proxmox cluster (on 1L SFFs with 5750GEs and 10GbE idling at 10W each!) mostly running Apache Spark, a Pi4B 8Gb k3s cluster and lots more. Most of what talks to it is either 40GbE or 10GbE. There is storage tiering set up so the disks are very rarely hit, so they’re asleep most of the time. It mostly is serving data to or from the U.2s, shuffling it around automatically later on. The SATA SSDs are just metadata. It actually boots off a SuperMicro SuperDOM. —— The Zen 3 Ryzen PRO 5750GEs are unicorns, but super low power. Very tiny idle (they’re laptop cores), integrated GPU, ECC support, and the memory protection features of EPYC. 92% of the performance of a 5800X, but all 8C/16T flat out (at 3.95GHz because of an undervolt) caps at just under 39W package power. The LSI 9500-16i gave me all the lanes I needed (8 PCIe, 16 SlimSAS) for the two enterprise U.2 and 8 HDDs, and was very low idle power by being a newer adapter. The Intel dual QSFP+ NIC was deliberate as using passive DACs over copper saved 4-5W per port (8 at the aggregation switch) between the NIC and the switch. Yes, really. Plus lower latency (than even fiber) which matters at these transfer speeds. The “pig” is honestly the ASRock X570D4U because the BMC is 3.2W on its own, and X570 is a bit power hungry itself. But all in all, the whole system idles at 50W, is usually 75-80W under most loads, but can theoretically peak probably around 180-190W if everything was going flat out. It uses EVERY single PCIe lane available from the chipset and CPU to its fullest! Very specific chassis fan choices and Noctua low profile cooler in a super short depth 2U chassis. I’ve never heard it make a peep, disks aside :)
- ianai 3y ago
  Is the “e” for embedded? Ie needs to be bought in a package? I’m not seeing many market options.
dist-epoch 3y ago
> 3-5W per 8GB of RAM I think that's wrong. It would mean 4*6=25W per DIMM. I also have 48GB DDR5 DIMMs and HwInfo shows 6W max per module.
- ThatMedicIsASpy 3y ago
  Where are those 48G dimms? Ryzen 7000 limits 4 dimms to 3600MHz. To get high clocks 2 dimm configs is the maximum for Ryzen 7000. Are they in use? I can cut off 10-15W going to 4800MHz 1.1V (from 6400 1.3V)
  - dist-epoch 3y ago
    > Ryzen 7000 limits 4 dimms to 3600MHz This is official support, just like 2 dimms are officially supported only at 5200MHz. People have been running them higher. How are you measuring the power? At the wall? I think you are measuring the memory controller too, not just the dimms.

eurekin 3y ago

What about networking? Did you go over 1gbit?

ThatMedicIsASpy 3y ago
It has 2.5G. There are X670E with 10G if you desire more. My home net is 1G with two MikroTik hAP ax3 which are connected with the single 2.5G poe port they have (and one powers the other).

hypercube33 3y ago

I really want the Ryzen Embedded and or Epyc 3000(?) series that has dual 10gbe on package for something like a NAS but both are super expensive or impossible to find.

ThatMedicIsASpy 3y ago
AsRock Rack B650D4U-2L2T/BCM, 2x10G, 2x1G, IPMI For less power consumption Ryzen 8000 is coming up (wait for Jan 8th, CES) and the APU tend to be monolithic and draw a lot less power than the chiplets.
- tw04 3y ago
  Even that uses Broadcom 10gbe, not the embedded AMD Ethernet. It’s really strange, I can only assume there’s something fatally wrong with the AMD Ethernet.
  - AdrianB1 3y ago
    Or it just tells the customers of this kind of equipment want proven solutions instead of other (novelty) options, so the manufacturers build their products with that in mind. Stability and support are very important to most buyers.
    - tw04 3y ago
      If that were the case I’d expect to still see at least SOME products utilizing the AMD chipset, even if budget focused. I have literally not seen a single board from any MFG that utilizes the built in NIC. Heck there are Intel Xeon-d chipsets that utilize both the onboard NIC and external Broadcom to get 4x for cheap.
      - formerly_proven 3y ago
        That's because neither AM4 nor AM5 include provisions for a NIC (MAC) on the SoC. Afaik the chipsets don't, either. Intel chipsets generally have a gigabit MAC, but this is rarely used (apparently it specifically requires an Intel PHY and that's more expensive than a complete NIC from say Realtek). Only the Embedded chips have MACs built in.
j45 3y ago
It may be possible to install, or add an external 2.5 or 10GbE device. Either way, it's awful there is not more 10 GbE connectivity available default. There's no reason it shouldn't be the next level up, we have been at 1 / 2.5 for far too long.
- ThatMedicIsASpy 3y ago
  You can find what you desire but you always have to pay for it. ASUS ProArt X670E-Creator WIFI, 10G & 2.5G at 460€ 10G simply isn't that cheap. The cheapest 5 port switch is 220€. Upgrading my home net would be rather expensive.
  - vetinari 3y ago
    What makes is more expensive is insisting on 10GBase-T. 10G over SFP+ is not that expensive; the cheapest 4 port switch (Mikrotik CRS305) is ~130 EUR.
    - selectodude 3y ago
      Optics aren’t free either.
    - j45 3y ago
      I’m currently working through this. My fibre internet is faster than the 2.5 GbE. A UDM-Pro could be added to the mix but simply pushes the issue out. A used datacenter/server grade SFP+ switch could be the way to go but have to get up to speed on the basics of the equipment I’d need. Open to ideas and suggestions!

MrFoof 3y ago

You can go down to 50W idle, but it requires some very specific hardware choices where the ROI will never materialize, some of which aren’t available yet for Zen4. I have… * AMD Ryzen 7 PRO 5750GE * 128GB ECC DDR4-3200 * Intel XL710-QDA2 (using QSFP+ to a quad-SFP+ passive DAC breakout) * LSI 9500-16i * Eight WD 16TB HDDs (shucked) * Two 2TB SK Hynix P41 Platinum M.2 NVMe SSD * Two Samsung 3.84TB PM9A3 U.2 NVMe SSD * Two Samsung 960GB PM893 SATA SSD So that’s the gist. Has a BMC, but dual 40GbE and can sustain about 55GbE over the network (in certain scenarios, 30-35GbE for almost all), running TrueNAS scale purely as a storage appliance for video editing, a Proxmox cluster (on 1L SFFs with 5750GEs and 10GbE idling at 10W each!) mostly running Apache Spark, a Pi4B 8Gb k3s cluster and lots more. Most of what talks to it is either 40GbE or 10GbE. There is storage tiering set up so the disks are very rarely hit, so they’re asleep most of the time. It mostly is serving data to or from the U.2s, shuffling it around automatically later on. The SATA SSDs are just metadata. It actually boots off a SuperMicro SuperDOM. —— The Zen 3 Ryzen PRO 5750GEs are unicorns, but super low power. Very tiny idle (they’re laptop cores), integrated GPU, ECC support, and the memory protection features of EPYC. 92% of the performance of a 5800X, but all 8C/16T flat out (at 3.95GHz because of an undervolt) caps at just under 39W package power. The LSI 9500-16i gave me all the lanes I needed (8 PCIe, 16 SlimSAS) for the two enterprise U.2 and 8 HDDs, and was very low idle power by being a newer adapter. The Intel dual QSFP+ NIC was deliberate as using passive DACs over copper saved 4-5W per port (8 at the aggregation switch) between the NIC and the switch. Yes, really. Plus lower latency (than even fiber) which matters at these transfer speeds. The “pig” is honestly the ASRock X570D4U because the BMC is 3.2W on its own, and X570 is a bit power hungry itself. But all in all, the whole system idles at 50W, is usually 75-80W under most loads, but can theoretically peak probably around 180-190W if everything was going flat out. It uses EVERY single PCIe lane available from the chipset and CPU to its fullest! Very specific chassis fan choices and Noctua low profile cooler in a super short depth 2U chassis. I’ve never heard it make a peep, disks aside :)

ianai 3y ago
Is the “e” for embedded? Ie needs to be bought in a package? I’m not seeing many market options.

dist-epoch 3y ago

> 3-5W per 8GB of RAM I think that's wrong. It would mean 4*6=25W per DIMM. I also have 48GB DDR5 DIMMs and HwInfo shows 6W max per module.

ThatMedicIsASpy 3y ago
Where are those 48G dimms? Ryzen 7000 limits 4 dimms to 3600MHz. To get high clocks 2 dimm configs is the maximum for Ryzen 7000. Are they in use? I can cut off 10-15W going to 4800MHz 1.1V (from 6400 1.3V)
- dist-epoch 3y ago
  > Ryzen 7000 limits 4 dimms to 3600MHz This is official support, just like 2 dimms are officially supported only at 5200MHz. People have been running them higher. How are you measuring the power? At the wall? I think you are measuring the memory controller too, not just the dimms.