4 ms·

Note that your power consumption is more than just the CPU (combined TDP of 2x225W [0]). You also have to consider SSD (16x20W when fully loaded [1]), NIC (4x24

by vitus 2y ago

Note that your power consumption is more than just the CPU (combined TDP of 2x225W [0]). You also have to consider SSD (16x20W when fully loaded [1]), NIC (4x24W [2]), and the rest of the actual system itself (e.g. cooling, backplane). [0] https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-7713.html https://www.amd.com/en/products/processors/server/epyc/7003-... [1] I couldn't find 14TB enterprise SSDs on Intel's website, so I'm using the numbers from 6.4TB drives: https://ark.intel.com/content/www/us/en/ark/products/202708/intel-ssd-d7-p5600-series-6-4tb-2-5in-pcie-4-0-x4-3d3-tlc.html https://ark.intel.com/content/www/us/en/ark/products/202708/... [2] I'm not sure offhand which model number to use, but both models that support 200GbE on page 93-96 have this maximum wattage: https://docs.nvidia.com/nvidia-connectx-6-dx-ethernet-adapter-cards-for-ocp-spec-3-0-user-manual.pdf https://docs.nvidia.com/nvidia-connectx-6-dx-ethernet-adapte... Or, you can skip all the hand calculations and just fiddle with Dell's website to put together an order for a rack while trying to mirror the specs as closely as possible (I only included two NICs, since it complained that the configuration didn't have enough low-profile PCIe slots for four): https://www.dell.com/en-us/shop/dell-poweredge-servers/poweredge-r7525-rack-server/spd/poweredge-r7525/pe_r7525_tm_vi_vp_sb?configurationid=d6108dd6-0c9c-4dcb-9c13-5eed63f6510e6 https://www.dell.com/en-us/shop/dell-poweredge-servers/power... In this case, I selected a 1100W power supply and it's still yelling at me that it's not enough; 1400W is enough to make that nag go away.

throwbigdata 2y ago
That’s not how TDP works.
- vitus 2y ago
  You're not wrong, but it's still a nominally usable lower bound for the actual power draw of the chip, and a reasonable proxy for how much heat you need to dissipate via your cooling solution.
ksec 2y ago
Well I am assuming Memory and SSD being the same. The only different should be CPU + NIC since Bluefield itself is the NIC. May be Drewg123 could expand on that ( if he is allowed to )
- vitus 2y ago
  That is a fair point, as the 2x CPU + 4x NIC are "only" about 550W put together. There's probably more overhead for cooling (as much as 40% of the datacenter's power -- multiplying by 1.5x pushes you just over that 800W number). That said, being able to do 800G in an 800W footprint doesn't automatically mean that you can drive 100G in a 100W footprint. Not every ISP needs that 800G footprint, so being able to deploy smaller nodes can be an advantage. Also: I was assuming that 100W was the whole package (which is super impressive if so), since the Netflix serving model should have most of the SSDs in standby most of the time, and so you're allowed to cheat a little bit in terms of actual power draw vs max rating of the system.

by vitus 2y ago

throwbigdata 2y ago

That’s not how TDP works.

vitus 2y ago
You're not wrong, but it's still a nominally usable lower bound for the actual power draw of the chip, and a reasonable proxy for how much heat you need to dissipate via your cooling solution.

ksec 2y ago

Well I am assuming Memory and SSD being the same. The only different should be CPU + NIC since Bluefield itself is the NIC. May be Drewg123 could expand on that ( if he is allowed to )

vitus 2y ago
That is a fair point, as the 2x CPU + 4x NIC are "only" about 550W put together. There's probably more overhead for cooling (as much as 40% of the datacenter's power -- multiplying by 1.5x pushes you just over that 800W number). That said, being able to do 800G in an 800W footprint doesn't automatically mean that you can drive 100G in a 100W footprint. Not every ISP needs that 800G footprint, so being able to deploy smaller nodes can be an advantage. Also: I was assuming that 100W was the whole package (which is super impressive if so), since the Netflix serving model should have most of the SSDs in standby most of the time, and so you're allowed to cheat a little bit in terms of actual power draw vs max rating of the system.