4 ms·
Xilinx HBM2 Internals (2023)
- willis936 2y agoI'm not an expert on memory interfaces. How do you use HBM2's 1024-bit interface when you have ~200 I/O on a zynq ultrascale+? Are these psuedo-channels a SerDes for the HBM2 bus?
- someguydave 2y agoLook at the (non-Zynq) VCU128 board for an example. The HBM2 is on the PL side, and the interconnect is via a die-to-die interface. So the 32 AXI3 interfaces to HBM2 here are hard silicon, not FPGA I/O pins.
- huntero 2y agoThe HBM stacks are on-package for these parts, so you don't have to use any external I/O to interface with them. You end up with a similar challenge accessing that much bandwidth internally from your FPGA logic though, it looks like the Xilinx HBM IP presents a set of 16 or 32 separate AXI interfaces, each of which gives you about 14.4GB/s of bandwidth (https://docs.amd.com/r/en-US/pg276-axi-hbm/Introduction https://docs.amd.com/r/en-US/pg276-axi-hbm/Introduction).
- TacticalCoder 2y ago> Conventions > MiB = Megabytes (2^20 bytes) > Gb = Gigabits (2^27 bytes, or 128MiB) > GiB = Gibibytes (2^30 bytes) Shouldn't MiB be Mebibytes then?
- Wolf9466 2y agoYeah, I messed that part up.
- deleted 2y ago[deleted]
- therealcamino 2y agoYeah, I found the Conventions section baffling. It defines both "giga" and "gibi" as 2^30! Why define both if they have the same value? Then it confuses the issue further by using "gibi" when the underlying unit is bytes, and "giga" when it is bits, which, given those definitions, doesn't convey any difference in meaning.
- pclmulqdq 2y agoI wonder if the author is doing anything to overclock the HBM here or if this is within the ratings of the Samsung HBM stacks. It's nice to be able to do this when you have a few cards, but if you are working with hundreds, it may not be practical to push the HBM this far without overvolting them a bit.
- latchkey 2y agoI automated the tuning of 150k gpus that were being used to mine ethereum. The trick was that as a whole, you knew the limits of the hardware. You know how to set the knobs to max performance. Due to the silicon lottery, cards that can't perform at max end up crashing. So what I did was kind of the opposite of what everyone else was doing. I first set everything at max, watched for a crash, then tuned the knobs to be a bit lower. All of this was done with an automated piece of software that I built. The cards we used essentially had 3 knobs to twist, which resulted in hundreds of combinations. Eventually, the cards stop crashing, so you're at the right settings, for that individual piece of hardware. We were running in seasonal climates too... so each winter/summer, I'd reset things and let it auto tune back again. Heat plays a huge factor on stability. Of course, each workload has different settings too... so that plays into it, but if everything else is static, this ended up being a great way to do things.
- rowanG077 2y agoThat seems great if a failure always results in a crash. There are a ton of failure modes where your result will just spuriously be wrong.
- latchkey 2y agoTo my knowledge, HPC rarely tunes cards for max performance. My MI300x are stock settings and I doubt I'll ever modify them.
- pclmulqdq 2y agoInteresting, I generally assumed Eth miners would undervolt their GPUs to get more life out of them rather than overclocking them for absolute max performance.
- akira2501 2y agoI feel like domains are pretty cheap so it would be easy to separate your fetishes from your work life.
- doctor_eval 2y agoI made the mistake of looking at the gallery. NSFW.
- formerly_proven 2y agoThere are no mistakes, just happy little accidents.
- doctor_eval 2y agoYou’re right, I shouldn’t have said mistake. The context switch nearly gave me whiplash, tho.
- deleted 2y ago[deleted]