Future deep learning hardware systems such as Nvidia's Grace system will use CPU memory in a hierarchy [0] (check the illustrations here). While HBM memory is

Future deep learning hardware systems such as Nvidia's Grace system will use CPU memory in a hierarchy [0] (check the illustrations here). While HBM memory is fast, it is limited in size (eg: 48GB-96GB), whereas DDRx memory can be in the TBs easily and that too inexpensively. You can think of the HBM-DDRx hierarchy analogously to the L1-L2-L(n) cache hierarchy for CPU processors. IIRC, even the CPU HBM implementations will use this HBM-DDRx hierarchy. [0] https://www.nextplatform.com/2021/04/12/nvidia-enters-the-arms-race-with-homegrown-grace-cpus/ https://www.nextplatform.com/2021/04/12/nvidia-enters-the-ar...