2 ms·
A half dozen GPT-4 instances
by HeatrayEnjoyer 2y ago
A half dozen GPT-4 instances
- metadat 2y agoLLM inference processors (GPUs) don't use DDR, it uses special, costly stacked HBM ram soldered to the board. I tested out running Llama on a 512GB machine, it's rather slow and inefficient. Maybe 1-token/sec.