3 ms·
(For a model of GPT-4's size, it could also be 8 nodes with several GPUs each, each node comprising a single expert.)
by cjbprime 2y ago
(For a model of GPT-4's size, it could also be 8 nodes with several GPUs each, each node comprising a single expert.)