4 ms·
It activates only 37B per query, but you don’t know which ones ahead of time, so you gotta store all 671B in (V)RAM.
by coolspot 2y ago
It activates only 37B per query, but you don’t know which ones ahead of time, so you gotta store all 671B in (V)RAM.
- cma 2y agoBut you don't need cluster networking or nvlink so much like with splitting out llama 405B. You could even split them out with friends over internet levels of bandwidth.