4 ms·
It should work. After the initial prompt processing, token generation is typically limited by memory bandwidth more than by compute.
by atq2119 3y ago
It should work. After the initial prompt processing, token generation is typically limited by memory bandwidth more than by compute.