4 ms·
They're memory bandwidth limited, you can basically just estimate the performance from the time it takes to read the entire model from ram for each token.
by nullc 2y ago
They're memory bandwidth limited, you can basically just estimate the performance from the time it takes to read the entire model from ram for each token.