3 ms·
regarding the prompt processing and token generation you are correct it makes sense to benchmark them infependently since prompt processing is done in parralel
by discobot 3y ago
regarding the prompt processing and token generation you are correct
it makes sense to benchmark them infependently since prompt processing is done in parralel for each token and is compute bound and token generation is sequential and bound by memory banwidth