3 ms·
Problem is prefill on these, right? Initial prompt processing takes forever? I suppose you’re right. Cost is not a thing on its own. It’s a performance-cost fro
by arjie 3mo ago
Problem is prefill on these, right? Initial prompt processing takes forever? I suppose you’re right. Cost is not a thing on its own. It’s a performance-cost frontier and one can do CPU inference in the worst case.