2 ms·
And potentially for distributing a model across several devices at inference time. You could devote a cluster of smaller/weaker machines to inference.
by entropicdrifter 3y ago
And potentially for distributing a model across several devices at inference time. You could devote a cluster of smaller/weaker machines to inference.
- sroussey 3y agoYou can do that today, the only advantage today though is being able to fix the model in memory. It’s sequential and slower due to communication costs, though batching might be faster?