7 ms·
If host models for others, then sure, I'm happy to save some computation time for really simple queries. Sure the cost is that the model will be effectively dis
by vintermann 1y ago
If host models for others, then sure, I'm happy to save some computation time for really simple queries. Sure the cost is that the model will be effectively dismissive of questions it judges to be "easy", but I'm not the one carrying that cost I suppose.
However, for a local model, answering my own queries? That's the last thing I want. I already spent way too much money on that GPU, might as well get use out of it.