3 ms·
I'm curious how Mozilla is preventing their inference from being too greedy. Are they doing something with `nice`? If I ask it to translate or summarize, I obvi
by bastawhiz 1y ago
I'm curious how Mozilla is preventing their inference from being too greedy. Are they doing something with `nice`? If I ask it to translate or summarize, I obviously want it done pronto, but otherwise it's almost certainly the case that slower inference that doesn't interrupt anything else is preferable. Since inference is almost always intensive, this seems like table stakes so I can't imagine they did nothing.