4 ms·

IIRC there were some OpenAI docs that recommended doing exactly this, make n generations and use a smaller fine tuned model to select the best one

Tostino 3y ago
Right, most inference servers support this already.
DaiPlusPlus 3y ago
...does this directly relate to the high operating costs of LLMs-as-a-service, if for every request they have to run n-many redundant LLM requests? So if they could improve things so that a single prompt/request+response has a higher chance of being high-quality they wouldn't need to run alternatives?
- ilaksh 3y ago
  A lot of people don't run multiple at a time. It can make it more expensive if that option becomes popular. But I think in most cases batching is actually the biggest _improvement_ in terms of cost effectiveness for operators, since it enables them to use the parallel throughout of the graphics device more fully by handling multiple inference requests (often from different customers) at once. (Unless they work like Bard by default).
- MillionOClock 3y ago
  Another point: Now that I think about it, I doubt this is compatible with streaming the output to the user, which might be an issue in some cases.
  - DaiPlusPlus 3y ago
    I thought the char-by-char teleprinter thing was just an effect (y’know, for user-engagement and to make the interaction feel more genuine) - and that these systems just return output in buffered blocks/pages or whatever-it-is that they wired-up their network to do. It’s not like DALL-E outputs pixels in scanout order - or in brushstroke order (…er… or does it?)
    - ilaksh 3y ago
      It's not an effect at all. It calculates and outputs one token at time. The algorithm requires all previous tokens in order to output the next one. DALL-E is a totally different algorithm. It does not have a scanout or brushstrokes.

4 ms·

IIRC there were some OpenAI docs that recommended doing exactly this, make n generations and use a smaller fine tuned model to select the best one

by MillionOClock 3y ago

Tostino 3y ago
Right, most inference servers support this already.
DaiPlusPlus 3y ago
...does this directly relate to the high operating costs of LLMs-as-a-service, if for every request they have to run n-many redundant LLM requests? So if they could improve things so that a single prompt/request+response has a higher chance of being high-quality they wouldn't need to run alternatives?
- ilaksh 3y ago
  A lot of people don't run multiple at a time. It can make it more expensive if that option becomes popular. But I think in most cases batching is actually the biggest _improvement_ in terms of cost effectiveness for operators, since it enables them to use the parallel throughout of the graphics device more fully by handling multiple inference requests (often from different customers) at once. (Unless they work like Bard by default).
- MillionOClock 3y ago
  Another point: Now that I think about it, I doubt this is compatible with streaming the output to the user, which might be an issue in some cases.
  - DaiPlusPlus 3y ago
    I thought the char-by-char teleprinter thing was just an effect (y’know, for user-engagement and to make the interaction feel more genuine) - and that these systems just return output in buffered blocks/pages or whatever-it-is that they wired-up their network to do. It’s not like DALL-E outputs pixels in scanout order - or in brushstroke order (…er… or does it?)
    - ilaksh 3y ago
      It's not an effect at all. It calculates and outputs one token at time. The algorithm requires all previous tokens in order to output the next one. DALL-E is a totally different algorithm. It does not have a scanout or brushstrokes.