3 ms·

Agree 100%. Have you noticed any lag with more "difficult" queries?

• bob1029 4y ago
  Yes. I think there are multiple levels of LLM for each use case now. The stuff that absolutely must run on 100B+ models can be pre-classified by something like Ada/Babbage. Attempts at arithmetic or dimensional analysis can be binned and shipped to a monster model. Anything that is more routine information retrieval needs maybe goes to a 7B parameter model. Lots of models with lots of temperature levels/hyperparameters/etc is the only way to achieve the kinds of performance we are seeing. The secret sauce is looking increasingly like a huge classification layer. Nothing you see in ChatGPT is as it appears. Lengthy conversations are managed with recursive summarization. Every conversational turn is potentially handled by a different LLM.