3 ms·

We're getting large context windows, but so long as pricing is by the input token, the 'throw everything into the context window' path isn't viable. That prici

We're getting large context windows, but so long as pricing is by the input token, the 'throw everything into the context window' path isn't viable. That pricing model, and the context window limits, are a consequence of the quadratic cost of transformers though, and whatever the big context models like Gemini 1.5 are doing must have an (undisclosed) workaround. What needs to happen is a way to cheaply suspend and rehydrate the memory state of the forward pass after you've fed it a lot of tokens. That would be a sort of light-weight/flexible/easily modifiable/versionable/real-time-editable alternative to fine tuning. It's readily doable with the open weights LLM's, but none of them (yet) have the context length to make it really worthwhile (some of the coding LLM's have long context windows, but it doesn't solve the 'knowledge base' scenario). From a hosting perspective, if fine tunes are like VM's, such frozen overlays are like docker containers: many versions can live on the same server, sharing the base model and differing in the overlay layer. (a startup idea? who wants to collaborate on a proof of concept?)

spencerchubb 2y ago
When you describe the overlay layer, that sounds similar to the idea of low rank adaptation (LoRA). LoRA is kind of like finetuning, but it doesn't update every parameter, it adds a relatively small number of parameters and finetunes those Am I understanding what you're describing about the VMs and containers analogy?
- waldrews 2y ago
  Yup. I guess LoRA counts as fine tuning. Except I've never seen inference engines where they actually let you take the base model and the LoRA parameters as separate inputs (maybe it exists and I just haven't seen it). Instead, they bake the LoRA part into the bigger tensors as the final step of the fine tune. That makes sense in terms of making inference faster, but prevents the scenario where a host can just run the base model with any finetune you like, maybe switching them mid-conversation. Instead, if you want to host a fine-tuned model, you take the tensor blob and run a separate instance of the inference program on it. Incidentally, this is the one place where OpenAI and Azure pricing differs; OpenAI just charges you a big per-token premium for fine-tuned 3.5, and Azure charges you for the server to host the custom model. Likewise, the hosts for the open-weights models will charge you more to run your fine-tuned model than a standard model, even though it's the almost the same amount of GPU cycles, just because it needs to run on a separate server that won't be shared by multiple customers; that wouldn't be necessary if overlays were separated. I wouldn't be surprised if GPT-4's rumored mixture of many models does something like this overlay management internally.
msp26 2y ago
Great post. This exact limitation of web LLMs is why I'm leaning strongly towards local models for the easier stuff. Prompt caching can dramatically speed up fixed tasks. But frontier models are just too damn good and convenient so I don't think its possible to fully get away from web LLMs.

spencerchubb 2y ago

When you describe the overlay layer, that sounds similar to the idea of low rank adaptation (LoRA). LoRA is kind of like finetuning, but it doesn't update every parameter, it adds a relatively small number of parameters and finetunes those Am I understanding what you're describing about the VMs and containers analogy?

waldrews 2y ago
Yup. I guess LoRA counts as fine tuning. Except I've never seen inference engines where they actually let you take the base model and the LoRA parameters as separate inputs (maybe it exists and I just haven't seen it). Instead, they bake the LoRA part into the bigger tensors as the final step of the fine tune. That makes sense in terms of making inference faster, but prevents the scenario where a host can just run the base model with any finetune you like, maybe switching them mid-conversation. Instead, if you want to host a fine-tuned model, you take the tensor blob and run a separate instance of the inference program on it. Incidentally, this is the one place where OpenAI and Azure pricing differs; OpenAI just charges you a big per-token premium for fine-tuned 3.5, and Azure charges you for the server to host the custom model. Likewise, the hosts for the open-weights models will charge you more to run your fine-tuned model than a standard model, even though it's the almost the same amount of GPU cycles, just because it needs to run on a separate server that won't be shared by multiple customers; that wouldn't be necessary if overlays were separated. I wouldn't be surprised if GPT-4's rumored mixture of many models does something like this overlay management internally.

msp26 2y ago

Great post. This exact limitation of web LLMs is why I'm leaning strongly towards local models for the easier stuff. Prompt caching can dramatically speed up fixed tasks. But frontier models are just too damn good and convenient so I don't think its possible to fully get away from web LLMs.