4 ms·

This is a fundamental misunderstanding of how LLMs work. You can’t really specialize a model. You specialize the harness. A well-trained general purpose LLM doe

This is a fundamental misunderstanding of how LLMs work. You can’t really specialize a model. You specialize the harness. A well-trained general purpose LLM doesn’t need examples in its training data, it can write good code in a new language you invented yesterday with just a spec definition. And it will perform better than a small model trained on lots of examples of your invented language. The reason is because of the “universal geometry of embeddings”, i.e all human languages have the same underlying pattern structure, so any model that is very good in any language is good in all languages. Attempting to specialize a model for a particular purpose often decreases overall performance. Fine-tuning is just a hack to make dumb models more reliable on limited tasks but they become incapable of doing anything else. Unless you are building a factory assembly line where a model is literally doing the same thing over and over, you almost always want a general purpose model over a specialized one.

chopete3 2mo ago
This is so right. We training Whisper Large model on 20,000 audio samples specific to a domain and it ended up reducing the ASR by 5% while improving WER of the finetuned domain by 0.5%. Instead we ended up with no finetuning. We give audio snippet to 2 AsR models, take 3 best transcriptions and ask the LLm to pick the best based on the context. That produced significantly higher accuracy in how an agent understands the users.
- twalla 2mo ago
  Can you go into more detail on this? I’ve been putting together the data pipeline for an ASR fine tune but your approach sounds more interesting.
  - teravor 2mo ago
    basically he is feeding the same input to multiple models, taking their outputs and dumping it into an LLM to sort out what the actual transcription probably is. expensive but effective.
- aetherspawn 2mo ago
  Deep Fusion is best, when words and phrase patterns in the domain are known. Deep Fusion means to hint the Whisper decoder about the next possible words using LLM-in-the-loop.
contingencies 2mo ago
Unless you are building a factory assembly line where a model is literally doing the same thing over and over, you almost always want a general purpose model over a specialized one. Turns out the world is made of simple, specialist processes, not generalists trying to achieve them. Adaptability may be of great benefit in evolutionary terms or for a walking anthropoid, but the majority of biology, chemistry, and mathematics rely upon specialist process for good reason. See also the old trope about robotics: that's what you call it before it works, otherwise it'd be a dishwasher. The upshot is: use a generalist to create a simple solution once, and scale that. Don't deploy the generalist at scale, that's a waste of resources and an inefficient solution.
weitendorf 2mo ago
You 100% can finetune or adapt/build on top of models, and specialize them or extend their capabilities. That’s literally what post training is. The problem is that “finetuning” was a 2023 AI FOTM associated with products/demos that were almost exclusively using it for LLM character role-play/output style purposes (ie not in actual systems where they served a more functional role). This made people think you could train models without replay/real evals by yoloing it with SFT (this is partially an artifact of that era being much heavier on autoregressive training and not so much evals). You really can finetune and get results but you have to treat it like a small ML training run, with real evals, and more intentionality than just “more examples”. You can find pretrained and -instruct models on huggingface that clearly demonstrate what specialization/staged training runs do. I’d be very wary of conflating finetuning with specialization/extending a model’s capabilities in general.
iugtmkbdfil834 2mo ago
<< all human languages have the same underlying pattern structure, Chomsky would like to have a word. Your statement is true only at a surface leve l( they have nouns, verbs and some limits), but it breaks down the moment you start to inspect it closely: - sign language ( I am not being petty; you put all languages ) - it is almost nothing like the underlying structure of other spoken languages primarily because it does not carry its restrictions - English vs Polish example - word can carry grammar or not; word order can carry meaning or not Those are just two examples, but both clearly show that little about human languages is actually the same. It is kinda like the history thing. It rhymes. I am addressing this part as other posters noted issues with other parts.
jhanschoo 2mo ago
> The reason is because of the “universal geometry of embeddings”, i.e all human languages have the same underlying pattern structure, so any model that is very good in any language is good in all languages. I think this claim is acceptable in context, but taken alone, this needs to be qualified. Some aspects of language are universal, like abstract information structure and other pragmatics, but no model is going to speak rural Khmer dialects well anytime soon because not enough of it has been digitized, fundamentally speaking, hence qualifying what it means to "be good at all languages".

chopete3 2mo ago

This is so right. We training Whisper Large model on 20,000 audio samples specific to a domain and it ended up reducing the ASR by 5% while improving WER of the finetuned domain by 0.5%. Instead we ended up with no finetuning. We give audio snippet to 2 AsR models, take 3 best transcriptions and ask the LLm to pick the best based on the context. That produced significantly higher accuracy in how an agent understands the users.

twalla 2mo ago
Can you go into more detail on this? I’ve been putting together the data pipeline for an ASR fine tune but your approach sounds more interesting.
- teravor 2mo ago
  basically he is feeding the same input to multiple models, taking their outputs and dumping it into an LLM to sort out what the actual transcription probably is. expensive but effective.
aetherspawn 2mo ago
Deep Fusion is best, when words and phrase patterns in the domain are known. Deep Fusion means to hint the Whisper decoder about the next possible words using LLM-in-the-loop.

contingencies 2mo ago

Unless you are building a factory assembly line where a model is literally doing the same thing over and over, you almost always want a general purpose model over a specialized one. Turns out the world is made of simple, specialist processes, not generalists trying to achieve them. Adaptability may be of great benefit in evolutionary terms or for a walking anthropoid, but the majority of biology, chemistry, and mathematics rely upon specialist process for good reason. See also the old trope about robotics: that's what you call it before it works, otherwise it'd be a dishwasher. The upshot is: use a generalist to create a simple solution once, and scale that. Don't deploy the generalist at scale, that's a waste of resources and an inefficient solution.

weitendorf 2mo ago

You 100% can finetune or adapt/build on top of models, and specialize them or extend their capabilities. That’s literally what post training is. The problem is that “finetuning” was a 2023 AI FOTM associated with products/demos that were almost exclusively using it for LLM character role-play/output style purposes (ie not in actual systems where they served a more functional role). This made people think you could train models without replay/real evals by yoloing it with SFT (this is partially an artifact of that era being much heavier on autoregressive training and not so much evals). You really can finetune and get results but you have to treat it like a small ML training run, with real evals, and more intentionality than just “more examples”. You can find pretrained and -instruct models on huggingface that clearly demonstrate what specialization/staged training runs do. I’d be very wary of conflating finetuning with specialization/extending a model’s capabilities in general.

iugtmkbdfil834 2mo ago

<< all human languages have the same underlying pattern structure, Chomsky would like to have a word. Your statement is true only at a surface leve l( they have nouns, verbs and some limits), but it breaks down the moment you start to inspect it closely: - sign language ( I am not being petty; you put all languages ) - it is almost nothing like the underlying structure of other spoken languages primarily because it does not carry its restrictions - English vs Polish example - word can carry grammar or not; word order can carry meaning or not Those are just two examples, but both clearly show that little about human languages is actually the same. It is kinda like the history thing. It rhymes. I am addressing this part as other posters noted issues with other parts.

jhanschoo 2mo ago

> The reason is because of the “universal geometry of embeddings”, i.e all human languages have the same underlying pattern structure, so any model that is very good in any language is good in all languages. I think this claim is acceptable in context, but taken alone, this needs to be qualified. Some aspects of language are universal, like abstract information structure and other pragmatics, but no model is going to speak rural Khmer dialects well anytime soon because not enough of it has been digitized, fundamentally speaking, hence qualifying what it means to "be good at all languages".