4 ms·

IMO LLMs are best when used as locally-run offline search engines. This is a clear and obvious disruptive technology. But we will need to get a lot better at f

IMO LLMs are best when used as locally-run offline search engines. This is a clear and obvious disruptive technology. But we will need to get a lot better at finetuning first. People don't want generalist LLMs, they want "expert systems".

danielbln 1y ago
Speak for yourself, I prefer generalist LLMs. Also, the bitter lesson of ML applies.
- otabdeveloper4 1y ago
  > I prefer generalist LLMs. No you don't. You personally probably don't need a Pokemon encyclopedia or Bengali spell checking in your daily LLM usage. These are the kinds of things you're paying for when you're using the newer models with huge parameter counts.
  - danielbln 1y ago
    while I probably won't need pokemon knowledge from an llm, I still think broad pre-training enables unpredictable cross-domain knowledge transfer - seemingly irrelevant data can contribute to reasoning abilities that improve specialized tasks. and most developers work across domains anyway, so better to start general and post-train than lose emergent capabilities from narrow foundations. even within 'coding' there's huge breadth. you might need to understand database schemas, api documentation, domain-specific algorithms, legacy system constraints, or business logic from completley different industries. a narrowly-tuned model might excel at leetcode problems but struggle when you need to build a financial trading system or whatever.
    - otabdeveloper4 1y ago
      We need 1000 different specialized LLMs, not one huge LLM with 1000 times more parameters. Whoever figures this out from a technical point of view will probably win this AI bubble iteration. Also, the big money is in local inference, not cloud chat. The pendulum will swing back towards personal computing. Whoever figures out the solution to this technical problem will also win. (I have no use whatsoever for C# or Java knowledge. I would pay for LLM weights that can figure a complex Nix codebase, preferably my own. I want to buy the weights though, cloud is a complete non-starter here.)

danielbln 1y ago

Speak for yourself, I prefer generalist LLMs. Also, the bitter lesson of ML applies.

otabdeveloper4 1y ago
> I prefer generalist LLMs. No you don't. You personally probably don't need a Pokemon encyclopedia or Bengali spell checking in your daily LLM usage. These are the kinds of things you're paying for when you're using the newer models with huge parameter counts.
- danielbln 1y ago
  while I probably won't need pokemon knowledge from an llm, I still think broad pre-training enables unpredictable cross-domain knowledge transfer - seemingly irrelevant data can contribute to reasoning abilities that improve specialized tasks. and most developers work across domains anyway, so better to start general and post-train than lose emergent capabilities from narrow foundations. even within 'coding' there's huge breadth. you might need to understand database schemas, api documentation, domain-specific algorithms, legacy system constraints, or business logic from completley different industries. a narrowly-tuned model might excel at leetcode problems but struggle when you need to build a financial trading system or whatever.
  - otabdeveloper4 1y ago
    We need 1000 different specialized LLMs, not one huge LLM with 1000 times more parameters. Whoever figures this out from a technical point of view will probably win this AI bubble iteration. Also, the big money is in local inference, not cloud chat. The pendulum will swing back towards personal computing. Whoever figures out the solution to this technical problem will also win. (I have no use whatsoever for C# or Java knowledge. I would pay for LLM weights that can figure a complex Nix codebase, preferably my own. I want to buy the weights though, cloud is a complete non-starter here.)