4 ms·
I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active rese
by Keyframe 2mo ago
I don't care who King Charles is every single time
that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.
- IdontKnowRust 2mo agoThis makes me think (by full ignorance) how quantum computing can be useful in this field of things reach a stable-ish point
- ComputerPerson 2mo agoSeems to me like there's a conflict of incentives in the current era of market capture. Not to say the cost-cutting wouldn't be valuable; today's race is predominantly about the model's reasoning capacity or "how hard of a math problem can the model solve". It'll be a nice day when research-oriented human capital gets redirected to things that benefit us layfolks's pockets more directly
- spot5010 2mo agoThe human analogy would be that I don’t remember everything in the books I have read, but I do recall reading a particular book and can always look it up. So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence.
- Gigachad 2mo agoLLMs can do that. The one that's built in to google search is relatively stupid, but it just uses web pages to fill in the gaps. The downside is it is incredibly easy to be tricked by a single false bit of info.
- troyzhxu 2mo ago[flagged]
- sdlion 2mo agoI guess that's why is an active research area, and an interdisciplinary one, what is intelligence? Is what have you practiced a ton of times? Is what you have purposely and efficiently practice? What is the implication of "purposely" doing something and how much memory is involved into it? If memory is involved how much of it is relevant? What is problem solving or wisdom? creativity? How much diversity in your memory do you need for creativity? So what should you cull and how much? There's already techniques in CNNs to trim unused or less active neuron paths to reduce a model's size, but how do you (and how much) do it in a general LLM? A product that will be used without supervision from a child to an 80yo elder?
- mandeepj 2mo ago> So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence. MoE (mixture of experts)?
- fhn 2mo agoI read a lot in middle school and can't recall any of them. What books did you read in college? What books did you read in high school?Maybe you have photographic memory.
- alexander2002 2mo agocant we have specialised models during onboarding i doubt a developer needs to know who king charles is?
- Keyframe 2mo agoWe can and we do https://en.wikipedia.org/wiki/Mixture_of_experts https://en.wikipedia.org/wiki/Mixture_of_experts and even then - maybe not every developer needs python, maybe some do need C++ and King Charles..
- antihipocrat 2mo agoClaude, create a webpage that provides family tree diagrams of all European royal families, clicking on each name expands the element to include any notable events from that person's life.
- kstenerud 2mo agoI've been dealing with it on the post-training (runtime) side with a large codebase that contains many moving parts, many rules and requirements. Putting all of that into the context has already gone far past 1MB, so it's untenable. Now I have all of that information stored in a pointers-to-resources fashion, where layers of small "directories" of triggers-to-information point the agent gradually towards deeper, more esoteric knowledge the more specific its needs become when tackling a goal.
- sznio 2mo agodistill. start with a general model, then create a smaller model trained only on relevant coding examples.
- giancarlostoro 2mo agoI have some ideas on how that specifically can be solved, but I've taken a stance to never give OpenAI or Anthropic any of my ideas for free. I am the most surprised that Google seems to be trailing behind them. I'm not sure if they're even taking this seriously anymore. I do appreciate the open models they do release on the other hand, I hope they never stop. I wish Microsoft would do more with Phi and similar.
- berbec 2mo ago> I have some ideas on how that specifically can be solved Write them in the margin of a book.... "It is impossible for a cube to be a sum of two cubes, a fourth power to be a sum of two fourth powers, or in general for any number that is a power greater than the second to be the sum of two like powers. I have discovered a truly remarkable proof, but this margin is too small to contain it."
- fhn 2mo agoEmail it to my gmail account. It's secure from prying eyes :)