16 ms·
Developing an intuition about high dimensional systems is pretty different from understanding the character of some specific point on a 1e9+ dimensional manifol
by tech_ken 1mo ago
Developing an intuition about high dimensional systems is pretty different from understanding the character of some specific point on a 1e9+ dimensional manifold of parameters, in my professional opinion (setting aside all the degrees of freedom that come from the structure of the thing). Sure one can understand generic principles like the curse of dimensionality, but truly groking how an LLM works is basically an open problem as far as I'm aware. I'm not saying there's no benefit for amateurs to study how LLMs work, but let's be realistic about how far mere intuition can truly take anyone in this space.
- deleted 1mo ago[deleted]
- s1artibartfast 1mo agoSeems like a silly goalpost. As you say, there's a lot of value to unlock with understanding the generic principles, and I would add specific application. Lots of people are building fantastic Tools or pulling down million dollar salaries without groking the precise representation of a single weight.
- deleted 1mo ago[deleted]
- tech_ken 1mo ago> Lots of people are building fantastic Tools or pulling down million dollar salaries without groking the precise representation of a single weight. I'll borrow my response from a far greater applied mathematician than myself: https://www.youtube.com/watch?v=_oNgyUAEv0Q https://www.youtube.com/watch?v=_oNgyUAEv0Q