3 ms·
6 Levels of algorithm people confuse: 1. The Model Architecture. Calculation of outputs from inputs. 2. The Training Algorithm, that alters parameters in the
by Nevermark 9mo ago
6 Levels of algorithm people confuse:
1. The Model Architecture. Calculation of outputs from inputs.
2. The Training Algorithm, that alters parameters in the architecture based on training data, often input, outputs vs. targets, but can be more complex than that. I.e. gradient descent, etc.
3. The Class of Problem being solved, i.e. approximation, prediction, etc.
4. The Instance of Problem of the problem being solved, i.e. approximation of chemical reaction completion vs. temperature, or prediction of textual responses.
5. The Data Embodiment of the problem, i.e. the actual data. How much, how complex, how general, how noisey, how accurate, how variable, how biased, ...?
And only after all those,
6. The Learned Algorithm that emerges from continual exposure to (5) in the basic form of (3), in order to specifically perform (4), by applying algorithm (2), to the model's parameters that control its input-output algorithm (1).
The latter, (6), has no limit in complexity or quality, or types of sub-problems, that must also be solved, to solve the umbrella problem successfully.
Data can be unbounded in complexity. Therefore, actual (successful) solutions are necessarily unbounded in complexity.
The "no limit, unbounded, any kind" of sub-problem part of (6) is missed by many people. To perform accurate predictions, of say the whole stock market, would require a model to learn everything from economic theory, geopolitics, human psychology, natural resources and their extraction, crime, electronic information systems and their optimizations, game theory, ...
That isn't a model I would call "just a stock price predictor".
Human language is an artifact created by complex beings. A high level of understanding of how those complex beings operate in conversation, writing, speeches, legal theory, their knowledge of 1000's of topics, psychologies, cultures, assumptions, motives, lifetime development, their modeling of each other, ... on and on ... becomes necessary to mimic general written artifacts between people with any resemblance at all.
LLM's, at the first point of being useful, were never "just" prediction machines.
I am still astonished there were ever technical people saying such a thing.