3 ms·
> ...generate answers near the center of existing thought. This is right in the Wikipedia's article on universal approximation theorem [1]. [1] https://en.wik
by thesz 7mo ago
> ...generate answers near the center of existing thought.
This is right in the Wikipedia's article on universal approximation theorem [1].
[1] https://en.wikipedia.org/wiki/Universal_approximation_theorem https://en.wikipedia.org/wiki/Universal_approximation_theore...
"n the field of machine learning, the universal approximation theorems (UATs) state that neural networks with a certain structure can, in principle, approximate any continuous function to any desired degree of accuracy. These theorems provide a mathematical justification for using neural networks, assuring researchers that a sufficiently large or deep network can model the complex, non-linear relationships often found in real-world data."
And then: "Notice also that the neural network is only required to approximate within a compact set K {\displaystyle K}. The proof does not describe how the function would be extrapolated outside of the region."
NNs, LLMs included, are interpolators, not extrapolators.
And the region NN approximates within can be quite complex and not easily defined as "X:R^N drawn from N(c,s)^N" as SolidGoldMagiKarp [2] clearly shows.
[2] https://github.com/NiluK/SolidGoldMagikarp https://github.com/NiluK/SolidGoldMagikarp
- fasterik 7mo agoIt has been proven that recurrent neural networks are Turing complete [0]. So for every computable function, there is a neural network that computes it. That doesn't say anything about size or efficiency, but in principle this allows neural networks to simulate a wide range of intelligent and creative behavior, including the kind of extrapolation you're talking about. [0] https://www.sciencedirect.com/science/article/pii/S0022000085710136 https://www.sciencedirect.com/science/article/pii/S002200008...
- gmueckl 7mo agoTuring conpleteness is not associated with crativity or intelligence in any ateaightforward manner. One cannot unconditionally imply the other.
- marcus_holmes 7mo agoafter all, CSS is Turing Complete ;) https://stackoverflow.com/questions/2497146/is-css-turing-complete https://stackoverflow.com/questions/2497146/is-css-turing-co...
- vidarh 7mo agoNo, but unless you find evidence to suggest we exceed the Turing computable, Turing completeness is sufficient to show that such systems are not precluded from creativity or intelligence.
- thesz 7mo agoI believe that quantum oracles are more powerful than Turing oracles, because quantum oracles can be constructed, from what I understand, and Turing oracles need infinite tape. Our brains use quantum computation within each neuron [1]. [1] https://www.nature.com/articles/s41598-024-62539-5 https://www.nature.com/articles/s41598-024-62539-5
- vidarh 7mo agoThere's no evidence to suggest a quantum computer exceeds the Turing computable.
- thesz 7mo agoThe difference is quantum oracles can be constructed [1] and Turing oracle can't be [2]: "An oracle machine or o-machine is a Turing a-machine that pauses its computation at state "o" while, to complete its calculation, it "awaits the decision" of "the oracle"—an entity unspecified by Turing "apart from saying that it cannot be a machine" (Turing (1939)." [1] https://arxiv.org/abs/2303.14959 [2] https://en.wikipedia.org/wiki/Turing_machine
- vidarh 7mo agoThis is meaningless. A Turing machine is defined in terms of state transitions. Between those state transitions, there is a pause in computation at any point where the operations takes time. Those pauses are just not part of the definition because they are irrelevant to the computational outcome. And given we have no evidence that quantum oracles exceeds the Turing computable, all the evidence we have suggests that they are Turing machines.
- legulere 7mo agoI think you cannot take the step from any turing machine being representable as a neural network to say anything about the prowess of learned neural networks instead of specifically crafted ones. I think a good example are calculations or counting letters: it's trivial to write turing machines doing that correctly, so you could create neural networks, that do just that. From LLM we know that they are bad at those tasks.
- chongli 7mo agoSo for every computable function, there is a neural network that computes it. That doesn't say anything about size or efficiency It also doesn't say anything about finding the desired function, rather than a different function which approximates it closely on some compact set but diverges from it outside that set. That's the trouble with extrapolation: you don't know how to compute the function you're looking for because you don't know anything about its behaviour outside of your sample.
- thesz 7mo agoJudging from what I read, their work is subject to regular hardware constraints, such as limited stack size. Because paper describes a mapping from regular hardware circuits to the continuous circuits. As an example, I would like to ask how to parse balanced brackets grammar (S ::= B <EOS>; B ::= | BB | (B) | [B] | {B};) with that Turing complete recurrent network and how it will deal with precision loss for relatively short inputs. Paper also does not address training (i.e., automatic search of the processors' equations given inputs and outputs).
- tovej 7mo agoNo, the size of those networks that would be capable of that are infeasible. That's a common fallacy. You hint at this but then dismiss it. Mathematically possible != actually possible.
- quikoa 7mo ago> approximate any continuous function It wouldn't surprise me if many interesting functions we'd like to approximate aren't continuous at all.
- thesz 7mo agoThis is what softmax [1] is for. [1] https://en.wikipedia.org/wiki/Softmax_function https://en.wikipedia.org/wiki/Softmax_function
- jandrewrogers 7mo agoThis is one of the reasons current AI tech is so poor at learning physical world dynamics. Relationships in the physical world are sparse, metastable graphs with non-linear dynamics at every resolution. And then we measure these dynamics using sparse, irregular sampling with a high noise floor. It is just about the worst possible data model for conventional AI stacks at a theoretical level.