3 ms·
Reasoning means "inference" or "deduction" to me, or at least some process related to first order logic.
by astrobe_ 2y ago
Reasoning means "inference" or "deduction" to me, or at least some process related to first order logic.
- nyrikki 2y agoThe known upper bound for transformers on the fly computation abilities is a complexity class called DLOGTIME-uniform TC^0. There is a lot to unpack there but if you take FO as being closed under conjunction (∧), negation (¬) and universal quantification (∀); you will find that DLOGTIME-uniform TC^0 is equal to FO+Majority Gates. So be careful about that distinction. To help break the above down: DLOGTIME = Constructible by a RAM or TM in logarithmic time. uniform = Only one circuit for all input sizes, when circuits families are the default convention TC^0: Constant-Depth Threshold Circuits Even NP == SO-E, the second-order queries where the second-order quantifiers are only existantials. DLOGTIME-uniform TC^0 is a WAY smaller group than most people realize, but anything that is an algorithm or a program basically is logic, with P being FO + transitive closure or a half a dozen other known mappings. Transformers can figure out syntax, but if you dig into that dlogtime part, you will see that semantic correctness isn't really an option...thus the need to leverage the pattern matching and finding of pre-training as much as possible.
- andai 2y agoThanks. If I'm reading this right, the limiting factor on the intelligence of current LLMs is not the network size, nor training data (size/quality) but rather the architecture? Do we know of a better one for complex computations / "reasoning"?