5 ms·
>All one will be able to say is that somewhere out there in the computational universe there’s some (typically computationally irreducible) process that “happen
by achrono 2y ago
>All one will be able to say is that somewhere out there in the computational universe there’s some (typically computationally irreducible) process that “happens” to be aligned with what we want.
>There’s no overarching theory to it in itself; it’s just a reflection of the resources that were out there. Or, in the case of machine learning, one can expect that what one sees will be to a large extent a reflection of the raw characteristics of computational irreducibility
Strikes me as a very reductive and defeatist take that flies in the face of the grand agenda Wolfram sets forth.
It would have been much more productive to chisel away at it to figure out something rather than expecting the Theory to be unveiled in full at once.
For instance, what I learn from the kinds of playing around that Wolfram does in the article is: neural nets are but one way to achieve learning & intellectual performance, and even within that there are a myriad different ways to do it, but most importantly: there is a breadth vs depth trade-off, in that neural nets being very broad/versatile are not quite the best at going deep/specialised; you need a different solution for that (e.g. even good old instruction set architecture might be the right thing in many cases). This is essentially why ChatGPT ended up needing Python tooling to reliably calculate 2+2.
- jstanley 2y ago> ChatGPT ended up needing Python tooling to reliably calculate 2+2. This is untrue. ChatGPT very reliably calculates 2+2 without invoking any tooling.
- achrono 2y agoSure, but I think you get my point.
- nyrikki 2y agoNit, it predicts that it is the token '4'. Token frequency in pre-training corpus and the way tokenization is implemented impacts arithmetic proficiency for LLMs. OpenAI calls this out in the GPT4 technical report.
- SkyBelow 2y agoYou can see this by giving it broken code and seeing what it can predict. I gave copilot a number of implementations of factorial with the input of 5. When it recognized the correct implementations, it was able to combine the ideas of "factorial", "5", and "correct implementation" to output 120. But when I gave it buggy implementations, it could recognize they were wrong, but the concepts of "factorial", "5", and "incorrect implementation" weren't enough for it to output the correct wrong result produced. Even when I explained its attempts to calculate the wrong output was itself wrong, it couldn't 'calculate' the right answer.
- e12e 2y agoThis makes very little sense (as a contrast to chatgpt predicted that the likely continuation of factorial and 5 is 120). Perhaps if you are able to share the chat session it's possible to see if you likely confused the issue with various factorial implementations - or got chatgpt to run your code with 5 as input? I mean the code is redundant: https://chatgpt.com/share/be249097-5067-4e3d-93c7-3eebedb510f3 https://chatgpt.com/share/be249097-5067-4e3d-93c7-3eebedb510...