5 ms·
Code vs. data. The code needed create, train, and perform inference on a Transformer is quite short. How short depends on how you count the `import` statements
by ben_w 2y ago
Code vs. data.
The code needed create, train, and perform inference on a Transformer is quite short. How short depends on how you count the `import` statements in https://github.com/openai/gpt-2/blob/master/src/model.py https://github.com/openai/gpt-2/blob/master/src/model.py and https://github.com/openai/gpt-2/blob/master/src/sample.py https://github.com/openai/gpt-2/blob/master/src/sample.py etc.
Spreadsheets performing linear regression etc., — Do they learn? Sure!
If you accept that Transformers adapt and learn then you must accept that a spreadsheet also does, because someone implemented GPT-2 in Excel: https://github.com/ianand/spreadsheets-are-all-you-need https://github.com/ianand/spreadsheets-are-all-you-need
Do polymorphic computer viruses adapt? Border Gateway Protocol? Exponential backoff? Autocomplete? And that's aside from any algorithmic search results or "social" feeds, which are nothing but that.
- andoando 2y agoI am confused, a spreadsheet running the code for what a neural network does, sure. But a traditional computer program isn't just excel.
- szvsw 2y agoAny matrix multiplication can be trivially unwrapped into a bunch of for loops with basic arithmetic. In fact, that is exactly what any GPU kernel ultimately does in some sense. What differentiates a “traditional computer program” from a single matmul? How about a bunch of matmuls? How about a bunch of matmuls with some non-linear arithmetic thrown in between them? And if we are adding in sampling from a distribution but set temperature to zero and use random seeds so that it is fully deterministic?
- andoando 2y agoWhat you seem to be saying is that the machinary your traditional computer program executes on is capable of executing intelligence. What I said is thst a traditional computer program (say Excel) is not intelligent in itself, as it cannot, itself, without further input from a human, adapt.
- ben_w 2y ago> What I said is thst a traditional computer program (say Excel) is not intelligent in itself, as it cannot, itself, without further input from a human, adapt. Excel is as capable of adapting to new input as any AI, and the input need not come from a human specifically: the act of adapting to input is an automatic consequence of being able to perform linear regression. In fact, this is the real value of a spreadsheet over a piece of gridded paper where the sums are done by hand. If "adapting to input" is your test for intelligence, however, it is a test which a thermostat will pass.
- andoando 2y ago>Excel is as capable of adapting to new input as any AI, and the input need not come from a human specifically: the act of adapting to input is an automatic consequence of being able to perform linear regression. I said 'sensory input', as in, "any primary, real world data". Can I just download excel, and without feeding it machine learning algorithms, feed it a bunch of videos or text and have it learn history? >If "adapting to input" is your test for intelligence, however, it is a test which a thermostat will pass. Not just adapting to specific input the system is designed for, adapt it to ANY input. A thermostats function will never change on its own without modification, just like excels, or whatever other traditional program.
- ben_w 2y ago> and without feeding it machine learning algorithms It has some built-in. Linear regression is a machine learning technique.