7 ms·

I wouldn't say that's true at all for traditional computer programs. They're doing explicitly what they are designed to do, there is no adaptation/learning.

ben_w 2y ago
Code vs. data. The code needed create, train, and perform inference on a Transformer is quite short. How short depends on how you count the `import` statements in https://github.com/openai/gpt-2/blob/master/src/model.py https://github.com/openai/gpt-2/blob/master/src/model.py and https://github.com/openai/gpt-2/blob/master/src/sample.py https://github.com/openai/gpt-2/blob/master/src/sample.py etc. Spreadsheets performing linear regression etc., — Do they learn? Sure! If you accept that Transformers adapt and learn then you must accept that a spreadsheet also does, because someone implemented GPT-2 in Excel: https://github.com/ianand/spreadsheets-are-all-you-need https://github.com/ianand/spreadsheets-are-all-you-need Do polymorphic computer viruses adapt? Border Gateway Protocol? Exponential backoff? Autocomplete? And that's aside from any algorithmic search results or "social" feeds, which are nothing but that.
- andoando 2y ago
  I am confused, a spreadsheet running the code for what a neural network does, sure. But a traditional computer program isn't just excel.
  - szvsw 2y ago
    Any matrix multiplication can be trivially unwrapped into a bunch of for loops with basic arithmetic. In fact, that is exactly what any GPU kernel ultimately does in some sense. What differentiates a “traditional computer program” from a single matmul? How about a bunch of matmuls? How about a bunch of matmuls with some non-linear arithmetic thrown in between them? And if we are adding in sampling from a distribution but set temperature to zero and use random seeds so that it is fully deterministic?
    - andoando 2y ago
      What you seem to be saying is that the machinary your traditional computer program executes on is capable of executing intelligence. What I said is thst a traditional computer program (say Excel) is not intelligent in itself, as it cannot, itself, without further input from a human, adapt.
      - ben_w 2y ago
        > What I said is thst a traditional computer program (say Excel) is not intelligent in itself, as it cannot, itself, without further input from a human, adapt. Excel is as capable of adapting to new input as any AI, and the input need not come from a human specifically: the act of adapting to input is an automatic consequence of being able to perform linear regression. In fact, this is the real value of a spreadsheet over a piece of gridded paper where the sums are done by hand. If "adapting to input" is your test for intelligence, however, it is a test which a thermostat will pass.
        andoando 2y ago
        >Excel is as capable of adapting to new input as any AI, and the input need not come from a human specifically: the act of adapting to input is an automatic consequence of being able to perform linear regression. I said 'sensory input', as in, "any primary, real world data". Can I just download excel, and without feeding it machine learning algorithms, feed it a bunch of videos or text and have it learn history? >If "adapting to input" is your test for intelligence, however, it is a test which a thermostat will pass. Not just adapting to specific input the system is designed for, adapt it to ANY input. A thermostats function will never change on its own without modification, just like excels, or whatever other traditional program.
        ben_w 2y ago
        > and without feeding it machine learning algorithms It has some built-in. Linear regression is a machine learning technique.
Nevermark 2y ago
A structured program analyzing data as a graph, and optimizing access, is interacting with phenomenon, updating its working knowledge of the phenomenon, and can produce results that are very non-intuitive. Likewise, any symbolic mathematical system that accumulates theorems that speed up future work as it solves current tasks, seems like a high intelligence type of activity. Deep learning is “just” structured arithmetic. I think different kinds of intelligence can look quite different, and they will all be “structured” or “tropic” at their implementation levels. — Stepping away from the means, I see at least four “intelligence” dimensions: 1. Span: The span of novel situations for which it can create successful responses. 2. Efficiency: The efficiency of problem solving. I.e. When vast lookup tables, exhaustive combination searches, and indiscriminate logging of past experience can be matched instead by more efficient Boolean logic, arithmetic, pattern recognition and logging, we consider the latter more intelligent. 3. Abstraction: The degree to which solving previous different novel situations improves the success or efficiency in solving new problems. I.e. generalizable, composable learning. 4. Social education: Ability to communicate and absorb learned information from other entities. Plants, and I expect all surviving life forms, are very high in intelligence types 1 and 2. Adaptive nervous systems and especially brains excel at 3. Many animals, but most profoundly humans (whose languages for communicating are themselves actively adapted for compounding effects), excel at 4. Today’s humans are effectively more intelligent than 10,000 year ago humans, not because of 1-3, but because of 4. Learning as a child to read/write, do arithmetic, understand zero and negative numbers, and countless other information processing activities and patterns, from others, profoundly impacts our intellectual abilities. Deep learning, as with the human species, non-trivially spans, and continues to improve, on all 4 types of intelligence.
szvsw 2y ago
How do you define adaptation and learning? What about say, an autoscaler which is programmed to just track the the load for every hour over the last week, and use the average of the last 7 days at 8am to pre-emptively auto-scale? Is that learning and adapting? Alternatively, neural networks are also just doing explicitly what they are designed to do… sure there is a larger computational graph with lots of operations, but it’s all deterministic… backprop is not really much different on a procedural level than the simple fitting algorithm that I outlined above, in as much as it is just a specific well-defined algorithm or sequence of steps designed to compute some parameters from data.
- bloomingkales 2y ago
  What if we define adaptation and learning as the ability to concentrate. Our single cell ancestors would have to concentrate and deliberately store the first memory. Otherwise it would have just taken in the world with its sensors but never do anything with it. Adapting and learning means it chose to concentrate on packing the world into retrievable storage. When do we not adapt and learn? When we ignore our inputs and do nothing with it (don’t store it, don’t retrieve it). In the example you gave, those classical programs cannot concentrate, it’s a one and done.
  - szvsw 2y ago
    > When do we not adapt and learn? When we ignore our inputs and do nothing with it (don’t store it, don’t retrieve it). In the example I mentioned, the program clearly is taking inputs (load), storing them, querying for previous values that satisfy certain conditions (the value at a certain timestamp for each of the last 7 days), running computation (computing a mean), and operationalizing the result (pre-emptively scaling) to achieve a goal (avoiding having insufficient capacity) that affects the world (whoever interacting with the system will have different experiences because of the actions taken by the system). That seems like it satisfies your criterion to me.
    - bloomingkales 2y ago
      C’mon now, you and I know both know your program is discontinuous. A few null or unknown inputs that you didn’t even consider will break it. You’ll have to keep going in there and adding more if/else statements. Your program couldn’t survive a 100 years without you debugging it.
empath75 2y ago
You can construct a "computer" that learns and adapts with some matchboxes and buttons. https://en.wikipedia.org/wiki/Matchbox_Educable_Noughts_and_Crosses_Engine https://en.wikipedia.org/wiki/Matchbox_Educable_Noughts_and_... I wouldn't say that's more "intelligent" than, I don't know, a workplace scheduling system.

ben_w 2y ago

Code vs. data. The code needed create, train, and perform inference on a Transformer is quite short. How short depends on how you count the `import` statements in https://github.com/openai/gpt-2/blob/master/src/model.py https://github.com/openai/gpt-2/blob/master/src/model.py and https://github.com/openai/gpt-2/blob/master/src/sample.py https://github.com/openai/gpt-2/blob/master/src/sample.py etc. Spreadsheets performing linear regression etc., — Do they learn? Sure! If you accept that Transformers adapt and learn then you must accept that a spreadsheet also does, because someone implemented GPT-2 in Excel: https://github.com/ianand/spreadsheets-are-all-you-need https://github.com/ianand/spreadsheets-are-all-you-need Do polymorphic computer viruses adapt? Border Gateway Protocol? Exponential backoff? Autocomplete? And that's aside from any algorithmic search results or "social" feeds, which are nothing but that.

andoando 2y ago
I am confused, a spreadsheet running the code for what a neural network does, sure. But a traditional computer program isn't just excel.
- szvsw 2y ago
  Any matrix multiplication can be trivially unwrapped into a bunch of for loops with basic arithmetic. In fact, that is exactly what any GPU kernel ultimately does in some sense. What differentiates a “traditional computer program” from a single matmul? How about a bunch of matmuls? How about a bunch of matmuls with some non-linear arithmetic thrown in between them? And if we are adding in sampling from a distribution but set temperature to zero and use random seeds so that it is fully deterministic?
  - andoando 2y ago
    What you seem to be saying is that the machinary your traditional computer program executes on is capable of executing intelligence. What I said is thst a traditional computer program (say Excel) is not intelligent in itself, as it cannot, itself, without further input from a human, adapt.
    - ben_w 2y ago
      > What I said is thst a traditional computer program (say Excel) is not intelligent in itself, as it cannot, itself, without further input from a human, adapt. Excel is as capable of adapting to new input as any AI, and the input need not come from a human specifically: the act of adapting to input is an automatic consequence of being able to perform linear regression. In fact, this is the real value of a spreadsheet over a piece of gridded paper where the sums are done by hand. If "adapting to input" is your test for intelligence, however, it is a test which a thermostat will pass.
      - andoando 2y ago
        >Excel is as capable of adapting to new input as any AI, and the input need not come from a human specifically: the act of adapting to input is an automatic consequence of being able to perform linear regression. I said 'sensory input', as in, "any primary, real world data". Can I just download excel, and without feeding it machine learning algorithms, feed it a bunch of videos or text and have it learn history? >If "adapting to input" is your test for intelligence, however, it is a test which a thermostat will pass. Not just adapting to specific input the system is designed for, adapt it to ANY input. A thermostats function will never change on its own without modification, just like excels, or whatever other traditional program.
        ben_w 2y ago
        > and without feeding it machine learning algorithms It has some built-in. Linear regression is a machine learning technique.

Nevermark 2y ago

A structured program analyzing data as a graph, and optimizing access, is interacting with phenomenon, updating its working knowledge of the phenomenon, and can produce results that are very non-intuitive. Likewise, any symbolic mathematical system that accumulates theorems that speed up future work as it solves current tasks, seems like a high intelligence type of activity. Deep learning is “just” structured arithmetic. I think different kinds of intelligence can look quite different, and they will all be “structured” or “tropic” at their implementation levels. — Stepping away from the means, I see at least four “intelligence” dimensions: 1. Span: The span of novel situations for which it can create successful responses. 2. Efficiency: The efficiency of problem solving. I.e. When vast lookup tables, exhaustive combination searches, and indiscriminate logging of past experience can be matched instead by more efficient Boolean logic, arithmetic, pattern recognition and logging, we consider the latter more intelligent. 3. Abstraction: The degree to which solving previous different novel situations improves the success or efficiency in solving new problems. I.e. generalizable, composable learning. 4. Social education: Ability to communicate and absorb learned information from other entities. Plants, and I expect all surviving life forms, are very high in intelligence types 1 and 2. Adaptive nervous systems and especially brains excel at 3. Many animals, but most profoundly humans (whose languages for communicating are themselves actively adapted for compounding effects), excel at 4. Today’s humans are effectively more intelligent than 10,000 year ago humans, not because of 1-3, but because of 4. Learning as a child to read/write, do arithmetic, understand zero and negative numbers, and countless other information processing activities and patterns, from others, profoundly impacts our intellectual abilities. Deep learning, as with the human species, non-trivially spans, and continues to improve, on all 4 types of intelligence.

szvsw 2y ago

How do you define adaptation and learning? What about say, an autoscaler which is programmed to just track the the load for every hour over the last week, and use the average of the last 7 days at 8am to pre-emptively auto-scale? Is that learning and adapting? Alternatively, neural networks are also just doing explicitly what they are designed to do… sure there is a larger computational graph with lots of operations, but it’s all deterministic… backprop is not really much different on a procedural level than the simple fitting algorithm that I outlined above, in as much as it is just a specific well-defined algorithm or sequence of steps designed to compute some parameters from data.

bloomingkales 2y ago
What if we define adaptation and learning as the ability to concentrate. Our single cell ancestors would have to concentrate and deliberately store the first memory. Otherwise it would have just taken in the world with its sensors but never do anything with it. Adapting and learning means it chose to concentrate on packing the world into retrievable storage. When do we not adapt and learn? When we ignore our inputs and do nothing with it (don’t store it, don’t retrieve it). In the example you gave, those classical programs cannot concentrate, it’s a one and done.
- szvsw 2y ago
  > When do we not adapt and learn? When we ignore our inputs and do nothing with it (don’t store it, don’t retrieve it). In the example I mentioned, the program clearly is taking inputs (load), storing them, querying for previous values that satisfy certain conditions (the value at a certain timestamp for each of the last 7 days), running computation (computing a mean), and operationalizing the result (pre-emptively scaling) to achieve a goal (avoiding having insufficient capacity) that affects the world (whoever interacting with the system will have different experiences because of the actions taken by the system). That seems like it satisfies your criterion to me.
  - bloomingkales 2y ago
    C’mon now, you and I know both know your program is discontinuous. A few null or unknown inputs that you didn’t even consider will break it. You’ll have to keep going in there and adding more if/else statements. Your program couldn’t survive a 100 years without you debugging it.

empath75 2y ago

You can construct a "computer" that learns and adapts with some matchboxes and buttons. https://en.wikipedia.org/wiki/Matchbox_Educable_Noughts_and_Crosses_Engine https://en.wikipedia.org/wiki/Matchbox_Educable_Noughts_and_... I wouldn't say that's more "intelligent" than, I don't know, a workplace scheduling system.