4 ms·

Well I think it's an interesting meta-question. Humans are good at figuring out why something works then creating a higher level abstraction that can be applie

by mason55 10y ago

Well I think it's an interesting meta-question. Humans are good at figuring out why something works then creating a higher level abstraction that can be applied in lots of different situations. This saves us time because we don't have to re-derive the rules or constants each time we encounter a similar problem. We figure out "why" something works so that we can understand when to apply our abstraction to other problems that might or might not be similar in a meaningful way. For a fast enough computer, there is little reason for it to develop any kind of high-level abstraction. It can re-derive the optimal solution whenever it needs it and the benefit is that it might be able to avoid local maxima that we would get stuck on because it doesn't try to apply an abstraction that only partially fits. So when we look at something like this and say "we can't know why the computer did it this way" it almost doesn't make sense. The computer did what it did because it figured out that was the best solution - it doesn't understand the concept of "why".

Cyph0n 10y ago
> So when we look at something like this and say "we can't know why the computer did it this way" it almost doesn't make sense. The computer did what it did because it figured out that was the best solution - it doesn't understand the concept of "why". This idea is what I don't agree with. At the most basic level, the entire "thought space" of a computer lies within the instructions fed to it by a human programmer. Until we can figure out how to build a general AI, whatever the computer decides to do or ends up doing is completely bounded by the content of the instructions. This implies that, given enough resources, a human equipped with a debugger and an understanding of the instructions e.g. algorithm could reverse exactly why and how a program arrived at a particular result. > it [the program] doesn't understand the concept of "why" Oh yeah, it doesn't "understand" the concept, but that doesn't mean it doesn't make decisions (branches) based on some parameters (inputs or intermediate values). In other words, there is in fact a "why"! So if we can step through the sequence of branches executed and their results, we can understand "why" even if the program itself can't do that.
- linkregister 10y ago
  Once the number of instructions approaches a certain point, it will increase the cost of human analysis drastically. From my previous work performing reverse engineering, I could take comfort that I was reconstructing algorithms designed by humans, compiled and assembled by programs written by humans, with really pleasantly un-optimized properties (frame pointers in some assemblies, loading and saving registers at the start & end of functions, logically-separated functions). My job would be much harder if I had to deobfuscate code that was optimized without a strict ruleset. After enough spaghetti assembly, it would be too time-consuming to reverse engineer code except for the most profitable enterprises (interoperability, vulnerability research for a very important bug). I'm using the following mental model of what machine learning-generated code would look like: https://news.ycombinator.com/item?id=8092359 https://news.ycombinator.com/item?id=8092359 (an evolutionary algorithm designed a circuit that is extremely difficult to analyze but is optimal at achieving its narrow purpose).
- thaumasiotes 10y ago
  > At the most basic level, the entire "thought space" of a computer lies within the instructions fed to it by a human programmer. Until we can figure out how to build a general AI, whatever the computer decides to do or ends up doing is completely bounded by the content of the instructions. (emphasis mine) How would this change if you built a "general AI"? Assume I don't believe that a computer, even a general AI, is ensouled.
  - Cyph0n 10y ago
    If we assume that a general AI can "understand" things in general and can "learn" over time, there is nothing stopping it from understanding the instructions it consists of, and subsequently learning how it can dynamically reprogram itself. If we extend that further, the program could also potentially obfuscate its activity by detecting logging or debugging activity. That's the way I think about it at least.
    - thaumasiotes 10y ago
      All of that is still part of its original instructions.

by mason55 10y ago

Cyph0n 10y ago

> So when we look at something like this and say "we can't know why the computer did it this way" it almost doesn't make sense. The computer did what it did because it figured out that was the best solution - it doesn't understand the concept of "why". This idea is what I don't agree with. At the most basic level, the entire "thought space" of a computer lies within the instructions fed to it by a human programmer. Until we can figure out how to build a general AI, whatever the computer decides to do or ends up doing is completely bounded by the content of the instructions. This implies that, given enough resources, a human equipped with a debugger and an understanding of the instructions e.g. algorithm could reverse exactly why and how a program arrived at a particular result. > it [the program] doesn't understand the concept of "why" Oh yeah, it doesn't "understand" the concept, but that doesn't mean it doesn't make decisions (branches) based on some parameters (inputs or intermediate values). In other words, there is in fact a "why"! So if we can step through the sequence of branches executed and their results, we can understand "why" even if the program itself can't do that.

linkregister 10y ago
Once the number of instructions approaches a certain point, it will increase the cost of human analysis drastically. From my previous work performing reverse engineering, I could take comfort that I was reconstructing algorithms designed by humans, compiled and assembled by programs written by humans, with really pleasantly un-optimized properties (frame pointers in some assemblies, loading and saving registers at the start & end of functions, logically-separated functions). My job would be much harder if I had to deobfuscate code that was optimized without a strict ruleset. After enough spaghetti assembly, it would be too time-consuming to reverse engineer code except for the most profitable enterprises (interoperability, vulnerability research for a very important bug). I'm using the following mental model of what machine learning-generated code would look like: https://news.ycombinator.com/item?id=8092359 https://news.ycombinator.com/item?id=8092359 (an evolutionary algorithm designed a circuit that is extremely difficult to analyze but is optimal at achieving its narrow purpose).
thaumasiotes 10y ago
> At the most basic level, the entire "thought space" of a computer lies within the instructions fed to it by a human programmer. Until we can figure out how to build a general AI, whatever the computer decides to do or ends up doing is completely bounded by the content of the instructions. (emphasis mine) How would this change if you built a "general AI"? Assume I don't believe that a computer, even a general AI, is ensouled.
- Cyph0n 10y ago
  If we assume that a general AI can "understand" things in general and can "learn" over time, there is nothing stopping it from understanding the instructions it consists of, and subsequently learning how it can dynamically reprogram itself. If we extend that further, the program could also potentially obfuscate its activity by detecting logging or debugging activity. That's the way I think about it at least.
  - thaumasiotes 10y ago
    All of that is still part of its original instructions.