4 ms·

What’s a plausible example of what an explanation of “how it actually works” would be? Like even a wrong example, but following the rough syntax or shape of wha

What’s a plausible example of what an explanation of “how it actually works” would be? Like even a wrong example, but following the rough syntax or shape of what a correct answer would look like? It doesn’t seem all that different to me than the CS textbook examples of simple neural networks that recognize images with a very low resolution grid of black-or-white pixels. Is there even a “how it actually works” beyond how the basic physical mechanisms work? Perhaps, but it doesn’t seem like we even know what kind of “language of explanation” to look for beyond the normal reductive physical explanation.

xg15 4y ago
This feels like giving up and accepting that LLMs are just magic - or "emergence" to use a modern term which is practically used in the same sense. I think there are a few practical questions which we can use to gauge the level of understanding we have: Do we know which parts of the architecture and the training process are actually essential and which can be left away? Do we know which of the weights are essential? Do we know how the network arrives at a particular token probability which suggests some deep, abstract understanding of the prompt? Or likewise, if the network arrives at an incorrect answers, can we say which exact part of the calculation went wrong? Or for the current thread, can we explain how the network decides when to treat a text as an instruction and when as data? (Because it certainly does treat parts of the text as data: I can prompt it to translate a sentence into a different language and this will also often work with imperative sentences, but not always - if the imperative sentence is formulated in the right way, the network will treat it as an instruction.)
- tshaddox 4y ago
  > This feels like giving up and accepting that LLMs are just magic - or "emergence" to use a modern term which is practically used in the same sense. I don’t intend to persuade any interested person to give up on any pursuit of knowledge. It does seem like there’s a lot we don’t understand, but to me it feels like figuring out what kind of answer we’re looking for is a pretty important first step. And, while it might not be the case here, I think there are places in a chain of inquiry where simply asking “okay, but how is it really working” stops being meaningful. Like would you ask that once you’ve thoroughly studied an algorithm like insertion sort? “Oh I fully understand every line of code, how the compiler works, how the assembly code works, and even how the semiconductors work, but I still want to know how insertion sort actually works to sort an array of integers.”
  - xg15 4y ago
    I think you can illustrate the difference very well with basic algorithms: We absolutely do know how sorting algorithms "really work", there is basically a traditional first-year CS lecture that is all about that. With "really work", I mean here that we know which principles make sure that the algorithm always returns the result that we expect - a sorted array - even for for unknown inputs. You can mathematically prove that a sorting algorithm will return the correct result for every single input out of a literally infinite set of possible inputs. Part of the definition of the algorithm and of the proof also involves creating "intermediate concepts" that capture some sort of structure inside the algorithm: If you just measured all the electrical charges inside the CPU, you wouldn't see much: just a bunch of memory cells changing state in seemingly "unpredictable" patterns until at some point, "magically" the result appears. However, with sorting algorithms, we know which memory cells represent arrays, the instruction pointer, pivot elements, etc. We know there is a specific way those cells are supposed to interact and we know why those interactions will in the end lead to a fully sorted array. The CPU itself is a similar example: It's trillions of transistors, switching in nanoseconds - but we can still explain what each transistor "does", because we know the higher-level functional groups that they belong to - such as logic gates, then counters, then arithmetic units or memory, etc etc. Conversely, if there is an error, we know how to trace it back. I feel with LLMs, we're still very much at the "measure the elextrical charges" stage: We can pass words to the network that resemble an instruction. The words are converted to a vector, which is transformed through a number of very large weight matrices and in the end is turned into a probability distribution on words - and if we sample from that distribution, we get words back that very much look like the execution of the instruction. However, that doesn't itself explain what principles result in the network reliably mapping a human-readable instruction to its execution. It doesn't tell you about higher-level functional units within the weights. That's what I mean with "understanding".
    - tshaddox 4y ago
      > I feel with LLMs, we're still very much at the "measure the elextrical charges" stage I think that's a great example. Say you somehow had a skilled electrical engineer analyzing how a little IC manages to spit out a sorted array when given an unsorted array, but you're in a world without computer science or even information theory and this IC had just come through a portal. How could they figure out the explanations that we have about programming languages, compilers, assembly code, transistors, etc.? Well, they'd probably have to invent information theory and computer science before they'd even know what such an explanation could even feasibly look like.
- mcaledonensis 4y ago
  It's not magic. It's a world model. World simulator. World includes many objects, like a calculator for example. Or "The Hitchhiker's Guide to the Galaxy" by Infocom. So does the simulation.

xg15 4y ago

This feels like giving up and accepting that LLMs are just magic - or "emergence" to use a modern term which is practically used in the same sense. I think there are a few practical questions which we can use to gauge the level of understanding we have: Do we know which parts of the architecture and the training process are actually essential and which can be left away? Do we know which of the weights are essential? Do we know how the network arrives at a particular token probability which suggests some deep, abstract understanding of the prompt? Or likewise, if the network arrives at an incorrect answers, can we say which exact part of the calculation went wrong? Or for the current thread, can we explain how the network decides when to treat a text as an instruction and when as data? (Because it certainly does treat parts of the text as data: I can prompt it to translate a sentence into a different language and this will also often work with imperative sentences, but not always - if the imperative sentence is formulated in the right way, the network will treat it as an instruction.)

tshaddox 4y ago
> This feels like giving up and accepting that LLMs are just magic - or "emergence" to use a modern term which is practically used in the same sense. I don’t intend to persuade any interested person to give up on any pursuit of knowledge. It does seem like there’s a lot we don’t understand, but to me it feels like figuring out what kind of answer we’re looking for is a pretty important first step. And, while it might not be the case here, I think there are places in a chain of inquiry where simply asking “okay, but how is it really working” stops being meaningful. Like would you ask that once you’ve thoroughly studied an algorithm like insertion sort? “Oh I fully understand every line of code, how the compiler works, how the assembly code works, and even how the semiconductors work, but I still want to know how insertion sort actually works to sort an array of integers.”
- xg15 4y ago
  I think you can illustrate the difference very well with basic algorithms: We absolutely do know how sorting algorithms "really work", there is basically a traditional first-year CS lecture that is all about that. With "really work", I mean here that we know which principles make sure that the algorithm always returns the result that we expect - a sorted array - even for for unknown inputs. You can mathematically prove that a sorting algorithm will return the correct result for every single input out of a literally infinite set of possible inputs. Part of the definition of the algorithm and of the proof also involves creating "intermediate concepts" that capture some sort of structure inside the algorithm: If you just measured all the electrical charges inside the CPU, you wouldn't see much: just a bunch of memory cells changing state in seemingly "unpredictable" patterns until at some point, "magically" the result appears. However, with sorting algorithms, we know which memory cells represent arrays, the instruction pointer, pivot elements, etc. We know there is a specific way those cells are supposed to interact and we know why those interactions will in the end lead to a fully sorted array. The CPU itself is a similar example: It's trillions of transistors, switching in nanoseconds - but we can still explain what each transistor "does", because we know the higher-level functional groups that they belong to - such as logic gates, then counters, then arithmetic units or memory, etc etc. Conversely, if there is an error, we know how to trace it back. I feel with LLMs, we're still very much at the "measure the elextrical charges" stage: We can pass words to the network that resemble an instruction. The words are converted to a vector, which is transformed through a number of very large weight matrices and in the end is turned into a probability distribution on words - and if we sample from that distribution, we get words back that very much look like the execution of the instruction. However, that doesn't itself explain what principles result in the network reliably mapping a human-readable instruction to its execution. It doesn't tell you about higher-level functional units within the weights. That's what I mean with "understanding".
  - tshaddox 4y ago
    > I feel with LLMs, we're still very much at the "measure the elextrical charges" stage I think that's a great example. Say you somehow had a skilled electrical engineer analyzing how a little IC manages to spit out a sorted array when given an unsorted array, but you're in a world without computer science or even information theory and this IC had just come through a portal. How could they figure out the explanations that we have about programming languages, compilers, assembly code, transistors, etc.? Well, they'd probably have to invent information theory and computer science before they'd even know what such an explanation could even feasibly look like.
mcaledonensis 4y ago
It's not magic. It's a world model. World simulator. World includes many objects, like a calculator for example. Or "The Hitchhiker's Guide to the Galaxy" by Infocom. So does the simulation.