3 ms·

I'm still struggling to understand what counts as 'knowing how it works' for you. In my view, if you randomly generated a piece of code to do a task, you know

I'm still struggling to understand what counts as 'knowing how it works' for you. In my view, if you randomly generated a piece of code to do a task, you know how it works - you can see the algorithm right there in front of you. If you randomly generated it and checked each instance until you found a good one, you even know how you got it and why it's good at the task (because you checked it and threw away the ones that weren't). Obviously if you start not knowing what the code is, you have a simple, surface level lack of understanding which you can resolve by tracing it. Once you've done that, you understand everything there is to understand about it. The fact that it produces nice wavelet transforms is a simple product of how it was found. What more do you want to understand about it? What more is there to understand about it?

famouswaffles 3y ago
>you know how it works - you can see the algorithm right there in front of you. Seeing the algorithm in front of you doesn't mean you know how it works. It's gibberish code. If you'd never learnt C or any other programming language in your life, i could show you the C Code for a popular application. You could inspect it all you like. You will still understand nothing. The best you can do is, "this code is running this application". In the real world, you can just pick up a C book and start learning. In this instance, no one on earth has learnt C and there are no books on it. and neural network calculations are not just one unvarying "algorithm"
- kybernetikos 3y ago
  Knowing the algorithm literally means that you could theoretically reproduce it yourself, step by step (absent practical worries about longevity). Each of the steps is simple and well understood. What to do at each step is simple and well understood. What more is there to understand?
  - famouswaffles 3y ago
    By that definition, i know COBOL too since i can look up programs coded in it. I can reproduce the program too by hand-typing it and running it elsewhere. Banks should hire me asap /s. I genuinely don't get what is so hard to understand here. You don't know the algorithm. You can see it. That's all. You don't suddenly understand information just because you can see and copy it. Would certainly be nice though.
    - kybernetikos 3y ago
      What I'm trying to work out is what you mean by 'understand'. When it comes to an algorithm, what is it that you need to know beyond how to execute it in order to believe you understand it?
      - famouswaffles 3y ago
        Being able to make predictable changes directly to the algorithm would be a starter.
blagie 3y ago
I have plenty of pieces of code I've written, decades ago, where I know what they do, but have NO idea how they work. If I want to understand a piece of code, I need to read it and understand it. Modern models have billions of parameters and are trained over >10^20 computations. That's more than I can ever hope to read. > because you checked it and threw away the ones that weren't I know what it does under specific circumstances. I don't know what it does elsewhere. We have a pretty good understanding of how GPT-4 works on training data, but we have a very poor understanding of what it does for the countless other uses we see. Code I write, I analyze carefully for corner cases. If we develop an AI which has a corner case of "exterminate humanity" which wasn't in the training set, that's, well, very possible. I've trained plenty of neural networks (even once coding in machine code straight to custom hardware, back in the day). I can't say I understand how very many of them work, though. > you can resolve by tracing it You can't trace through 200B parameters or through 10^20 computations. That's beyond human capacity. We have no idea how it works, and we have a very poor understanding of emergent behaviors. Evolution "trains" biological organisms to survive to have as many babies as possible. Vengeance? Love? Loyalty? Pain? Hate? Emergent behaviours.
- kybernetikos 3y ago
  > I know what it does under specific circumstances. I don't know what it does elsewhere. No, but if you get given a circumstance, you know how to work out what it does in that circumstance. Are you saying that you need to keep every input -> output mapping in your head to feel you understand a piece of code? I feel like I understand multiplication pretty well, but there are many multiplication calculations you could give me where I wouldn't know the answer without a lot of thinking. There's some I couldn't work out by myself in my lifetime. That doesn't stop me feeling like I understand multiplication pretty well. > Vengeance? Love? Loyalty? Pain? Hate? Emergent behaviours. Sure, and arguably at the level we're talking about, those are descriptive rather than explanatory. 'Vengeance' isn't something a neuron knows about, nor is it a biological mechanism in our cells, it's how we describe high level behavior resulting from the interactions of the cells. It's an abstraction. If you had the accurate model you were talking about earlier, you'd be able to work out that given the right input, a particular behavior is output. That others might call that behavior 'vengeance', makes not a single iota of difference to your ability to predict the behavior of the system. Are you saying that you need to have developed high level descriptions of the behavior of a system in order to feel you understand it? What if there are no high level descriptions? In the hypothetical scenario where we hit on an algorithm randomly, there's no requirement that it translates to any specific high level concepts, there's just input, output and the algorithm, all of which we can understand. Or perhaps you mean that you already have a set of categories for output behavior and to truly understand something you need to be able to categorise the inputs and know which broad input categories result in which output categories? I could probably accept that as a broad definition of understanding, but there's a lot of flex there in terms of exactly what level of granularity you're requiring.

famouswaffles 3y ago

>you know how it works - you can see the algorithm right there in front of you. Seeing the algorithm in front of you doesn't mean you know how it works. It's gibberish code. If you'd never learnt C or any other programming language in your life, i could show you the C Code for a popular application. You could inspect it all you like. You will still understand nothing. The best you can do is, "this code is running this application". In the real world, you can just pick up a C book and start learning. In this instance, no one on earth has learnt C and there are no books on it. and neural network calculations are not just one unvarying "algorithm"

kybernetikos 3y ago
Knowing the algorithm literally means that you could theoretically reproduce it yourself, step by step (absent practical worries about longevity). Each of the steps is simple and well understood. What to do at each step is simple and well understood. What more is there to understand?
- famouswaffles 3y ago
  By that definition, i know COBOL too since i can look up programs coded in it. I can reproduce the program too by hand-typing it and running it elsewhere. Banks should hire me asap /s. I genuinely don't get what is so hard to understand here. You don't know the algorithm. You can see it. That's all. You don't suddenly understand information just because you can see and copy it. Would certainly be nice though.
  - kybernetikos 3y ago
    What I'm trying to work out is what you mean by 'understand'. When it comes to an algorithm, what is it that you need to know beyond how to execute it in order to believe you understand it?
    - famouswaffles 3y ago
      Being able to make predictable changes directly to the algorithm would be a starter.

blagie 3y ago

I have plenty of pieces of code I've written, decades ago, where I know what they do, but have NO idea how they work. If I want to understand a piece of code, I need to read it and understand it. Modern models have billions of parameters and are trained over >10^20 computations. That's more than I can ever hope to read. > because you checked it and threw away the ones that weren't I know what it does under specific circumstances. I don't know what it does elsewhere. We have a pretty good understanding of how GPT-4 works on training data, but we have a very poor understanding of what it does for the countless other uses we see. Code I write, I analyze carefully for corner cases. If we develop an AI which has a corner case of "exterminate humanity" which wasn't in the training set, that's, well, very possible. I've trained plenty of neural networks (even once coding in machine code straight to custom hardware, back in the day). I can't say I understand how very many of them work, though. > you can resolve by tracing it You can't trace through 200B parameters or through 10^20 computations. That's beyond human capacity. We have no idea how it works, and we have a very poor understanding of emergent behaviors. Evolution "trains" biological organisms to survive to have as many babies as possible. Vengeance? Love? Loyalty? Pain? Hate? Emergent behaviours.

kybernetikos 3y ago
> I know what it does under specific circumstances. I don't know what it does elsewhere. No, but if you get given a circumstance, you know how to work out what it does in that circumstance. Are you saying that you need to keep every input -> output mapping in your head to feel you understand a piece of code? I feel like I understand multiplication pretty well, but there are many multiplication calculations you could give me where I wouldn't know the answer without a lot of thinking. There's some I couldn't work out by myself in my lifetime. That doesn't stop me feeling like I understand multiplication pretty well. > Vengeance? Love? Loyalty? Pain? Hate? Emergent behaviours. Sure, and arguably at the level we're talking about, those are descriptive rather than explanatory. 'Vengeance' isn't something a neuron knows about, nor is it a biological mechanism in our cells, it's how we describe high level behavior resulting from the interactions of the cells. It's an abstraction. If you had the accurate model you were talking about earlier, you'd be able to work out that given the right input, a particular behavior is output. That others might call that behavior 'vengeance', makes not a single iota of difference to your ability to predict the behavior of the system. Are you saying that you need to have developed high level descriptions of the behavior of a system in order to feel you understand it? What if there are no high level descriptions? In the hypothetical scenario where we hit on an algorithm randomly, there's no requirement that it translates to any specific high level concepts, there's just input, output and the algorithm, all of which we can understand. Or perhaps you mean that you already have a set of categories for output behavior and to truly understand something you need to be able to categorise the inputs and know which broad input categories result in which output categories? I could probably accept that as a broad definition of understanding, but there's a lot of flex there in terms of exactly what level of granularity you're requiring.