3 ms·

Even if we can explain the function of a single neuron what do we gain? If the goal is to reason about safety of computer vision in automated driving as an exam

Even if we can explain the function of a single neuron what do we gain? If the goal is to reason about safety of computer vision in automated driving as an example, we would need to understand the system as a whole. The whole point of neural networks is to solve nuanced problems we can't clearly define. The fuzziness of the problems those systems solve is fundamentally at odds with the intent to reason about them.

ChatGTP 3y ago
I have to agree. I often think, “maybe I should use ChatGPT for this” then I realise I have very little way to verify what it tells me and as someone working in engineering, If I don’t understand the black box, I just can’t do it. I’m attracted to open source, because I can look at the code understand it.
- famouswaffles 3y ago
  Open source doesn't mean you can explain the black box any better. and humans are black boxes that don't understand their mental processes either. We're currently better than LLMs at it i suppose but we're still very poor at it. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522 We can't recreate previous mental states, we just do a pretty good job (usually) of rationalizing decisions after the fact.
  - ChatGTP 3y ago
    Humans can be held accountable so it’s not the same. Even if we’re a black box, we share common traits with other humans. We’re trained in similar ways. So we mostly understand what we will and won’t do. I think this constant degradation of humans is really foolish and harmful personally. “We’re just black boxes etc”, we might not know how brains work but we do and can understand each other. On the other hand I’m starting to feel like “AI researchers” are the greatest black box I’ve ever seen, the logic of what they’re trying to create and their hopes for it really baffle me. By the way, I have infinitely more hope of understanding an open source black box compared to a closed source one?
    - famouswaffles 3y ago
      >Humans can be held accountable so it’s not the same. 1. Don't worry, LLMs will be held accountable eventually. There's only so much embodiment and unsupervised tool control we can grant machines before personhood is in the best interests of everybody. May be forced like all the times in the past but it'll happen. 2. not every use case cares about accountability 3. accountability can be shifted. we have experience. >I think this constant degradation of humans is really foolish and harmful personally. Maybe you think so but there's nothing degrading about it. We are black boxes that poorly understand how said box actually works even if we like to believe otherwise. Don't know what's degrading about stating truth that's been backed by multiple studies. Degrading is calling an achievement we hold people in high regard who accomplish stupid because a machine can do it. >By the way, I have infinitely more hope of understanding an open source black box compared to a closed source one? Sure i guess so.
      - ChatGTP 3y ago
        Degrading is calling an achievement we hold people in high regard who accomplish stupid because a machine can do it. Not sure you worded this as intended ? Anyway if I read you correctly, this assumes you believe the idea of self and ego have anything to do with it. Humans should treat ants, lab rats and each other with equal respect. I don’t believe we should avoid self-degradation because we think we’re smart or special, but for completely opposite reasons. We are mostly lucky we have what we have because something bigger than us, call it God, nature whatever, has provided that existence. When we degrade one another, we degrade that magic. This is where we fuck up time and time again. I’m talking about the water you drink, the food you eat and the air your breathe, the inspiration for neural networks etc. We take that for granted. I liken it to the idea that humans killed God, the idea of God and morals etc, so we could do terrible things to the world and living things. We just got rid of the idea someone is looking over our shoulder because it made wars and genocides easier to do. Killing God got rid of a whole lot of moral baggage.
wilg 3y ago
> reason about safety of computer vision in automated driving An interesting analogue. I think we simply aren't going to reason about the internals of neural networks to analyze safety for driving, we're just going to measure safety empirically. This will make many people very upset but it's the best we can do, and probably good enough.
FeepingCreature 3y ago
If there is a limited number of deception neurons, we can automatically look for coactivation of the "my future behavior" neuron and the deception neuron.

ChatGTP 3y ago

I have to agree. I often think, “maybe I should use ChatGPT for this” then I realise I have very little way to verify what it tells me and as someone working in engineering, If I don’t understand the black box, I just can’t do it. I’m attracted to open source, because I can look at the code understand it.

famouswaffles 3y ago
Open source doesn't mean you can explain the black box any better. and humans are black boxes that don't understand their mental processes either. We're currently better than LLMs at it i suppose but we're still very poor at it. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522 We can't recreate previous mental states, we just do a pretty good job (usually) of rationalizing decisions after the fact.
- ChatGTP 3y ago
  Humans can be held accountable so it’s not the same. Even if we’re a black box, we share common traits with other humans. We’re trained in similar ways. So we mostly understand what we will and won’t do. I think this constant degradation of humans is really foolish and harmful personally. “We’re just black boxes etc”, we might not know how brains work but we do and can understand each other. On the other hand I’m starting to feel like “AI researchers” are the greatest black box I’ve ever seen, the logic of what they’re trying to create and their hopes for it really baffle me. By the way, I have infinitely more hope of understanding an open source black box compared to a closed source one?
  - famouswaffles 3y ago
    >Humans can be held accountable so it’s not the same. 1. Don't worry, LLMs will be held accountable eventually. There's only so much embodiment and unsupervised tool control we can grant machines before personhood is in the best interests of everybody. May be forced like all the times in the past but it'll happen. 2. not every use case cares about accountability 3. accountability can be shifted. we have experience. >I think this constant degradation of humans is really foolish and harmful personally. Maybe you think so but there's nothing degrading about it. We are black boxes that poorly understand how said box actually works even if we like to believe otherwise. Don't know what's degrading about stating truth that's been backed by multiple studies. Degrading is calling an achievement we hold people in high regard who accomplish stupid because a machine can do it. >By the way, I have infinitely more hope of understanding an open source black box compared to a closed source one? Sure i guess so.
    - ChatGTP 3y ago
      Degrading is calling an achievement we hold people in high regard who accomplish stupid because a machine can do it. Not sure you worded this as intended ? Anyway if I read you correctly, this assumes you believe the idea of self and ego have anything to do with it. Humans should treat ants, lab rats and each other with equal respect. I don’t believe we should avoid self-degradation because we think we’re smart or special, but for completely opposite reasons. We are mostly lucky we have what we have because something bigger than us, call it God, nature whatever, has provided that existence. When we degrade one another, we degrade that magic. This is where we fuck up time and time again. I’m talking about the water you drink, the food you eat and the air your breathe, the inspiration for neural networks etc. We take that for granted. I liken it to the idea that humans killed God, the idea of God and morals etc, so we could do terrible things to the world and living things. We just got rid of the idea someone is looking over our shoulder because it made wars and genocides easier to do. Killing God got rid of a whole lot of moral baggage.

wilg 3y ago

> reason about safety of computer vision in automated driving An interesting analogue. I think we simply aren't going to reason about the internals of neural networks to analyze safety for driving, we're just going to measure safety empirically. This will make many people very upset but it's the best we can do, and probably good enough.

FeepingCreature 3y ago

If there is a limited number of deception neurons, we can automatically look for coactivation of the "my future behavior" neuron and the deception neuron.