3 ms·

To start with, most computers are running about 20-30 microcontrollers. No source code, most of the time even no access to the firmware. Phones have a bit fewe

by vardump 10y ago

To start with, most computers are running about 20-30 microcontrollers. No source code, most of the time even no access to the firmware. Phones have a bit fewer, but I think you'll find at least about 10. A simple cheap SD memory card has an embedded core, usually ARM7TDMI. Even most lithium batteries have an integrated microcontroller. I'm also a programmer, I do bare metal embedded, kernel drivers and such. I often don't have anyone else to ask from, except occasionally hardware guys. They're also often stumped.

jimbokun 10y ago
Interesting. So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network? Even with a company's reputation at stake, it's not possible to hire a consultant with a deep understanding of obscure micro controllers? I only deal with the source code level, don't really have experience digging into the hardware level.
- vardump 10y ago
  We rely on many layers of more or less leaky abstractions. Components have always a set of intended behavior and some, often unknown or poorly understood, set of unintended behavior. It's very important to understand that abstractions mask both intended and unintended behavior. You deal with unintended side the best you can, with sanity checks and so on. But some of it inevitably leaks through. Those outcomes that result from multiple systems being individually in good state but collectively causing undesired behavior are often very tricky to understand. On the flip side, when you do understand them, you tend to gain new surprising insights into the whole system. Also timing makes understanding hard. When it comes to a complex network of systems, it's hard to concurrently reproduce the state where the effect occurs. > So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network? If Tesla has good, exhaustive, event logging facilities, a "black box", I guess they'll be able to figure out what happened by analyzing all of the available data with the systems knowledge they have. Anyways, if it's not systematic, why should Tesla autopilot mistake be handled so differently from a human being making a mistake? You can't analyze why a human being caused an accident either. The best you can usually do is to develop scenarios "theories" based on the available evidence, but that's it.
  - jamescostian 10y ago
    I don't think analyzing why a human caused an accident is important either. However, being able to fix the bug is very important. If a programmer can identify the problem in their code, they can fix it. But how can one identify the problem in a neural network and fix it? At least right now, understanding an (actually useful) neural network might as well be impossible
- wrs 10y ago
  Just look at the Toyota acceleration lawsuit, where a consultant spent months trying to figure the system out and still ended up with a probabilistic explanation (basically, the software uses a bunch of bad practices that could result in bad effects that could include the alleged accelerator problem). http://www.edn.com/design/automotive/4423428/Toyota-s-killer-firmware--Bad-design-and-its-consequences http://www.edn.com/design/automotive/4423428/Toyota-s-killer...
Florin_Andrei 10y ago
I think it's fair to say that we don't have systems anymore that could be 100% understood by a single person. Your smartphone for example, I think it could be 100% understood by a crowd of engineers just big enough to fill a modest concert hall. Way, way back, I had an 8 bit computer that I built myself from chips and solder. The "permanent" memory was 16kB; I knew most of that code pretty well. Even then the CPU itself required quite a bit of study to comprehend.
Florin_Andrei 10y ago
On the flip side, it may well turn out that systems complex enough to be 100% understood by a single person would be dumb and limited. There are pretty significant limits to our conscious, logical understanding of the world.

by vardump 10y ago

jimbokun 10y ago

Interesting. So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network? Even with a company's reputation at stake, it's not possible to hire a consultant with a deep understanding of obscure micro controllers? I only deal with the source code level, don't really have experience digging into the hardware level.

vardump 10y ago
We rely on many layers of more or less leaky abstractions. Components have always a set of intended behavior and some, often unknown or poorly understood, set of unintended behavior. It's very important to understand that abstractions mask both intended and unintended behavior. You deal with unintended side the best you can, with sanity checks and so on. But some of it inevitably leaks through. Those outcomes that result from multiple systems being individually in good state but collectively causing undesired behavior are often very tricky to understand. On the flip side, when you do understand them, you tend to gain new surprising insights into the whole system. Also timing makes understanding hard. When it comes to a complex network of systems, it's hard to concurrently reproduce the state where the effect occurs. > So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network? If Tesla has good, exhaustive, event logging facilities, a "black box", I guess they'll be able to figure out what happened by analyzing all of the available data with the systems knowledge they have. Anyways, if it's not systematic, why should Tesla autopilot mistake be handled so differently from a human being making a mistake? You can't analyze why a human being caused an accident either. The best you can usually do is to develop scenarios "theories" based on the available evidence, but that's it.
- jamescostian 10y ago
  I don't think analyzing why a human caused an accident is important either. However, being able to fix the bug is very important. If a programmer can identify the problem in their code, they can fix it. But how can one identify the problem in a neural network and fix it? At least right now, understanding an (actually useful) neural network might as well be impossible
wrs 10y ago
Just look at the Toyota acceleration lawsuit, where a consultant spent months trying to figure the system out and still ended up with a probabilistic explanation (basically, the software uses a bunch of bad practices that could result in bad effects that could include the alleged accelerator problem). http://www.edn.com/design/automotive/4423428/Toyota-s-killer-firmware--Bad-design-and-its-consequences http://www.edn.com/design/automotive/4423428/Toyota-s-killer...

Florin_Andrei 10y ago

I think it's fair to say that we don't have systems anymore that could be 100% understood by a single person. Your smartphone for example, I think it could be 100% understood by a crowd of engineers just big enough to fill a modest concert hall. Way, way back, I had an 8 bit computer that I built myself from chips and solder. The "permanent" memory was 16kB; I knew most of that code pretty well. Even then the CPU itself required quite a bit of study to comprehend.

Florin_Andrei 10y ago

On the flip side, it may well turn out that systems complex enough to be 100% understood by a single person would be dumb and limited. There are pretty significant limits to our conscious, logical understanding of the world.