6 ms·

Wait. Are there still computer systems in 2016 we can fully understand? Except maybe some simpler embedded systems. Last computer system I can somewhat fully u

• gambler 10y ago
  Normal software is split into logical modules and layers, which you can understand on their own and then use that to reason about code at higher and higher levels. Many ML models do not offer anything comparable.
• deleted 10y ago
  [deleted]
• jimbokun 10y ago
  Unequivocally yes. Obviously no one has a full mental model in their head of everything going on in their phone (for example) at a given moment in time. But when it really matters, and there is a specific question to answer, and a programmer with experience writing the kind of software in question has full access to all of the source code, its possible to get an answer to why a program behaved in a particular way. I'm a programmer, and this is what I do every day. And in those rare cases where knowledge of the specific hardware is required, there are experts who understand the hardware and can answer the relevant questions. So for machine learning approaches, what is the equivalent of the "source code"? What resource is there to examine to determine how the system will respond to a given set of inputs, and why? Evidently, there are not clear answers, at least for some machine learning approaches.
  • vardump 10y ago
    To start with, most computers are running about 20-30 microcontrollers. No source code, most of the time even no access to the firmware. Phones have a bit fewer, but I think you'll find at least about 10. A simple cheap SD memory card has an embedded core, usually ARM7TDMI. Even most lithium batteries have an integrated microcontroller. I'm also a programmer, I do bare metal embedded, kernel drivers and such. I often don't have anyone else to ask from, except occasionally hardware guys. They're also often stumped.
    • jimbokun 10y ago
      Interesting. So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network? Even with a company's reputation at stake, it's not possible to hire a consultant with a deep understanding of obscure micro controllers? I only deal with the source code level, don't really have experience digging into the hardware level.
      • vardump 10y ago
        We rely on many layers of more or less leaky abstractions. Components have always a set of intended behavior and some, often unknown or poorly understood, set of unintended behavior. It's very important to understand that abstractions mask both intended and unintended behavior. You deal with unintended side the best you can, with sanity checks and so on. But some of it inevitably leaks through. Those outcomes that result from multiple systems being individually in good state but collectively causing undesired behavior are often very tricky to understand. On the flip side, when you do understand them, you tend to gain new surprising insights into the whole system. Also timing makes understanding hard. When it comes to a complex network of systems, it's hard to concurrently reproduce the state where the effect occurs. > So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network? If Tesla has good, exhaustive, event logging facilities, a "black box", I guess they'll be able to figure out what happened by analyzing all of the available data with the systems knowledge they have. Anyways, if it's not systematic, why should Tesla autopilot mistake be handled so differently from a human being making a mistake? You can't analyze why a human being caused an accident either. The best you can usually do is to develop scenarios "theories" based on the available evidence, but that's it.
        • jamescostian 10y ago
          I don't think analyzing why a human caused an accident is important either. However, being able to fix the bug is very important. If a programmer can identify the problem in their code, they can fix it. But how can one identify the problem in a neural network and fix it? At least right now, understanding an (actually useful) neural network might as well be impossible
  • l_dopa 10y ago
    Is knowing, for example, that there's a race triggered by a new optimization in compiler version X that happens under conditions Y and Z really a qualitatively different type of understanding than "the weights are such that the ANN computed this incorrect output for these inputs?" In both cases we can only fix a particular instance of an error and effectively have no way to show that similar errors don't exist, or even that we didn't introduce new bugs with the fix.
    • jimbokun 10y ago
      "the weights are such that the ANN computed this incorrect output for these inputs?" Can we actually do this, for complex ANNs?
      • espadrine 10y ago
        Yes. The matrix is stored. In fact, there has been a number of visualization tools for neural nets to help track down which weights contribute to which results. Additionally, the weights are constant in production, which makes analyzing production issues just a matter of logging the inputs and the outputs. Of course, for most convolutional nets, the inputs can get quite large (for example, a camera feed).
    • Klockan 10y ago
      The difference is that all human written systems was once understood by a human. ANN on the other hand creates unlabeled spaghetti which gets impossible to reason about if you scale that up to megabytes. You could try to fix a bug by changing one of the values, but that would ruin the ANN since all the values were chosen to be optimal. Thus the only way to make it better would be to change lots of unlabeled values at the same time and that is something no human can do. The only plausible way to fix a bug in an ANN is to incorporate said bug in the training set.
      • l_dopa 10y ago
        > The only plausible way to fix a bug in an ANN is to incorporate said bug in the training set My point was not something about manipulating weights directly vs. augmenting the training set. The analogy is roughly, find a misclassification : add particular input to training set :: find a bug : add a patch/test case. I'm playing devil's advocate, of course, and there are obvious, important differences. > The difference is that all human written systems was once understood by a human But that's not one of them! In practice, we create software systems by getting some rough intuition about an interface, wiring together some components, then checking the results empirically. Basic components that everything else is built on, C compilers for instance, are based on ambiguous, even contradictory specifications.
  • borplefark 10y ago
    the only difference is you need code + data. Just because you don't understand it doesn't mean it's impossible to understand deep learning
  • bhntr3 10y ago
    Right. What we're talking about here are tools. We can't understand or debug the electrons moving through the chip. But we build systems using languages and we build tools to translate execution back into those languages (Debuggers) Or tools in the execution to allow us to understand what happened later (Logging, metrics.) I think the key problem right now is that there are not too many models for which tracing the execution lets us know much about what's happening. A single decision tree. A linear model. But add ensembles to trees or layers to linear models and the ability to understand what's happening through logging disappears. In this sense I think we're essentially in the pre-assembly language state of machine learning. For most programming languages, we assume correctness as a given. If it produces the wrong result on correct inputs, that's really bad. With machine learning, we don't have perfect correctness. Aka accuracy. We focus so much on improving accuracy that we rarely consider the trade-off in debuggability or interpretability. I think this will have to change. Advances in NLP and image stuff from deep learning are very exciting. Ignoring the evil AI concerns, I think the best step will be trying to understand better why the models did what they did. I think it will be a while before we can just ask the model and get a reliable answer. In the meantime, I think we'll work on a kind of meta programming where we integrate models better with their broader context. An example might be sequence to sequence models that have a way to expose sentence fragments and documents in the training set that influenced a word choice. Or automatic labeling of clusters in image representation layers that have semantic meaning to a human. There's some interesting work already on debugging tools for image classification that I really liked. On the phone so I'll try to provide the link later. The trouble is it all seems very domain specific. Figuring out how do it in a way that's not so domain specific is i think a very interesting thing to work on.
• marcosdumay 10y ago
  You are being a bit too pedantic here. Take this python program: print "Hello, world!" Do you know what it will do without running it? You have no chance of knowing that about a program created by machine learning.
  • rwallace 10y ago
    Of course you do. You've presented a trivial Python program; an equivalent in neural networks would be a single scalar input multiplied by a single weight to produce a single output. That's just about as easy to understand without having to run it.
    • Klockan 10y ago
      No, the equivalent in neural networks is a network which is trained to always output "Hello, world!", and I dare say that such a network would be pretty hard to reason about compared to the python equivalent.
      • rwallace 10y ago
        Actually, the minimal neural network that always outputs that string would basically have a weight vector equivalent to the ASCII coding of the string, so it would be equally easy to reason about compared to the Python equivalent.
        • Klockan 10y ago
          You are comparing apples to oranges. The python line is a good representation of a part of a larger program, while the neural network you are talking about is not a good representation of any part of a larger neural network. You wont get that solution if you train it normally without knowing how to solve the problem yourself. Try setting up a network with 3 layers of nodes and 120 nodes in each layer and train it to output "Hello, world!" using a standard training method, you wont get something easily comprehensible from that.
    • jrimclean 10y ago
• ninjakeyboard 10y ago
  You don't understand software. You have the source, and then you go read it to understand the little slice you want to understand. Then you forget it and move on unless you have to understand that part again and again. Then you might remember that little piece :) that's it.
  • vardump 10y ago
    Well put. You state the problem very concisely! We don't understand computer systems. Like you said, We won't even understand software. What we understand is a small slice of it and we make mistakes even with that. Or fail to consider some cases or scenarios. Then we put those small slices together to form a module and make the same mistakes again.