10 ms·

Computers Can Be Understood

170 points by miked85 7y ago

jeffrallen 7y ago
The author has clearly never chased a heisenbug in a 25 year old Makefile for embedded systems. The kind where a timing bug, plus NFS automounter, plus WAN weather, equals an unexpected PATH, resulting in a different minor version of a linker than expected, thereby changing the order of static initializers on some targets, thereby resulting in sshd not checking passwords, but only on MIPS and only with the 16 mbps flash config.
- fit2rule 7y ago
  .. you thus provide a perfect example of the claim made.
- evmar 7y ago
  The author had a followup post about this that I found more interesting: https://blog.nelhage.com/post/systems-that-defy-understanding/ https://blog.nelhage.com/post/systems-that-defy-understandin...
  - munchbunny 7y ago
    As someone who works on a system that spans 10^big nodes, I really like the point in that post. At a certain level of complexity even your most baseline assumptions will have counterexamples, and 99.9% uptime means 1 in 1000 nodes are in various states of failure at any time. You have to build fault tolerance into the design at each layer to keep errors at base layers from snowballing into real outages. And that’s inherently at odds with the “fail fast” principle, so you have to spend a lot of energy on how to fail fast inwardly but stay operational outwardly.
- geofft 7y ago
  All of those can be understood, no? (I'd expect to go at least that deep in a postmortem of that problem, and I regularly do go about that deep in similar postmortems.)
  - ken 7y ago
    That might be stretching the definition of "deterministic". If it depends on implicit state, precise timing, external conditions, etc., I would say it's closer to fitting the definition of "chaotic" than "deterministic". With perfect observation and recording, it may be possible to see a tornado and work backwards to figure out which butterflies flapped their wings to cause it, but that would still be a long ways off from being able to train all butterflies to move only in ways which won't cause tornados.
    - geofft 7y ago
      Sure, but you don't need to know which butterflies flapped their wings - you just need to know "if there's a tornado, the house falls down, and we're in tornado country." In the example above, it seems sufficient and entirely testable to conclude, if for whatever reason NFS is flaky, $PATH resolution will skip the directory with the right linker and fall back to the old one, and if for whatever reason you call the old linker, it has different behavior about static initializer order. That's enough to both describe the problem, reproduce it, and fix it. No, you're not going to be able to identify which particular packet was lost, but (and I say this as someone who's maintained production systems that rely on cross-continent NFS) you expect that packets can be lost and you figure out how to make the system robust against it. Don't put multiple files with the same name on $PATH, or call the linker by its full path, or something. And that leaves you a very far distance from "something spooky is happening with the computer and so passwords weren't being checked today, who knows."
mjw1007 7y ago
I think this is a good attitude. I wish that people who publish advice on writing documentation would put more emphasis on writing for people who have this attitude. I see too much emphasis on providing usage examples, and on "write for a reader who is trying to get something specific done", rather than explaining the system for someone who wishes to understand it.
- api 7y ago
  The emphasis on getting things done vs. understanding is another manifestation of our age's extreme time poverty. People don't have time to understand anything. Add to that the fact that computing is so trendy. Take the time to learn something -- web development, mobile development, the latest trendy language -- and it won't be hip anymore in 5-10 years. As an experienced programmer I only take the time to deeply understand things that my intuition tells me will be around for a while and for which understanding is a major benefit to me. My intuition is guided by a mix of market share, speed of change in the target market, and robustness of the thing in question.
  - EvanAnderson 7y ago
    It's another manifestation of the classic "it's expensive to be poor" situation. My personal struggle is with making the call re: understanding vs. getting things done compared with the return on investment. Understanding things is fun, though, and our systems are built upon enough analogy that understanding usually transfers (if only in part) to something else down the road.
    - thisisnico 7y ago
      Realistically it depends on how invested I am in the system. If it's something that I'm using all-the-time, it absolutely makes sense for me to have a deeper understanding because that understanding could save me time and give me a better perspective on why certain aspects of the system are the way they are. If it's something I use only as an auxiliary system, I don't invest as much time in understanding the system, only what the system can do for me. A prime example of understanding how to use the system, but not the system itself is e-commerce. There are 17+ year old kids making 100K+ a month with e-commerce, having no programming skills or a deep understanding of the web or computers. They are experts at the tools they use, but honestly are not "computer" people. We've abstracted e-commerce to the point that it's not even necessary to deeply understand the web, with tools that exist now (Shopify etc.)
Koshkin 7y ago
But sometimes trying to understand something just isn't worth it.
- nemetroid 7y ago
  Indeed, as the article expands on in the "Pitfalls of this approach" section.
- umvi 7y ago
  This happened to me. We were seeing a USB issue between a Marvell SoC we purchased and a Qualcomm SoC. The more I tried to understand this issue, the deeper the rabbit hole went until we were in the weeds of the XHCI controller looking at scatter/gather operations, etc. Eventually we bought a USB protocol analyzer that seemed to indicate that the actual host controller had a bug. We contacted the Marvell, who in turn contacted the vendor of the USB core (Synopsys). They bisected USB cores with an FPGA until they found a day which had a breaking change. Unfortunately, it appears Synopsys didn't use version control with their source verilog a few years ago (they just had archived daily builds), so while they could say "yep, looks like something that changed on 13 Jan 2015 broke it" they didn't know what change that was. However, before it could be resolved the company I was working for went bankrupt and so I never did get closure on the issue. I sunk many, many months into understanding that issue over the course of several years only to give up because it was near impossible to understand enough to come up with a workaround (and because the company went bankrupt).
empath75 7y ago
What shifted my career was actually an almost opposite revelation. I'm a self-taught programmer who never went to college, and bounced around in entry level sort of "IT jobs" like support for close to a decade -- and I assumed that "real" software engineers and sysadmins were geniuses who did understand everything their systems were doing on a deep level and just typed out perfect code off the top of their head. Then I went to a large tech company, and got some attention for hacking away at a few bash scripts to automate things, got promoted to a sys admin job, and realized that hardly anyone understood systems, or anything at all on a deep level, and that everyone was just hacking away like I was. What I've eventually realized is that while, yes, everything in a complex system in principle can be understood, that human beings have limited bandwidth and memory and that it will absolutely paralyze you if you wait until you understand something thoroughly before diving into do work on it. I jumped into AWS without understanding it, Chef without understanding it, Jenkins without understanding it, and Kubernetes without understanding of it, and every time I did that, my salary went up by 20-50%, to the point where i'm making something like 3-4 times now than what I was making 5 years ago. There are two reasons to spend the time to learn something on the job: 1) You need to learn it to do the task that you're working on right now. Whether it's troubleshooting a problem or trying to install a new system. If you're trying to get a kubernetes cluster running, you absolutely do not need to know all the details of how docker containers work. While it's interesting, and it will probably be necessarily to learn it eventually, it would be a waste of time to do that before just getting started deploying kubernetes. 2) Satisfying your own personal curiosity -- sometimes things are just inherently interesting to you, and it's usually worth chasing those things down, I've found. It may not be immediately applicable, but usually I'll figure out how to apply it somewhere.
- jcranmer 7y ago
  > 2) Satisfying your own personal curiosity -- sometimes things are just inherently interesting to you, and it's usually worth chasing those things down, I've found. It may not be immediately applicable, but usually I'll figure out how to apply it somewhere. A few months ago, while searching for a personal project, I stumbled across a mention that the performance events on Linux can be used to set hardware watchpoints. Literally the very next day at work, we had a problem where we needed a watchpoint but gdb was too intrusive to timing to let the bug happen, so I was able to fall back on perf to confirm the stack trace of the site corrupting memory. Debugging is ultimately about hypothesizing which part of the system is failing and why, so understanding more of how your system works is likely to be beneficial towards future debugging sessions.
troughway 7y ago
From the post: "My advice for a practical upshot from this post would be: cultivate a deep sense of curiosity about the systems you work with. Ask questions about how they work, why they work that way, and how they were built." This is the poignant thing about it all. The individual for who this post is written does not need to be told to cultivate this. The individual who does need to be told, on the other hand, is unlikely to. In some ways the post reads as a "keep doing what you are doing" reminder.
- hinkley 7y ago
  Sometimes, it's good for people who engage in persuasive speaking to talk to each other and compare notes. "Preaching to the choir" is known in some circles as rehearsing. Similarly, I have owned a couple of books I got almost nothing out of, other than something I can hand to people who ask a lot of questions. They're good books to own. One is quite beat up at this point, but worth it.
bitwize 7y ago
Again -- "Walk into the fire, Morty. Embrace the suck." https://news.ycombinator.com/item?id=22601623 https://news.ycombinator.com/item?id=22601623
umvi 7y ago
Counterexample: Convolutional Neural Networks (CNNs) (or any ML model, really) We still don't fully understand how or why they work so well. If you have software that queries a CNN and the CNN returns some prediction, there's absolutely no way of knowing or understanding the reasoning behind the decision. It's a black box of magical layers and weights that have been "trained" to make good predictions. If you have fraud detection software using ML and it flags something as fraud, that's the extent of the information you get. In this case the computer cannot be understood - it cannot explain to you how it arrived at the conclusion that something is fraud, nor can you explain how the computer arrived at the decision. It just has huge tables of weights and biases that magically "know" when something is fraud.
- thisisbrians 7y ago
  True, but you also aren't tasked with going in and manually adjusting weights to improve the predictions. And if you are curious enough, you can certainly look at the theory and mathematics that underpin how these systems do work, even if you don't understand exactly how the training data is making its way into your model: someone clearly figured this out and that's why these systems work. Put another way: you can treat an algorithm or routine as a black box if you know it works and understand its context in your system. You can even learn the right questions to ask if you do need to dig into that particular box at some point. Without some abstractions, we'd all be constantly paralyzed. (edit: formatting)
  - yuliyp 7y ago
    The problem comes when you need to peer inside the black box because it's working counter to your expectations of it: With most systems you can do so by reading code / documentation etc. With ML you're kind of stuck. You can conduct experiments to see what changes about the behavior of the model, but even that proves tricky when the model has thousands of inputs.
- JoshTriplett 7y ago
  There are numerous research results about building understandable, analyzable neural network models. Most models don't bother, because that isn't a necessary step to get reasonable answers. But it's possible.
mwcampbell 7y ago
I’d like to add another pitfall: obsessing over optimizing or eliminating intermediate layers when one should be focusing on meeting business requirements in higher layers. I’m prone to this one myself. This one is tough in the current move-fast-and-disrupt-things environment of startups, or even teams within larger companies that are trying to be nimble like a startup.
- daze42 7y ago
  This is the hardest for me. It's almost like those two forces are in opposition to each other. I'd love to find an industry where they could align.
  - chubot 7y ago
    Yeah I think there is a tension there. If you care about systems and internal quality and want to get paid for it, I'd suggest software performance or security as promsiing fields. Performance and security are both cross-cutting concerns that span layers, i.e. you're not just thinking at the application level. It really is a separate kind of thinking and a separate kind of work. Though it seems that it's mostly large companies (e.g. "big tech" and a few other places) that care enough about performance and security to have dedicate staff for them. Most other shops are too busy with "business stuff", and they may reasonably want to outsource that work to experienced consultants.
- marcosdumay 7y ago
  I guess the keyword here is "obsessing". Software performance and maleability have serious impacts on business and can not be overlooked. But there is an optimum, and it is usually much lower than what most developers want to create. Honestly, I've learned to take fulfillment from the value my software adds to people, instead of it's abstract quality. But well, I still overengineer things all the time.
gpanders 7y ago
> A belief in the understandability of software systems can very easily become a need to understand the systems you work with. I relate to this point especially, as this is how I often am as well. I do a lot of work with FPGAs and embedded systems, and the tools offered by the vendors in this space are often huge and complex in an attempt to create an easy "happy path" for hardware engineers who don't want to/can't learn software engineering and just want to get something working. I, however, stubbornly chose to eschew these tools in favor of "doing it from scratch". In the end I think it was 100% worth it as this process deepened my understanding of the field and has left me with some valuable skills, but the end result was that it took me weeks to do something that another colleague could have done in a day or two, maybe less. I understand the underlying system and implementation better, but that may or may not be considered valuable depending on the project and your company's culture.
m0ther 7y ago
I am a senior software architect. My job is to balance performance against complexity. If my system is slow, it's my fault. If debugging or expanding the system is too difficult, it's my fault. If someone wants to know how the system or business works in depth, I am the one that they should come to. I spend the majority of my time chasing down, enforcing, and simplifying the universal theory of our business (the core of our software solution). The universal theory of our business is a living collection of concepts, designed to accurately model non-virtual concerns in virtual space. If there are too many edge cases, it is a sign that the universal theory is inaccurate, or not robust enough in some areas. If there are too many bugs, it is a sign that the universal theory has not been communicated or enforced well enough, is inaccurate, or is too complex in some areas. As our business grows, or our understanding of the business expands, the universal theory will change; at times dramatically. Malleability (the ability to adapt our software to these changes effectively and efficiently) is one of my top two concerns; the other is latency (how long it takes for any one request to get a complete response). The theory shrinks and becomes better documented as it evolves; the goal is to move from describing behaviors as correlation to describing them as accurate causation. To fill in the blanks. I should mention here that this does not mean every line of code in my projects is easy to understand. Writing a fast system of high complexity requires at least some components that are written exclusively for the computer's benefit (that is, highly optimized and inherently difficult to read). These components should be written with clear documentation, clearly defined public members, written discussions of why it works the way it does (and common ways to accidentally break it), redundant ownership, and regular auditing to ensure code rot is avoided. I have yet to meet another architect that sees their job the way I see mine. If the system you're working on defies complete understanding, you can probably blame your architect.
- elevenoh 7y ago
  I like this take on software arch. essence.
- gen220 7y ago
  Thank you for a beautiful description of your philosophy, and your philosophy itself. I wholeheartedly agree with you, and see things the same way; though I haven't seen it expressed so clearly before. I'm curious – and a little saddened – that you haven't met anyone else thinking this way? Anecdotally, there are maybe 10 out of 200 engineers at my current company who demonstrably take this "Quality of Craft" part seriously and explicitly: they think in a similar way to you. It's a rare set of skills: empathy for future humans, understanding how the universal theory is likely to evolve, caring enough about it to enforce it consistently, and effectively actuating on all of these thoughts and emotions. Those add up to somebody special, I suppose. Anyways, I think I'll link to this comment many times in the future; thank you again.
adrianmonk 7y ago
I agree, and I vastly prefer this mindset. But saying "computers can be understood" is a bit like saying "loans can be paid off". Both absolutely true in principle. In practice, much to my chagrin, it isn't always feasible. I've had to learn the art of strategically choosing which things shall remain black boxes for the time being. My curiosity doesn't disappear, but I can't go doing a depth-first search through all knowledge all the time.
- outworlder 7y ago
  > I can't go doing a depth-first search through all knowledge all the time No-one can, not even the author. What we have to understand is: 1) Which components exist. Can't understand what you don't know exists. 2) How those components interface, at a high level. Once you know which black boxes exist and how they are interconnected, you may then choose to open them. For instance, we know that a keypress is detected by the keyboard and sent to the computer, and it will eventually appear somewhere (graphical terminal?). Do we need to know how the wiring works? Only if we want to understand why multiple simultaneous keypresses don't always work. Do you need to understand how information is encoded? Or how the USB protocol works (assuming it's USB)? Or what are keyboard interrupts, and so on. You'll end up learning about font rendering starting from a keypress. You may not need to know how these things work, but it's useful to know they are there, so you can easily find information when you need to. You are more likely to worry about terminal escape codes rather than keyboard wiring, but still. That goes for anything else. Do you need to worry how compilers work? Maybe not. But a high level understanding is important, so you won't immediately say stuff like "language X is faster than Y because it's compiled". Yeaahh, sure, it's possible that it is true. Since we know what compilers do, and they are not a magical "get faster code box", we'll probably say... "it depends, I don't know. Let's benchmark/disassemble/etc".
- jimbokun 7y ago
  > I've had to learn the art of strategically choosing which things shall remain black boxes for the time being. "For the time being" being the operative words, and knowing when the right time is to open up that black box. But it's important to be confident about opening that box when it's time to do so, and knowing you will be able to figure it out.
petrogradphilos 7y ago
> You will never understand every detail of the implementation of every level on that stack; but you can understand all of them to some level of abstraction, and any specific layer to essentially any depth necessary for any purpose. As David Deutsch argues in The Beginning of Infinity, this is true not just for computers, but for everything.

170 points by miked85 7y ago

The author has clearly never chased a heisenbug in a 25 year old Makefile for embedded systems. The kind where a timing bug, plus NFS automounter, plus WAN weather, equals an unexpected PATH, resulting in a different minor version of a linker than expected, thereby changing the order of static initializers on some targets, thereby resulting in sshd not checking passwords, but only on MIPS and only with the 16 mbps flash config.

fit2rule 7y ago
.. you thus provide a perfect example of the claim made.
evmar 7y ago
The author had a followup post about this that I found more interesting: https://blog.nelhage.com/post/systems-that-defy-understanding/ https://blog.nelhage.com/post/systems-that-defy-understandin...
- munchbunny 7y ago
  As someone who works on a system that spans 10^big nodes, I really like the point in that post. At a certain level of complexity even your most baseline assumptions will have counterexamples, and 99.9% uptime means 1 in 1000 nodes are in various states of failure at any time. You have to build fault tolerance into the design at each layer to keep errors at base layers from snowballing into real outages. And that’s inherently at odds with the “fail fast” principle, so you have to spend a lot of energy on how to fail fast inwardly but stay operational outwardly.
geofft 7y ago
All of those can be understood, no? (I'd expect to go at least that deep in a postmortem of that problem, and I regularly do go about that deep in similar postmortems.)
- ken 7y ago
  That might be stretching the definition of "deterministic". If it depends on implicit state, precise timing, external conditions, etc., I would say it's closer to fitting the definition of "chaotic" than "deterministic". With perfect observation and recording, it may be possible to see a tornado and work backwards to figure out which butterflies flapped their wings to cause it, but that would still be a long ways off from being able to train all butterflies to move only in ways which won't cause tornados.
  - geofft 7y ago
    Sure, but you don't need to know which butterflies flapped their wings - you just need to know "if there's a tornado, the house falls down, and we're in tornado country." In the example above, it seems sufficient and entirely testable to conclude, if for whatever reason NFS is flaky, $PATH resolution will skip the directory with the right linker and fall back to the old one, and if for whatever reason you call the old linker, it has different behavior about static initializer order. That's enough to both describe the problem, reproduce it, and fix it. No, you're not going to be able to identify which particular packet was lost, but (and I say this as someone who's maintained production systems that rely on cross-continent NFS) you expect that packets can be lost and you figure out how to make the system robust against it. Don't put multiple files with the same name on $PATH, or call the linker by its full path, or something. And that leaves you a very far distance from "something spooky is happening with the computer and so passwords weren't being checked today, who knows."

mjw1007 7y ago

I think this is a good attitude. I wish that people who publish advice on writing documentation would put more emphasis on writing for people who have this attitude. I see too much emphasis on providing usage examples, and on "write for a reader who is trying to get something specific done", rather than explaining the system for someone who wishes to understand it.

api 7y ago
The emphasis on getting things done vs. understanding is another manifestation of our age's extreme time poverty. People don't have time to understand anything. Add to that the fact that computing is so trendy. Take the time to learn something -- web development, mobile development, the latest trendy language -- and it won't be hip anymore in 5-10 years. As an experienced programmer I only take the time to deeply understand things that my intuition tells me will be around for a while and for which understanding is a major benefit to me. My intuition is guided by a mix of market share, speed of change in the target market, and robustness of the thing in question.
- EvanAnderson 7y ago
  It's another manifestation of the classic "it's expensive to be poor" situation. My personal struggle is with making the call re: understanding vs. getting things done compared with the return on investment. Understanding things is fun, though, and our systems are built upon enough analogy that understanding usually transfers (if only in part) to something else down the road.
  - thisisnico 7y ago
    Realistically it depends on how invested I am in the system. If it's something that I'm using all-the-time, it absolutely makes sense for me to have a deeper understanding because that understanding could save me time and give me a better perspective on why certain aspects of the system are the way they are. If it's something I use only as an auxiliary system, I don't invest as much time in understanding the system, only what the system can do for me. A prime example of understanding how to use the system, but not the system itself is e-commerce. There are 17+ year old kids making 100K+ a month with e-commerce, having no programming skills or a deep understanding of the web or computers. They are experts at the tools they use, but honestly are not "computer" people. We've abstracted e-commerce to the point that it's not even necessary to deeply understand the web, with tools that exist now (Shopify etc.)

Koshkin 7y ago

But sometimes trying to understand something just isn't worth it.

nemetroid 7y ago
Indeed, as the article expands on in the "Pitfalls of this approach" section.
umvi 7y ago
This happened to me. We were seeing a USB issue between a Marvell SoC we purchased and a Qualcomm SoC. The more I tried to understand this issue, the deeper the rabbit hole went until we were in the weeds of the XHCI controller looking at scatter/gather operations, etc. Eventually we bought a USB protocol analyzer that seemed to indicate that the actual host controller had a bug. We contacted the Marvell, who in turn contacted the vendor of the USB core (Synopsys). They bisected USB cores with an FPGA until they found a day which had a breaking change. Unfortunately, it appears Synopsys didn't use version control with their source verilog a few years ago (they just had archived daily builds), so while they could say "yep, looks like something that changed on 13 Jan 2015 broke it" they didn't know what change that was. However, before it could be resolved the company I was working for went bankrupt and so I never did get closure on the issue. I sunk many, many months into understanding that issue over the course of several years only to give up because it was near impossible to understand enough to come up with a workaround (and because the company went bankrupt).

empath75 7y ago

What shifted my career was actually an almost opposite revelation. I'm a self-taught programmer who never went to college, and bounced around in entry level sort of "IT jobs" like support for close to a decade -- and I assumed that "real" software engineers and sysadmins were geniuses who did understand everything their systems were doing on a deep level and just typed out perfect code off the top of their head. Then I went to a large tech company, and got some attention for hacking away at a few bash scripts to automate things, got promoted to a sys admin job, and realized that hardly anyone understood systems, or anything at all on a deep level, and that everyone was just hacking away like I was. What I've eventually realized is that while, yes, everything in a complex system in principle can be understood, that human beings have limited bandwidth and memory and that it will absolutely paralyze you if you wait until you understand something thoroughly before diving into do work on it. I jumped into AWS without understanding it, Chef without understanding it, Jenkins without understanding it, and Kubernetes without understanding of it, and every time I did that, my salary went up by 20-50%, to the point where i'm making something like 3-4 times now than what I was making 5 years ago. There are two reasons to spend the time to learn something on the job: 1) You need to learn it to do the task that you're working on right now. Whether it's troubleshooting a problem or trying to install a new system. If you're trying to get a kubernetes cluster running, you absolutely do not need to know all the details of how docker containers work. While it's interesting, and it will probably be necessarily to learn it eventually, it would be a waste of time to do that before just getting started deploying kubernetes. 2) Satisfying your own personal curiosity -- sometimes things are just inherently interesting to you, and it's usually worth chasing those things down, I've found. It may not be immediately applicable, but usually I'll figure out how to apply it somewhere.

jcranmer 7y ago
> 2) Satisfying your own personal curiosity -- sometimes things are just inherently interesting to you, and it's usually worth chasing those things down, I've found. It may not be immediately applicable, but usually I'll figure out how to apply it somewhere. A few months ago, while searching for a personal project, I stumbled across a mention that the performance events on Linux can be used to set hardware watchpoints. Literally the very next day at work, we had a problem where we needed a watchpoint but gdb was too intrusive to timing to let the bug happen, so I was able to fall back on perf to confirm the stack trace of the site corrupting memory. Debugging is ultimately about hypothesizing which part of the system is failing and why, so understanding more of how your system works is likely to be beneficial towards future debugging sessions.

troughway 7y ago

From the post: "My advice for a practical upshot from this post would be: cultivate a deep sense of curiosity about the systems you work with. Ask questions about how they work, why they work that way, and how they were built." This is the poignant thing about it all. The individual for who this post is written does not need to be told to cultivate this. The individual who does need to be told, on the other hand, is unlikely to. In some ways the post reads as a "keep doing what you are doing" reminder.

hinkley 7y ago
Sometimes, it's good for people who engage in persuasive speaking to talk to each other and compare notes. "Preaching to the choir" is known in some circles as rehearsing. Similarly, I have owned a couple of books I got almost nothing out of, other than something I can hand to people who ask a lot of questions. They're good books to own. One is quite beat up at this point, but worth it.

bitwize 7y ago

Again -- "Walk into the fire, Morty. Embrace the suck." https://news.ycombinator.com/item?id=22601623 https://news.ycombinator.com/item?id=22601623

umvi 7y ago

Counterexample: Convolutional Neural Networks (CNNs) (or any ML model, really) We still don't fully understand how or why they work so well. If you have software that queries a CNN and the CNN returns some prediction, there's absolutely no way of knowing or understanding the reasoning behind the decision. It's a black box of magical layers and weights that have been "trained" to make good predictions. If you have fraud detection software using ML and it flags something as fraud, that's the extent of the information you get. In this case the computer cannot be understood - it cannot explain to you how it arrived at the conclusion that something is fraud, nor can you explain how the computer arrived at the decision. It just has huge tables of weights and biases that magically "know" when something is fraud.

thisisbrians 7y ago
True, but you also aren't tasked with going in and manually adjusting weights to improve the predictions. And if you are curious enough, you can certainly look at the theory and mathematics that underpin how these systems do work, even if you don't understand exactly how the training data is making its way into your model: someone clearly figured this out and that's why these systems work. Put another way: you can treat an algorithm or routine as a black box if you know it works and understand its context in your system. You can even learn the right questions to ask if you do need to dig into that particular box at some point. Without some abstractions, we'd all be constantly paralyzed. (edit: formatting)
- yuliyp 7y ago
  The problem comes when you need to peer inside the black box because it's working counter to your expectations of it: With most systems you can do so by reading code / documentation etc. With ML you're kind of stuck. You can conduct experiments to see what changes about the behavior of the model, but even that proves tricky when the model has thousands of inputs.
JoshTriplett 7y ago
There are numerous research results about building understandable, analyzable neural network models. Most models don't bother, because that isn't a necessary step to get reasonable answers. But it's possible.

mwcampbell 7y ago

I’d like to add another pitfall: obsessing over optimizing or eliminating intermediate layers when one should be focusing on meeting business requirements in higher layers. I’m prone to this one myself. This one is tough in the current move-fast-and-disrupt-things environment of startups, or even teams within larger companies that are trying to be nimble like a startup.

daze42 7y ago
This is the hardest for me. It's almost like those two forces are in opposition to each other. I'd love to find an industry where they could align.
- chubot 7y ago
  Yeah I think there is a tension there. If you care about systems and internal quality and want to get paid for it, I'd suggest software performance or security as promsiing fields. Performance and security are both cross-cutting concerns that span layers, i.e. you're not just thinking at the application level. It really is a separate kind of thinking and a separate kind of work. Though it seems that it's mostly large companies (e.g. "big tech" and a few other places) that care enough about performance and security to have dedicate staff for them. Most other shops are too busy with "business stuff", and they may reasonably want to outsource that work to experienced consultants.
marcosdumay 7y ago
I guess the keyword here is "obsessing". Software performance and maleability have serious impacts on business and can not be overlooked. But there is an optimum, and it is usually much lower than what most developers want to create. Honestly, I've learned to take fulfillment from the value my software adds to people, instead of it's abstract quality. But well, I still overengineer things all the time.

gpanders 7y ago

> A belief in the understandability of software systems can very easily become a need to understand the systems you work with. I relate to this point especially, as this is how I often am as well. I do a lot of work with FPGAs and embedded systems, and the tools offered by the vendors in this space are often huge and complex in an attempt to create an easy "happy path" for hardware engineers who don't want to/can't learn software engineering and just want to get something working. I, however, stubbornly chose to eschew these tools in favor of "doing it from scratch". In the end I think it was 100% worth it as this process deepened my understanding of the field and has left me with some valuable skills, but the end result was that it took me weeks to do something that another colleague could have done in a day or two, maybe less. I understand the underlying system and implementation better, but that may or may not be considered valuable depending on the project and your company's culture.

m0ther 7y ago

I am a senior software architect. My job is to balance performance against complexity. If my system is slow, it's my fault. If debugging or expanding the system is too difficult, it's my fault. If someone wants to know how the system or business works in depth, I am the one that they should come to. I spend the majority of my time chasing down, enforcing, and simplifying the universal theory of our business (the core of our software solution). The universal theory of our business is a living collection of concepts, designed to accurately model non-virtual concerns in virtual space. If there are too many edge cases, it is a sign that the universal theory is inaccurate, or not robust enough in some areas. If there are too many bugs, it is a sign that the universal theory has not been communicated or enforced well enough, is inaccurate, or is too complex in some areas. As our business grows, or our understanding of the business expands, the universal theory will change; at times dramatically. Malleability (the ability to adapt our software to these changes effectively and efficiently) is one of my top two concerns; the other is latency (how long it takes for any one request to get a complete response). The theory shrinks and becomes better documented as it evolves; the goal is to move from describing behaviors as correlation to describing them as accurate causation. To fill in the blanks. I should mention here that this does not mean every line of code in my projects is easy to understand. Writing a fast system of high complexity requires at least some components that are written exclusively for the computer's benefit (that is, highly optimized and inherently difficult to read). These components should be written with clear documentation, clearly defined public members, written discussions of why it works the way it does (and common ways to accidentally break it), redundant ownership, and regular auditing to ensure code rot is avoided. I have yet to meet another architect that sees their job the way I see mine. If the system you're working on defies complete understanding, you can probably blame your architect.

elevenoh 7y ago
I like this take on software arch. essence.
gen220 7y ago
Thank you for a beautiful description of your philosophy, and your philosophy itself. I wholeheartedly agree with you, and see things the same way; though I haven't seen it expressed so clearly before. I'm curious – and a little saddened – that you haven't met anyone else thinking this way? Anecdotally, there are maybe 10 out of 200 engineers at my current company who demonstrably take this "Quality of Craft" part seriously and explicitly: they think in a similar way to you. It's a rare set of skills: empathy for future humans, understanding how the universal theory is likely to evolve, caring enough about it to enforce it consistently, and effectively actuating on all of these thoughts and emotions. Those add up to somebody special, I suppose. Anyways, I think I'll link to this comment many times in the future; thank you again.

adrianmonk 7y ago

I agree, and I vastly prefer this mindset. But saying "computers can be understood" is a bit like saying "loans can be paid off". Both absolutely true in principle. In practice, much to my chagrin, it isn't always feasible. I've had to learn the art of strategically choosing which things shall remain black boxes for the time being. My curiosity doesn't disappear, but I can't go doing a depth-first search through all knowledge all the time.

outworlder 7y ago
> I can't go doing a depth-first search through all knowledge all the time No-one can, not even the author. What we have to understand is: 1) Which components exist. Can't understand what you don't know exists. 2) How those components interface, at a high level. Once you know which black boxes exist and how they are interconnected, you may then choose to open them. For instance, we know that a keypress is detected by the keyboard and sent to the computer, and it will eventually appear somewhere (graphical terminal?). Do we need to know how the wiring works? Only if we want to understand why multiple simultaneous keypresses don't always work. Do you need to understand how information is encoded? Or how the USB protocol works (assuming it's USB)? Or what are keyboard interrupts, and so on. You'll end up learning about font rendering starting from a keypress. You may not need to know how these things work, but it's useful to know they are there, so you can easily find information when you need to. You are more likely to worry about terminal escape codes rather than keyboard wiring, but still. That goes for anything else. Do you need to worry how compilers work? Maybe not. But a high level understanding is important, so you won't immediately say stuff like "language X is faster than Y because it's compiled". Yeaahh, sure, it's possible that it is true. Since we know what compilers do, and they are not a magical "get faster code box", we'll probably say... "it depends, I don't know. Let's benchmark/disassemble/etc".
jimbokun 7y ago
> I've had to learn the art of strategically choosing which things shall remain black boxes for the time being. "For the time being" being the operative words, and knowing when the right time is to open up that black box. But it's important to be confident about opening that box when it's time to do so, and knowing you will be able to figure it out.

petrogradphilos 7y ago

> You will never understand every detail of the implementation of every level on that stack; but you can understand all of them to some level of abstraction, and any specific layer to essentially any depth necessary for any purpose. As David Deutsch argues in The Beginning of Infinity, this is true not just for computers, but for everything.