3 ms·

> Shouldn't they be? No. And the people who make nuclear powerplants and planes certainly don't engage in results-oriented thinking: you could easily take out

by tikhonj 11y ago

> Shouldn't they be? No. And the people who make nuclear powerplants and planes certainly don't engage in results-oriented thinking: you could easily take out half the safety features they build and not run into any catastrophes, but that's not an acceptable risk. Software companies, on the other hand, are perfectly happy taking on long-term and long-tail risk if it's more or less worked in the past, in large part because failure is far more forgivable. (Which, to be clear, is a good thing and and of itself, but does incentivize some unfortunate kinds of behavior.) The whole point is that good decisions lead to bad outcomes and bad decisions lead to good outcomes all the time, but politics pushes people to react strongly to results, ignoring this fact. This is quite different from other fields in engineering which by necessity, tradition, culture or even regulation operate differently. As I said, "results oriented thinking" is a terrible name which leads to confusion, but it's what we have. Unfortunately I can't think of a better one and if I did it probably wouldn't catch on. (Which, amusingly, is an example of [2] above :).)

pron 11y ago
> And the people who make nuclear powerplants and planes certainly don't engage in results-oriented thinking: you could easily take out half the safety features they build and not run into any catastrophes As someone who's once been there (not power plants but ensuring the safety of systems whose failure may endanger the lives of many people), it's much more complicated than that. If by "results oriented" you mean "look at nothing but the bottom line", I think you're disrespecting a lot of people by assuming (with little evidence) that this is indeed the process. I can tell you that it isn't, and not just in safety-critical software. Of course, there may be some bad people as in every field, but I've seen no evidence to suggest that "decision makers" are on average any worse at their job than anyone else. We were result-oriented in the sense that we preferred not to let anything that didn't have industry-proven benefits into our system. Ironically, it is sometimes the safety-critical systems (well, some classes of them) that can afford to take more risk, because they have the resources to build a new system and then run it in production alongside the old one only it's outputs are not piped to real actuators (we called it a "shadow system"), and do that for a long time (a year if not more) before flipping the switch. "Plain" software usually can't afford to be so patient. > The whole point is that good decisions lead to bad outcomes and bad decisions lead to good outcomes all the time Of course. Our disagreement is on what constitutes a good decision. > politics pushes people to react strongly to results, ignoring this fact. I don't think so. I've been a decision maker (though not alone, and sometimes didn't have the final say) in both safety-critical settings and less risky ones, and it's not politics but often the lack of a better metric. I can assure you that we've extensively studied the reasons for each failure or success to the best of our abilities and never looked just at the bottom line if we had any more information. We took quite a few calculated risks (changed our numerical algorithms, chose the yet-unproven-at-the-time real-time Java over C++ for a safety-critical, hard-realtime system and more), but always opted for things that had been proven in the industry (or tried to prove them ourselves). It's not politics that led us to that, but lots of experience. That's because things that work in the lab fail in the field more often than not. Ignoring that fact is just stupid (or, rather, shows a lack of experience in the field), especially when the stakes are high (and they're always high to some degree in the industry). Besides, why risk anything on something that doesn't have strong evidence of significant benefits? We look at people like you to show us those benefits, and if you don't -- or you think you've found clear benefits, but those benefits simply aren't significant enough for us -- why take any risk? We gladly risk quite a bit when there's good reason to believe the payoff would be large enough. Even that (let alone battle-testing) is something some of the things you'd put in the "works" category have yet to demonstrate. That is something we in the industry tell researchers all the time. If you're content doing pure research, do it and don't whine about us not adopting your work. If not, and you would like us to use your stuff, there's a lot more you need to do than your pure research, like extensive applied research with empirical results, and, most importantly, it requires understanding the industry's needs (e.g. in the industry something is not adopted because it is "better"; it will only be adopted if it is better enough to offset the adoption cost plus associated risk). But what you can't do is have it both ways -- do only the pure research and whine about us not using it. Doing so shows a lack of understanding of what the industry is, and reflects badly on those researchers much more than on the industry (which, BTW, has made amazing achievements in software). The industry adopts brand new stuff all the time when the researchers do their job properly (well, to be fair, for some it's a lot easier). Suppose I'm a decision maker and someone says to me, my new sorting algorithm would be 20% faster for the data you're sorting. Well, the adoption cost is nearly zero, I can test they payoff rather easily, there is risk involved but it's mitigated by my ability to switch back, so I find the 20% payoff to be high enough and I go for it. This happens all the time. In fact, millions of people will soon switch to a brand new -- and rather ambitious -- GC algorithm (granted, one that's been tested for nearly a decade). But then some other guy shows up and says, my new programming language would make your development better. How much better? I ask. Much! he says. Talk to me in numbers, I say. By how much would my development cost be reduced? Hmm, he says. I'm not sure; 30%... maybe? So I look at the adoption cost (huge), risk (very high), and find that 30% to be rather low, but let's suppose it's borderline good-enough. Only he's not even sure about the 30% because he has no experience with large systems and he's never even tested his claim. Maybe it's 5%, maybe 70% and maybe -20%! So let me tell you that in that case, I will be very results oriented, look around, see that practically no one else uses that language and those who do see payoffs nowhere near high enough, so I pass[1]... and that's when you say I'm motivated by politics, hostile to academia and afraid of new stuff. [1]: True, it isn't fair. Technologies with high adoption costs incur a much higher testing burden and are required to yield much higher payoffs than those with lower adoption costs. But that's no one's fault, and just the way it is. Researchers in those fields should keep on working, realizing that their payoffs would be high enough to be adopted by the industry every 20 years instead of every 5 as in low-adoption-cost cases.

by tikhonj 11y ago

pron 11y ago

> And the people who make nuclear powerplants and planes certainly don't engage in results-oriented thinking: you could easily take out half the safety features they build and not run into any catastrophes As someone who's once been there (not power plants but ensuring the safety of systems whose failure may endanger the lives of many people), it's much more complicated than that. If by "results oriented" you mean "look at nothing but the bottom line", I think you're disrespecting a lot of people by assuming (with little evidence) that this is indeed the process. I can tell you that it isn't, and not just in safety-critical software. Of course, there may be some bad people as in every field, but I've seen no evidence to suggest that "decision makers" are on average any worse at their job than anyone else. We were result-oriented in the sense that we preferred not to let anything that didn't have industry-proven benefits into our system. Ironically, it is sometimes the safety-critical systems (well, some classes of them) that can afford to take more risk, because they have the resources to build a new system and then run it in production alongside the old one only it's outputs are not piped to real actuators (we called it a "shadow system"), and do that for a long time (a year if not more) before flipping the switch. "Plain" software usually can't afford to be so patient. > The whole point is that good decisions lead to bad outcomes and bad decisions lead to good outcomes all the time Of course. Our disagreement is on what constitutes a good decision. > politics pushes people to react strongly to results, ignoring this fact. I don't think so. I've been a decision maker (though not alone, and sometimes didn't have the final say) in both safety-critical settings and less risky ones, and it's not politics but often the lack of a better metric. I can assure you that we've extensively studied the reasons for each failure or success to the best of our abilities and never looked just at the bottom line if we had any more information. We took quite a few calculated risks (changed our numerical algorithms, chose the yet-unproven-at-the-time real-time Java over C++ for a safety-critical, hard-realtime system and more), but always opted for things that had been proven in the industry (or tried to prove them ourselves). It's not politics that led us to that, but lots of experience. That's because things that work in the lab fail in the field more often than not. Ignoring that fact is just stupid (or, rather, shows a lack of experience in the field), especially when the stakes are high (and they're always high to some degree in the industry). Besides, why risk anything on something that doesn't have strong evidence of significant benefits? We look at people like you to show us those benefits, and if you don't -- or you think you've found clear benefits, but those benefits simply aren't significant enough for us -- why take any risk? We gladly risk quite a bit when there's good reason to believe the payoff would be large enough. Even that (let alone battle-testing) is something some of the things you'd put in the "works" category have yet to demonstrate. That is something we in the industry tell researchers all the time. If you're content doing pure research, do it and don't whine about us not adopting your work. If not, and you would like us to use your stuff, there's a lot more you need to do than your pure research, like extensive applied research with empirical results, and, most importantly, it requires understanding the industry's needs (e.g. in the industry something is not adopted because it is "better"; it will only be adopted if it is better enough to offset the adoption cost plus associated risk). But what you can't do is have it both ways -- do only the pure research and whine about us not using it. Doing so shows a lack of understanding of what the industry is, and reflects badly on those researchers much more than on the industry (which, BTW, has made amazing achievements in software). The industry adopts brand new stuff all the time when the researchers do their job properly (well, to be fair, for some it's a lot easier). Suppose I'm a decision maker and someone says to me, my new sorting algorithm would be 20% faster for the data you're sorting. Well, the adoption cost is nearly zero, I can test they payoff rather easily, there is risk involved but it's mitigated by my ability to switch back, so I find the 20% payoff to be high enough and I go for it. This happens all the time. In fact, millions of people will soon switch to a brand new -- and rather ambitious -- GC algorithm (granted, one that's been tested for nearly a decade). But then some other guy shows up and says, my new programming language would make your development better. How much better? I ask. Much! he says. Talk to me in numbers, I say. By how much would my development cost be reduced? Hmm, he says. I'm not sure; 30%... maybe? So I look at the adoption cost (huge), risk (very high), and find that 30% to be rather low, but let's suppose it's borderline good-enough. Only he's not even sure about the 30% because he has no experience with large systems and he's never even tested his claim. Maybe it's 5%, maybe 70% and maybe -20%! So let me tell you that in that case, I will be very results oriented, look around, see that practically no one else uses that language and those who do see payoffs nowhere near high enough, so I pass[1]... and that's when you say I'm motivated by politics, hostile to academia and afraid of new stuff. [1]: True, it isn't fair. Technologies with high adoption costs incur a much higher testing burden and are required to yield much higher payoffs than those with lower adoption costs. But that's no one's fault, and just the way it is. Researchers in those fields should keep on working, realizing that their payoffs would be high enough to be adopted by the industry every 20 years instead of every 5 as in low-adoption-cost cases.