7 ms·

We can write software like this. The reason why we don't has been explained many times: It's too expensive, by many orders of magnitude.

by kabdib 11y ago

hlieberman 11y ago
I don't mean to say that all code has to be bug free; certainly, I hope the life-safety critical code is. But I think we, as hackers and software engineers, need to consider the ramifications that bugs can have, and then making the decision based on that balance of risk. Take an anonymous currency system, for example. Handwave away whether or not that it's actually possible to build one -- let's say you have a way to create electronic, untraceable money, and let's go one step further and say that there's no personal risk to doing so (e.g., it creates no criminal or civil liability to doing so, and costs nothing). Do you build that, or not?
- WildUtah 11y ago
  "Take an anonymous currency system, for example....Do you build that, or not?" Obviously you do. I don't understand why that would be a question.
  - hlieberman 11y ago
    Even knowing it's likely to be used for buying and selling drugs, child porn, and assassinations? I personally believe that, even given those potential ramifications, the benefits outweigh the costs -- but my personal balance of those is not everyone's.
    - GauntletWizard 11y ago
      Yes, because it will also be used by marginalized groups to buy groceries, by sex workers who would otherwise be stuck paying huge commissions to pimps, to help hide who is buying anti-goverment printings. Every tool can be used for evil - Your drill is just as capable of drilling through someone's skull as wood.
      - XorNot 11y ago
        This is garbage and you know it. You're choosing to be deliberately blind to an accurate assessment of the consequences. If we took the article at its word, you would be put to death. Proposals for codes of ethics are about this exact issue - you don't get to build something that provides capability, then put up your hands and say you "didn't know what it was for" when it's used for something bad.
        GauntletWizard 11y ago
        There is a major distinction between enforcing anti-fraud in advertising capabilities of your construction, and ensuring your construction is not used by anyone for malicious purposes. To claim otherwise is fraudulent.
- kabdib 11y ago
  I've thought about this one a lot: Yes. Definitely. "What about child porn?" We have that already; not having anonymous cash won't stop it. "What about tax fraud?" Pressuring governments to re-think corrupt, unbalanced tax systems might be a very great good. And so on. I have no question that it would be ultimately better to have this than not, although governments (forever forgetting who is really in charge) don't like it.
PhasmaFelis 11y ago
If it's financially impossible to make a product that doesn't harm people, you don't say "gosh, sorry, but your safety is incompatible with our fiscal goals." You cancel the fucking product. And beyond that, I don't believe for a moment that--to use the example from the article--it would have increased Grindr's costs by 1000x if they had taken a little while to brainstorm possible negative consequences of an app that lets gay people be anonymously tracked and located worldwide. "Too expensive" is code for "we've decided that the safety of our users is less important than maximizing our profit margin."
- roel_v 11y ago
  That's nonsense because it's so binary. Let's say you have product that takes 5 engineers to build without any consideration for security and safety whatsoever, and it gives you a profit margin of x. Now say you add one engineer, which halves risk, and gives you a margin of (0.9 * x). Then you add another, halves risk again, profit is (0.8 * x). And so on. How many engineers should this company add to act 'ethically'? According to you, they need to cancel the product (I think - only for an ungenerously rigid interpretation of your comment). Risk is not 'it harms people or not'. Knowing whether your product harms people is more difficult still. And the question of who is responsible, more still. (and that's not just between the supplier and the user - also to other actors. To take the Grindr example, was it an engineering problem, a product design problem, a user education problem or a governance problem? I could argue for all of them. Who should act to mitigate the risk? I don't know, and it's not as clear-cut (in the abstract) as some here are letting on).
- vinceguidry 11y ago
  I agree with your sentiment, but I am unclear as to how to actually implement it. An imperative to cancel the product in the absence of a court judgment seems a bit extreme. I think a series of escalation steps is in order. Escalation steps inevitably involve a 'nuclear option' whose size and expense relegate it to just the most sure and most aggrieved wrong parties. These days that option is a class-action lawsuit. It seems to work best when deterrence is the primary goal, plaintiffs rarely ever get anything of value out of it. It works as you would expect, but one really wants a pathway to punish negligence without having to get lawyers involved. I believe this is usually done with market solutions to make 'buyer beware' less awful on the ignorant. But the sheer scale of the stated problem, forcing all web service developers everywhere, to consider possible lines of attack on their software, under pain of loss of livelihood, seems too general and too unenforceable to be useful.
calinet6 11y ago
This is simply false. True systemic quality will lower costs, speed production, and increase value all at once. The reason is that, beyond a certain organizational complexity, we fail at creating the systems necessary to create the positive feedback loops necessary for quality, low cost, and fast execution to flourish naturally. Instead, we live in dysfunctional organizations where the tradeoffs prevail, and we choose speed and cost over quality in nearly all cases. Most of the time, we don't understand how to even begin to think about quality at an organizational level. It's fully possible. Quality pioneer W. Edwards Deming explained the full system for creating an organization that is capable of quality in the first place. It mainly involves systems thinking, knowledge of variation, understanding psychology deeply, and creating systems of effective learning—and management and leadership versed in those new leadership competencies. Instead our instincts are to blame, to punish, to hold accountable, effectively to leave each engineer as an island in charge of his or her own level of quality—when an organization and its production is so far removed from individual control it's not even funny. We absolutely can write software like this—and we can do it quicker, and for lower cost. We need to work on the systems that produce and influence that quality/speed/cost relationship—not each individual part in isolation. There is a positive cycle to be achieved. https://en.wikipedia.org/wiki/W._Edwards_Deming https://en.wikipedia.org/wiki/W._Edwards_Deming
- eloff 11y ago
  You've stated that it's not more expensive. But everyone in the industry knows otherwise. Unless you can produce some evidence to the contrary, I'm going to insist it's you who is incorrect here.
  - calinet6 11y ago
    That's a highly simplistic and incorrect viewpoint—and the correctness of a model is not dependent on popular vote. When looking at increasing quality (essentially, scope) in isolation of systems and the environment in which that quality is created, then the only way to increase quality is to increase cost. That much is true, but it's also incomplete. When looking at and eventually optimizing the whole system—from the customer and their needs, to the leadership of the company, to the clarity of purpose, to values and culture, to the intrinsic motivation of your employees, to methods of management, to your learning and education processes, to your processes of production, to your communication links, to your output and how it's measured, and how each is controlled and understood in great detail—you can and will improve quality much more than you could by simply increasing cost in isolation—and at the same time, you'll decrease your overall costs in the long term, you'll speed up production, and you'll increase sales and customer satisfaction. The end result is a better product, for lower cost, with more predictability, proud happy employees, and faster to boot. That's what Quality really means. Looking at quality in terms of cost alone is the naive mistake. Look at the whole system. And for some evidence, look at Japan circa 1948. These exact methods of systemic quality, "total quality control" and statistical quality control taught to Japan by W. Edwards Deming turned them from the ruins of World War II into the second largest economy in the world, to the point of such success that Japanese companies were putting American ones to shame by the 70's. They did this by following exactly the model I've described above, decreasing total costs, increasing quality, and improving time to market, all by focusing on their total systems across the organization. The proof is in the success of an entire country's economy, I don't know how it can get much clearer than that. It is not my fault that everyone in our industry has forgotten these lessons on running companies, but the prevailing methods remain incorrect, as does your original statement.

We can write software like this. The reason why we don't has been explained many times: It's too expensive, by many orders of magnitude.

by kabdib 11y ago

We can write software like this. The reason why we don't has been explained many times: It's too expensive, by many orders of magnitude.

hlieberman 11y ago

I don't mean to say that all code has to be bug free; certainly, I hope the life-safety critical code is. But I think we, as hackers and software engineers, need to consider the ramifications that bugs can have, and then making the decision based on that balance of risk. Take an anonymous currency system, for example. Handwave away whether or not that it's actually possible to build one -- let's say you have a way to create electronic, untraceable money, and let's go one step further and say that there's no personal risk to doing so (e.g., it creates no criminal or civil liability to doing so, and costs nothing). Do you build that, or not?

WildUtah 11y ago
"Take an anonymous currency system, for example....Do you build that, or not?" Obviously you do. I don't understand why that would be a question.
- hlieberman 11y ago
  Even knowing it's likely to be used for buying and selling drugs, child porn, and assassinations? I personally believe that, even given those potential ramifications, the benefits outweigh the costs -- but my personal balance of those is not everyone's.
  - GauntletWizard 11y ago
    Yes, because it will also be used by marginalized groups to buy groceries, by sex workers who would otherwise be stuck paying huge commissions to pimps, to help hide who is buying anti-goverment printings. Every tool can be used for evil - Your drill is just as capable of drilling through someone's skull as wood.
    - XorNot 11y ago
      This is garbage and you know it. You're choosing to be deliberately blind to an accurate assessment of the consequences. If we took the article at its word, you would be put to death. Proposals for codes of ethics are about this exact issue - you don't get to build something that provides capability, then put up your hands and say you "didn't know what it was for" when it's used for something bad.
      - GauntletWizard 11y ago
        There is a major distinction between enforcing anti-fraud in advertising capabilities of your construction, and ensuring your construction is not used by anyone for malicious purposes. To claim otherwise is fraudulent.
kabdib 11y ago
I've thought about this one a lot: Yes. Definitely. "What about child porn?" We have that already; not having anonymous cash won't stop it. "What about tax fraud?" Pressuring governments to re-think corrupt, unbalanced tax systems might be a very great good. And so on. I have no question that it would be ultimately better to have this than not, although governments (forever forgetting who is really in charge) don't like it.

PhasmaFelis 11y ago

If it's financially impossible to make a product that doesn't harm people, you don't say "gosh, sorry, but your safety is incompatible with our fiscal goals." You cancel the fucking product. And beyond that, I don't believe for a moment that--to use the example from the article--it would have increased Grindr's costs by 1000x if they had taken a little while to brainstorm possible negative consequences of an app that lets gay people be anonymously tracked and located worldwide. "Too expensive" is code for "we've decided that the safety of our users is less important than maximizing our profit margin."

roel_v 11y ago
That's nonsense because it's so binary. Let's say you have product that takes 5 engineers to build without any consideration for security and safety whatsoever, and it gives you a profit margin of x. Now say you add one engineer, which halves risk, and gives you a margin of (0.9 * x). Then you add another, halves risk again, profit is (0.8 * x). And so on. How many engineers should this company add to act 'ethically'? According to you, they need to cancel the product (I think - only for an ungenerously rigid interpretation of your comment). Risk is not 'it harms people or not'. Knowing whether your product harms people is more difficult still. And the question of who is responsible, more still. (and that's not just between the supplier and the user - also to other actors. To take the Grindr example, was it an engineering problem, a product design problem, a user education problem or a governance problem? I could argue for all of them. Who should act to mitigate the risk? I don't know, and it's not as clear-cut (in the abstract) as some here are letting on).
vinceguidry 11y ago
I agree with your sentiment, but I am unclear as to how to actually implement it. An imperative to cancel the product in the absence of a court judgment seems a bit extreme. I think a series of escalation steps is in order. Escalation steps inevitably involve a 'nuclear option' whose size and expense relegate it to just the most sure and most aggrieved wrong parties. These days that option is a class-action lawsuit. It seems to work best when deterrence is the primary goal, plaintiffs rarely ever get anything of value out of it. It works as you would expect, but one really wants a pathway to punish negligence without having to get lawyers involved. I believe this is usually done with market solutions to make 'buyer beware' less awful on the ignorant. But the sheer scale of the stated problem, forcing all web service developers everywhere, to consider possible lines of attack on their software, under pain of loss of livelihood, seems too general and too unenforceable to be useful.

calinet6 11y ago

This is simply false. True systemic quality will lower costs, speed production, and increase value all at once. The reason is that, beyond a certain organizational complexity, we fail at creating the systems necessary to create the positive feedback loops necessary for quality, low cost, and fast execution to flourish naturally. Instead, we live in dysfunctional organizations where the tradeoffs prevail, and we choose speed and cost over quality in nearly all cases. Most of the time, we don't understand how to even begin to think about quality at an organizational level. It's fully possible. Quality pioneer W. Edwards Deming explained the full system for creating an organization that is capable of quality in the first place. It mainly involves systems thinking, knowledge of variation, understanding psychology deeply, and creating systems of effective learning—and management and leadership versed in those new leadership competencies. Instead our instincts are to blame, to punish, to hold accountable, effectively to leave each engineer as an island in charge of his or her own level of quality—when an organization and its production is so far removed from individual control it's not even funny. We absolutely can write software like this—and we can do it quicker, and for lower cost. We need to work on the systems that produce and influence that quality/speed/cost relationship—not each individual part in isolation. There is a positive cycle to be achieved. https://en.wikipedia.org/wiki/W._Edwards_Deming https://en.wikipedia.org/wiki/W._Edwards_Deming

eloff 11y ago
You've stated that it's not more expensive. But everyone in the industry knows otherwise. Unless you can produce some evidence to the contrary, I'm going to insist it's you who is incorrect here.
- calinet6 11y ago
  That's a highly simplistic and incorrect viewpoint—and the correctness of a model is not dependent on popular vote. When looking at increasing quality (essentially, scope) in isolation of systems and the environment in which that quality is created, then the only way to increase quality is to increase cost. That much is true, but it's also incomplete. When looking at and eventually optimizing the whole system—from the customer and their needs, to the leadership of the company, to the clarity of purpose, to values and culture, to the intrinsic motivation of your employees, to methods of management, to your learning and education processes, to your processes of production, to your communication links, to your output and how it's measured, and how each is controlled and understood in great detail—you can and will improve quality much more than you could by simply increasing cost in isolation—and at the same time, you'll decrease your overall costs in the long term, you'll speed up production, and you'll increase sales and customer satisfaction. The end result is a better product, for lower cost, with more predictability, proud happy employees, and faster to boot. That's what Quality really means. Looking at quality in terms of cost alone is the naive mistake. Look at the whole system. And for some evidence, look at Japan circa 1948. These exact methods of systemic quality, "total quality control" and statistical quality control taught to Japan by W. Edwards Deming turned them from the ruins of World War II into the second largest economy in the world, to the point of such success that Japanese companies were putting American ones to shame by the 70's. They did this by following exactly the model I've described above, decreasing total costs, increasing quality, and improving time to market, all by focusing on their total systems across the organization. The proof is in the success of an entire country's economy, I don't know how it can get much clearer than that. It is not my fault that everyone in our industry has forgotten these lessons on running companies, but the prevailing methods remain incorrect, as does your original statement.