16 ms·

This is an outrage. It is also a perfect example of how software is used to create increasingly more elaborate and faceless bureaucracies that force individual

This is an outrage. It is also a perfect example of how software is used to create increasingly more elaborate and faceless bureaucracies that force individuals to spend more and more time contending with them. Somehow software has become the ultimate vehicle for bureaucratic violence. Software is simultaneously infallible and the perfect scapegoat. The inmate who lost their phone privileges for 30 days is an example. They did nothing wrong but the computer says so and nothing can be done. The computer is right in the sense that its decision cannot be undone, and solely to blame since no human can undo its edict or be held accountable, apparently. It is tragic and absurd. There was an Ask HN question the other day where the poster asked if the software we are building is making the world a better place. There were hardly any replies at all. Is this because for the most part our efforts in producing software are actually doing the opposite? It certainly seems that way reading articles like this.

ehnto 6y ago
It's such a hard question to answer, because software doesn't exist in a vacuum. Hopefully this example is relevant: You're a software developer maintaining an eCommerce platform, on the one hand your platform helps perpetuate low margin and wasteful consumerism, on the other hand your software enables small businesses to compete in the new online world. Consumerism is bad, but commerce is as old as civilization and supports all of our lifestyles, so on a macro level you're in a tough spot. You're a talented developer putting their skills to work building something the community needs, I personally think that means you're doing good work in the context of your society, but it is difficult to say if it's making the world a better place. Social media is the same. On the one hand, it connects family and friends, on the other it drives narcissisms, consumerism and misinformation. You almost have to try and calculate the "Net Good" or "Net Bad" of a type of software and see how the cards fall. For social media I would suggest that it's currently in a "Net Bad" situation, causing more harm than good for example.
lbriner 6y ago
Software that infringes on the public (even if they are criminals) as opposed to software that people can opt to use or not, needs to have a very serious question asked at design time: If the software produces an incorrect result, what mechanism exists to override it/audit it/provide damages etc. The fact people are not asking that is worrying. I understand why the system was not designed to do something that happened later (even if it could have been reasonably foreseen) but the fact that it was implemented with no override is really the scandal. I don't know whether this comes down to an amount of power that exists in a Governor that means the rest of the organisation can't say, "sorry Guv, but we can't do this because the software wasn't written to". If TV is to be believed, Governors want things done yesterday and you worry about the problems.
- frompdx 6y ago
  I have a very cynical take. Probably too cynical. The ability to shift blame to software as opposed to the humans responsible for administering a bureaucracy is exactly what makes it so appealing. The question is ignored intentionally.
  - marmaduke 6y ago
    Not cynical enough: if private prisons with a profit motive delay prisoner release, they’ve made more money.
  - blackgirldev 6y ago
    That was my first thought actually. We are probably not cynical enough! In addition to blame shifting, the prison industrial complex is benefitting from having the prisoners stay longer, so there is zero incentive to fix the problem.
- Enginerrrd 6y ago
  As someone with a civil engineering background: This right here is the difference between conventional engineering disciplines where designs require a Stamp from an Engineer of Record who takes on personal responsibility in the event of design failures vs. the current discipline of software engineering. There's a big difference between a software developer and a software engineer, and I think that difference should be codified with a licensure and a stamp like it is in every other engineering field in the states. Software like this ought to require a stamp. A decent analogy is the environmental work I've done. When we come up with solutions and mitigations to environmental problems, like software, we can't always predict the result because of the complexities involved. So we stamp a design, but we, or the agencies responsible for allowing the project often specify additional monitoring or other stipulations with very specific performance guidelines. It's a flexible system and possible to adapt to, but there are real consequences and fines when targets aren't met. When bad things happen, the specifics of what went wrong and why are very relevant and the engineer may be to blame, or the owner/site manager, or the contractor who did the work, or sometimes no one is to be blamed but the agencies are able to say: "Hey this isn't working and needs to be addressed, do it by this date or else." In engineering, there's an enormous amount of public trust given to engineered designs. The engineer takes personal responsibility for that public trust that a building or bridge isn't going to fall down. And if you're negligent, it's a BFD. Given the current level of public trust that we are putting into software systems, it's crazy to me that we haven't adopted a similar system.
mattmcknight 6y ago
I dropped this in a comment elsewhere in this discussion, but also makes sense here... I find the government "requirements" process tends to create situations like this. Rather than build flexible software that puts some degree of trust in the person using it, they tend to overspecify the current bureaucratic process. In many cases, the person pushing for the software is looking to use software to enforce bureaucratic control that they have been unable to otherwise exercise, with the effect of the people the project initiator wants to use the software simply working around it. They then institute all sorts of punishments and controls to insure it must be used. This then results in the kind of insane situation we have here, where you can't do something perfectly legal because "computer says no".
- frompdx 6y ago
  the person pushing for the software is looking to use software to enforce bureaucratic control that they have been unable to otherwise exercise This is frequently my observation as well. In the process of creating stricter control the bureaucrat increases the the power of their bureaucracy while shifting the blame for any problems to a faceless entity. They then institute all sorts of punishments and controls to insure it must be used. This leads me to one of my primary frustrations with the bureaucratization of our lives. Severe consequences are attached to low stakes situations and rational individuals who see the harm caused by the situation are rendered powerless to make changes.
  - malandrew 6y ago
    > In the process of creating stricter control the bureaucrat increases the the power of their bureaucracy while shifting the blame for any problems to a faceless entity. They are basically using software to preserve the problem to which they are the solution. i.e. the shirky principle https://kk.org/thetechnium/the-shirky-prin/ https://kk.org/thetechnium/the-shirky-prin/
  - panic 6y ago
    > Severe consequences are attached to low stakes situations and rational individuals who see the harm caused by the situation are rendered powerless to make changes. You can see the process at work within this very thread -- "And within that chain, there should be legal recourse and, in most cases, penal consequences, especially in the case of inadequate software quality/testing/validation, should the software fail to perform its task correctly." (https://news.ycombinator.com/item?id=26228195 https://news.ycombinator.com/item?id=26228195) People seem unable to imagine any way to improve things except by adding more and more legal consequences. We need to stop doing this!
pm90 6y ago
Software is just a tool, it can be used to build good or bad things. It would be hard to see this in e.g. Scandinavian countries, where incarceration is seen as rehabilitative rather than punitive. In the US, racial discrimination, free market extremism along with "tough on crime" laws have created unimaginably cruel systems; together with private prisons, the goal has been on cutting costs rather than rehabilitating prisoners. Software is just a tool to further that goal.
- frompdx 6y ago
  Indeed, software is an enabling technology and is morally ambivalent just like any other tool. A machine tool can make a medical device to save lives or military device to destroy lives. At the heart of the issue is the intricate web of institutional mouse traps designed to convert low stakes issues into serious offenses. For example, a person who does does respond to a citation for expired vehicle tags is now ensured in a mechanism that turns a minor issue related to tax collection into a real criminal offense. It is up to humans to make these things so. I brought up the Ask HN question mostly because I felt the lack of replies were a silent acknowledgement of the realities of most software endeavors. That they are not making the world a better place. Most aren't going out of their way to make it worse. Probably, it isn't even a consideration.
- P_I_Staker 6y ago
  Is rehabilitation not the main goal in the USA? We call the places correctional facilities, and do other things that are ostensibly there to correct behavior and prevent recidivism. In fact, it seems like most of the things that prevent people from being successful are unintended side effects. Here's a thought: Why do we permit private companies to not hire ex-cons? Why do you just get to decide that you don't want to hold up your civic responsibilities like that? Who wants to work with someone that used to be violent maniac, sleazy thief, or worse? I agree about cost cutting measures and the criminal justice industrial complex. Still we have bigger issues around crime and reconciliation that prevent us from making progress. To be honest, I have trouble understanding how we're going to change, unless the average person can live with someone ruining their life, then spending "only" a year or so in prison and moving on to be successful in a decent paying job. We still find that outrageous in the US, and it's going to be very tough to make progress that way. It's not about making something "a goal", especially in a country like the US, it's about convincing the wealthy and powerful class to do anything at all about it and stop making it worse.
malandrew 6y ago
> The inmate who lost their phone privileges for 30 days is an example. They did nothing wrong but the computer says so and nothing can be done. The computer is right in the sense that its decision cannot be undone, and solely to blame since no human can undo its edict or be held accountable, apparently. It is tragic and absurd. All government software should be open source and anyone should be able to investigate the code and submit bug reports, including inmates. If they know there is something wrong, they have a lot of time on their hands to learn a useful skill to fix these issues. The government should then not be allowed to close a bug as wontfix or invalid without approval from other citizen watchdogs verifying if a bug report is legitimate.
- frompdx 6y ago
  I agree with what you are proposing in principle. However, the notion that it is up to each individual to combat the system when it has wronged them while they languish in some kind of bureaucratic limbo is one of the core sicknesses of our system. Apart from having direct access to the source code and the ability to make pull requests, that is exactly what is happening here. The bureaucrats involved know there is a problem but are leaving it up to individual inmates and their advocates, both inside and outside the system, to sort it out.
  - malandrew 6y ago
    > However, the notion that it is up to each individual to combat the system when it has wronged them while they languish in some kind of bureaucratic limbo is one of the core sicknesses of our system. What's the alternative though? No human system I'm aware of can address this in any cost-effective manner. Linus' Law has been demonstrated as being one of the best human approaches. The only software approach I can think of that has addressed this is fault tolerant voting systems used in avionics (NASA, SpaceX, Boing) where the cost of failure is so high that typically three independent implementations vote on the outcome. It's impractical to treat every software system used in government to be built to the same standard. Even in some of the better run software companies (e.g. Google and Facebook), it's incredibly challenging to achieve system correctness across the entire system. There are always tradeoffs to be made and even in the most critical systems there are limits to how to practically achieve correctness. I work at one of these better run companies specifically on measuring and guaranteeing correctness and detecting failures in correctness both in production systems (via continuous probing) and as part of change management (integration testing the change against the system in production). It's a really hard problem even for us and we have far better engineers than you find working on the overwhelming majority of government software systems. The only alternative (and the one I prefer) is to have less government involvement so that fewer systems are involved and you can have more eyeballs scrutinizing fewer systems. Government is far too big already, and there's too strong a desire to keep making it bigger before we adequately tame the complexity we've already built. The co-dependence between two codifications: (1) the law code and (2) software code, further contributes to ossification that is almost impossible to undo. https://en.wikipedia.org/wiki/Linus%27s_law https://en.wikipedia.org/wiki/Linus%27s_law
dimitrios1 6y ago
We know exactly how to fix it. Our cowardly politicians and toothless regulatory agencies are not up for the challenge. For every piece of software that can directly and materially harm someone's life like this, there should be a chain of responsibility. And within that chain, there should be legal recourse and, in most cases, penal consequences, especially in the case of inadequate software quality/testing/validation, should the software fail to perform its task correctly. Bonus side effect, software quality will go up across the board in the industry.
- bendbro 6y ago
  Nobody will take that contract. Any mistake in the government specifying what they want you to build can be politically manipulated into a legal matter with some probability you'll end up in prison.
  - AdrianB1 6y ago
    Not at all: any mistake in the government specifications is the government's problem. If you have requirements and a requirement tracing matrix to deliverables and you deliver what is required, you are good. But usually in most parts of the world these government contracts are intentionally made ambiguous for so many reasons including corruption and incompetence of the people writing requirements.
- frompdx 6y ago
  My cynical take is that the lack of accountability is exactly what makes software enabled bureaucracy so appealing. If this true, there is no incentive to change.
  - rbanffy 6y ago
    It's a cost/benefit analysis. They'll be taken to court by a few inmates, have to pay them or settle out of court and bill the government for their stays. Since everything is a matter of money, what the company will do is what brings them the most profit after they paid their legal bills. It seems keeping inmates longer pays better than releasing them on time.
dustingetz 6y ago
software is a tool used by people. You are measuring America.
davidkhess 6y ago
What if the root cause is the ever increasing complexity that software is trying to manage? At all levels (legislature, management, bureaucrats, programming languages, developers, testers, users, subjects) we are creating more and more complex situations we ask software and the institutions that produce it to manage for us. But as the complexity goes up and the number of these complex situations increases, are we reaching a point where we outstrip the amount of money, talent and experience our institutions would need to deliver solutions to successfully manage them? With our resources and intelligence as a species being capped, it seems at some point this is inevitable.
PragmaticPulp 6y ago
> The computer is right in the sense that its decision cannot be undone, and solely to blame since no human can undo its edict or be held accountable, apparently. This is why penalties are such an important part of the feedback loop. Obviously we can't go back in time and restore someone's phone privileges, but we can award monetary damages for the mistake. Monetary damages alone won't discourage this behavior, though, as ultimately taxpayers foot the bill. There also must be some degree of accountability for those in charge of the system. Software can't become a tool for dodging accountability. Those in charge of implementing the software, providing the inputs, and managing the outputs must be held accountable for related mistakes. > There was an Ask HN question the other day where the poster asked if the software we are building is making the world a better place. There were hardly any replies at all. Few Ask HN questions get many responses. This is also a loaded question, as HN is notorious for nit-picking every response and putting too much emphasis on the downsides. For example, I know farmers who have increased their farm productivity massively using modern hardware and software. However, if I posted that it would inevitably draw concerns about replacing human jobs, right-to-repair issues, and other issues surrounding the space. The world is definitely better off for having more efficient and productive farming techniques, freeing most of us up to do things other than farm. However, all new advances bring a different set of problems. Instead of trying to force everything into broad categories of better or worse I think it's important to acknowledge that technology makes the world different. Different is a combination of better and worse. The modern world has different problems than we did 100 years ago, but given the choice I wouldn't choose to roll back to the pre-computer era. > It certainly seems that way reading articles like this. Both news and social media have a strong bias toward articles that spark anger or outrage. For me, the whole world stops feeling like a dumpster fire when I disconnect from news and social media for a while. I'm looking forward to the post-COVID era where we can get back to interacting with each other in person rather than gathering around a constant stream of negative stories on social media.
- frompdx 6y ago
  Both news and social media have a strong bias toward articles that spark anger or outrage. Absolutely, and I agree that disconnecting can have positive benefits. On the other hand, at least for me personally, covid has disrupted the mechanisms that normally prevent in depth observation. It has given me time to read books I normally would not have read because that time went to things like waiting for my car to warm up so I can get to work on time, commuting, going out to lunch with co-workers, and going out for drinks with co-workers, friends, and family. What is described in the article is outrageous. My concerns about bureaucracy and software's role in enabling it, on the other hand, have developed separately because I have the time to consider it.
majormajor 6y ago
The lack of desire to get things right throughout the bureaucracy is the problem. The software is just a mechanism. Other organizations that actually care figure out ways to get things right even when the software has issues. You can see in the film Brazil, from 35 years ago, that this was already a problem and concern even without modern software.
- dTal 6y ago
  Much older than that. The blackly humorous 1965 short story "Computers Don't Argue" by Gordon R. Dickson is pretty much the definitive "software as a bureaucracy" story. No spoilers - it's short and well worth it: https://www.atariarchives.org/bcc2/showpage.php?page=133 https://www.atariarchives.org/bcc2/showpage.php?page=133
6gvONxR4sf7o 6y ago
A big problem is that while we might improve the typical use with software, the failure mode is generally ignored and swept under the rug. See google's customer service. You can speed up and improve the average case a thousand times more, driving costs down by maybe a thousand times, or you can bring costs down like 2x but keep the benefits of manual, person centric failure recovery. Even then, non-automation doesn't make it "human" in the sense we all want. A rep in a call center who is only allowed to follow the playbook almost might as well be an automaton for all the freedom they have. Are faceless economies of scale and the bureaucracies they bring the root issue?
amelius 6y ago
There is plenty of useful software. For example: scientific software.
- frompdx 6y ago
  That's true, and I am not arguing that useful software does not exist. Instead, a lot of energy producing software is often not useful, or useful in perverse ways.
eunos 6y ago
I am not sure how software bug is the exclusive enabler since it is plausible as well for administration bug to occur with pen and paper along with the compliant warden.
- frompdx 6y ago
  It's not that software is the exclusive enabler. It is that software is the ideal enabler because of its ability to create a truly faceless entity that seems to exist outside the power of even those who administer it. Of course these issues were always possible without software. Software is just so much more efficient and useful for creating these kinds of issues because it can scale and because it can be the scapegoat.
austincheney 6y ago
The best solution to the problem is to hold developers personally liable for the software they write, as well as the owners. That could mean criminal penalties for negligent violations of industry standards and processes but will mostly result in civil penalties. The second and third order consequences is that developers will insulate themselves behind licensing and proofs of practice like every other industry. Until people actually advocate for real penalties for such harmful violations they don’t care. All their temporary whining and crying is just blowing smoke up our asses.
bondolo 6y ago
Little Britain's recurring bit Computer says "No", has always been a great illustration of this point. https://www.youtube.com/watch?v=0n_Ty_72Qds https://www.youtube.com/watch?v=0n_Ty_72Qds
mewpmewp2 6y ago
I'd argue that software like this has saved people from having to do millions of years worth of mundane work. This news is essentially like a traffic accident. Doesn't mean vehicles in general haven't benefitted the human experience. The fact that it is news worthy is evidence that it doesn't happen too often.
asdfasgasdgasdg 6y ago
> Is this because for the most part our efforts in producing software are actually doing the opposite? It certainly seems that way reading articles like this. I think the most likely explanation is just that people didn't see the question or weren't interested in having the discussion. Most people believe the work they're doing is at worst neutral. A less likely candidate for the reason (but still more likely than your guess) is that people didn't want to be subjected to unfounded criticism of their work from people who don't know anything about it.
hodgesrm 6y ago
> It is also a perfect example of how software is used to create increasingly more elaborate and faceless bureaucracies that force individuals to spend more and more time contending with them. You are attacking the wrong target. It's the government that's broken. This kind of outrage can happen just as easily with pencil and paper. The root cause is the lack of accountability and desire to make the government function better.
- frompdx 6y ago
  Except software allows a scale and efficiency that is impossible with pencil and paper while also creating an ideal scapegoat. Software is being used to avoid accountability at a scale much greater than what was possible with manual process.
  - deleted 6y ago
    [deleted]
  - Kalium 6y ago
    The old saw about computers is that they're designed to allow humans to make millions of mistakes, very quickly and very accurately.
  - phone8675309 6y ago
    Isn't this the basis of the book "IBM and the Holocaust"[1] where the author lays out how IBM's technology helped facilitate the wide scale of Nazi genocide? [1] https://en.wikipedia.org/wiki/IBM_and_the_Holocaust https://en.wikipedia.org/wiki/IBM_and_the_Holocaust
  - Kinrany 6y ago
    > Software is being used to avoid accountability Right, in the same way knives are used to rob people. This is not a new problem: an organization strategically builds an unmanageable bureacracy and then profits off the issue while claiming incompetence. Computers just make said bureaucracy cheaper to operate.
- Tepix 6y ago
alephnil 6y ago
Yes, software can be used as a cover for abuse like in this case, but that happens because some people in power let it happen. For other pieces of software that have consequences for people with more power than prisoners, the society will not allow failures to happen. I only need to mention the MCAS software of Boeing 737 Max for a counterexample. Software does not have its own will. Software is only allowed to make decisions on our behalf because we let it do so.
- frompdx 6y ago
  It is interesting that you bring up the 737 Max. I was actually talking the 737 Max in the context of software being both infallible and the perfect scapegoat with someone last week. The 737 Max is an example of how software was believed to be infallible (it wasn't) and was ultimately the scapegoat for a design flaw. That doesn't mean the 737 Max was something that happened intentionally. However, when the time came to assign blame fingers began pointing at the software. I do agree that software has no will. It is a tool for facilitating our will for better or worse.
whack 6y ago
The following is very illuminating: > Instead of fixing the bug, department sources said employees are attempting to identify qualifying inmates manually... But sources say the department isn’t even scratching the surface of the entire number of eligible inmates. “The only prisoners that are getting into programming are the squeaky wheels,” a source said, “the ones who already know they qualify or people who have family members on the outside advocating for them.” > In the meantime, Lamoreaux confirmed the “data is being calculated manually and then entered into the system.” Department sources said this means “someone is sitting there crunching numbers with a calculator and interpreting how each of the new laws that have been passed would impact an inmate.” “It makes me sick,” one source said, noting that even the most diligent employees are capable of making math errors that could result in additional months or years in prison for an inmate. “What the hell are we doing here? People’s lives are at stake.” Comments like yours seem to glorify a pre-software world filled with manual entry. The reality is that manual entry is even more error-prone, bias-prone, with more people falling through the cracks. If nothing else, software can be uniformly applied at a mass scale, and audited for any and all bugs. And faulty software can be exposed through leaks like the above, to expose and fix systemic problems. Whereas a world of manual entry simply ignores vast numbers of errors and biases which are extremely hard to detect/prove, and even then, can simply be scapegoated with some unlucky individuals, without any effort to fix systemically.
- UweSchmidt 6y ago
  One of the issues is that laws are made on paper and then everyone needs to figure out how to map it to software. Instead, laws should be codified in software and legal APIs should be binding. This would do wonders for efficiency, but also force laws to be cleaned up, be consistent, simple and logical.
  - jakelazaroff 6y ago
    I don't even know what this would entail. Reality is continuous and subjective; computers are not. And there's no reason that "legal APIs" would be any more cleaned up, consistent, simple or logical than our current legal system.
dmead 6y ago
most of us have jobs that service the revenue streams of rich owners. what did you think was going to happen?
rectang 6y ago
> making the world a better place For a large segment of the US electorate, anything that inflicts pain on "bad people" is "making the world a better place". If the software was causing prisoners to be released early, most US voters would be up in arms. But if they're being held too long, the calculus is different. In software terms, for many Americans, a "tough on crime" outcome is a "feature not a bug".
bitL 6y ago
> if the software we are building is making the world a better place No, it's now all about "extracting value", "rent seeking", "subscriptions", "censorship", "monopoly" and "control". We got bribed by FAANG and this is the consequence.
- Bakary 6y ago
  Bribed implies the natural course of the tech worker would be altruistic, which is quite the assumption
danielnixon 6y ago
You might like this https://www.berglas.org/Articles/ImportantThatSoftwareFails/ImportantThatSoftwareFails.html https://www.berglas.org/Articles/ImportantThatSoftwareFails/...

ehnto 6y ago

It's such a hard question to answer, because software doesn't exist in a vacuum. Hopefully this example is relevant: You're a software developer maintaining an eCommerce platform, on the one hand your platform helps perpetuate low margin and wasteful consumerism, on the other hand your software enables small businesses to compete in the new online world. Consumerism is bad, but commerce is as old as civilization and supports all of our lifestyles, so on a macro level you're in a tough spot. You're a talented developer putting their skills to work building something the community needs, I personally think that means you're doing good work in the context of your society, but it is difficult to say if it's making the world a better place. Social media is the same. On the one hand, it connects family and friends, on the other it drives narcissisms, consumerism and misinformation. You almost have to try and calculate the "Net Good" or "Net Bad" of a type of software and see how the cards fall. For social media I would suggest that it's currently in a "Net Bad" situation, causing more harm than good for example.

lbriner 6y ago

Software that infringes on the public (even if they are criminals) as opposed to software that people can opt to use or not, needs to have a very serious question asked at design time: If the software produces an incorrect result, what mechanism exists to override it/audit it/provide damages etc. The fact people are not asking that is worrying. I understand why the system was not designed to do something that happened later (even if it could have been reasonably foreseen) but the fact that it was implemented with no override is really the scandal. I don't know whether this comes down to an amount of power that exists in a Governor that means the rest of the organisation can't say, "sorry Guv, but we can't do this because the software wasn't written to". If TV is to be believed, Governors want things done yesterday and you worry about the problems.

frompdx 6y ago
I have a very cynical take. Probably too cynical. The ability to shift blame to software as opposed to the humans responsible for administering a bureaucracy is exactly what makes it so appealing. The question is ignored intentionally.
- marmaduke 6y ago
  Not cynical enough: if private prisons with a profit motive delay prisoner release, they’ve made more money.
- blackgirldev 6y ago
  That was my first thought actually. We are probably not cynical enough! In addition to blame shifting, the prison industrial complex is benefitting from having the prisoners stay longer, so there is zero incentive to fix the problem.
Enginerrrd 6y ago
As someone with a civil engineering background: This right here is the difference between conventional engineering disciplines where designs require a Stamp from an Engineer of Record who takes on personal responsibility in the event of design failures vs. the current discipline of software engineering. There's a big difference between a software developer and a software engineer, and I think that difference should be codified with a licensure and a stamp like it is in every other engineering field in the states. Software like this ought to require a stamp. A decent analogy is the environmental work I've done. When we come up with solutions and mitigations to environmental problems, like software, we can't always predict the result because of the complexities involved. So we stamp a design, but we, or the agencies responsible for allowing the project often specify additional monitoring or other stipulations with very specific performance guidelines. It's a flexible system and possible to adapt to, but there are real consequences and fines when targets aren't met. When bad things happen, the specifics of what went wrong and why are very relevant and the engineer may be to blame, or the owner/site manager, or the contractor who did the work, or sometimes no one is to be blamed but the agencies are able to say: "Hey this isn't working and needs to be addressed, do it by this date or else." In engineering, there's an enormous amount of public trust given to engineered designs. The engineer takes personal responsibility for that public trust that a building or bridge isn't going to fall down. And if you're negligent, it's a BFD. Given the current level of public trust that we are putting into software systems, it's crazy to me that we haven't adopted a similar system.

mattmcknight 6y ago

I dropped this in a comment elsewhere in this discussion, but also makes sense here... I find the government "requirements" process tends to create situations like this. Rather than build flexible software that puts some degree of trust in the person using it, they tend to overspecify the current bureaucratic process. In many cases, the person pushing for the software is looking to use software to enforce bureaucratic control that they have been unable to otherwise exercise, with the effect of the people the project initiator wants to use the software simply working around it. They then institute all sorts of punishments and controls to insure it must be used. This then results in the kind of insane situation we have here, where you can't do something perfectly legal because "computer says no".

frompdx 6y ago
the person pushing for the software is looking to use software to enforce bureaucratic control that they have been unable to otherwise exercise This is frequently my observation as well. In the process of creating stricter control the bureaucrat increases the the power of their bureaucracy while shifting the blame for any problems to a faceless entity. They then institute all sorts of punishments and controls to insure it must be used. This leads me to one of my primary frustrations with the bureaucratization of our lives. Severe consequences are attached to low stakes situations and rational individuals who see the harm caused by the situation are rendered powerless to make changes.
- malandrew 6y ago
  > In the process of creating stricter control the bureaucrat increases the the power of their bureaucracy while shifting the blame for any problems to a faceless entity. They are basically using software to preserve the problem to which they are the solution. i.e. the shirky principle https://kk.org/thetechnium/the-shirky-prin/ https://kk.org/thetechnium/the-shirky-prin/
- panic 6y ago
  > Severe consequences are attached to low stakes situations and rational individuals who see the harm caused by the situation are rendered powerless to make changes. You can see the process at work within this very thread -- "And within that chain, there should be legal recourse and, in most cases, penal consequences, especially in the case of inadequate software quality/testing/validation, should the software fail to perform its task correctly." (https://news.ycombinator.com/item?id=26228195 https://news.ycombinator.com/item?id=26228195) People seem unable to imagine any way to improve things except by adding more and more legal consequences. We need to stop doing this!

pm90 6y ago

Software is just a tool, it can be used to build good or bad things. It would be hard to see this in e.g. Scandinavian countries, where incarceration is seen as rehabilitative rather than punitive. In the US, racial discrimination, free market extremism along with "tough on crime" laws have created unimaginably cruel systems; together with private prisons, the goal has been on cutting costs rather than rehabilitating prisoners. Software is just a tool to further that goal.

frompdx 6y ago
Indeed, software is an enabling technology and is morally ambivalent just like any other tool. A machine tool can make a medical device to save lives or military device to destroy lives. At the heart of the issue is the intricate web of institutional mouse traps designed to convert low stakes issues into serious offenses. For example, a person who does does respond to a citation for expired vehicle tags is now ensured in a mechanism that turns a minor issue related to tax collection into a real criminal offense. It is up to humans to make these things so. I brought up the Ask HN question mostly because I felt the lack of replies were a silent acknowledgement of the realities of most software endeavors. That they are not making the world a better place. Most aren't going out of their way to make it worse. Probably, it isn't even a consideration.
P_I_Staker 6y ago
Is rehabilitation not the main goal in the USA? We call the places correctional facilities, and do other things that are ostensibly there to correct behavior and prevent recidivism. In fact, it seems like most of the things that prevent people from being successful are unintended side effects. Here's a thought: Why do we permit private companies to not hire ex-cons? Why do you just get to decide that you don't want to hold up your civic responsibilities like that? Who wants to work with someone that used to be violent maniac, sleazy thief, or worse? I agree about cost cutting measures and the criminal justice industrial complex. Still we have bigger issues around crime and reconciliation that prevent us from making progress. To be honest, I have trouble understanding how we're going to change, unless the average person can live with someone ruining their life, then spending "only" a year or so in prison and moving on to be successful in a decent paying job. We still find that outrageous in the US, and it's going to be very tough to make progress that way. It's not about making something "a goal", especially in a country like the US, it's about convincing the wealthy and powerful class to do anything at all about it and stop making it worse.

malandrew 6y ago

> The inmate who lost their phone privileges for 30 days is an example. They did nothing wrong but the computer says so and nothing can be done. The computer is right in the sense that its decision cannot be undone, and solely to blame since no human can undo its edict or be held accountable, apparently. It is tragic and absurd. All government software should be open source and anyone should be able to investigate the code and submit bug reports, including inmates. If they know there is something wrong, they have a lot of time on their hands to learn a useful skill to fix these issues. The government should then not be allowed to close a bug as wontfix or invalid without approval from other citizen watchdogs verifying if a bug report is legitimate.

frompdx 6y ago
I agree with what you are proposing in principle. However, the notion that it is up to each individual to combat the system when it has wronged them while they languish in some kind of bureaucratic limbo is one of the core sicknesses of our system. Apart from having direct access to the source code and the ability to make pull requests, that is exactly what is happening here. The bureaucrats involved know there is a problem but are leaving it up to individual inmates and their advocates, both inside and outside the system, to sort it out.
- malandrew 6y ago
  > However, the notion that it is up to each individual to combat the system when it has wronged them while they languish in some kind of bureaucratic limbo is one of the core sicknesses of our system. What's the alternative though? No human system I'm aware of can address this in any cost-effective manner. Linus' Law has been demonstrated as being one of the best human approaches. The only software approach I can think of that has addressed this is fault tolerant voting systems used in avionics (NASA, SpaceX, Boing) where the cost of failure is so high that typically three independent implementations vote on the outcome. It's impractical to treat every software system used in government to be built to the same standard. Even in some of the better run software companies (e.g. Google and Facebook), it's incredibly challenging to achieve system correctness across the entire system. There are always tradeoffs to be made and even in the most critical systems there are limits to how to practically achieve correctness. I work at one of these better run companies specifically on measuring and guaranteeing correctness and detecting failures in correctness both in production systems (via continuous probing) and as part of change management (integration testing the change against the system in production). It's a really hard problem even for us and we have far better engineers than you find working on the overwhelming majority of government software systems. The only alternative (and the one I prefer) is to have less government involvement so that fewer systems are involved and you can have more eyeballs scrutinizing fewer systems. Government is far too big already, and there's too strong a desire to keep making it bigger before we adequately tame the complexity we've already built. The co-dependence between two codifications: (1) the law code and (2) software code, further contributes to ossification that is almost impossible to undo. https://en.wikipedia.org/wiki/Linus%27s_law https://en.wikipedia.org/wiki/Linus%27s_law

dimitrios1 6y ago

We know exactly how to fix it. Our cowardly politicians and toothless regulatory agencies are not up for the challenge. For every piece of software that can directly and materially harm someone's life like this, there should be a chain of responsibility. And within that chain, there should be legal recourse and, in most cases, penal consequences, especially in the case of inadequate software quality/testing/validation, should the software fail to perform its task correctly. Bonus side effect, software quality will go up across the board in the industry.

bendbro 6y ago
Nobody will take that contract. Any mistake in the government specifying what they want you to build can be politically manipulated into a legal matter with some probability you'll end up in prison.
- AdrianB1 6y ago
  Not at all: any mistake in the government specifications is the government's problem. If you have requirements and a requirement tracing matrix to deliverables and you deliver what is required, you are good. But usually in most parts of the world these government contracts are intentionally made ambiguous for so many reasons including corruption and incompetence of the people writing requirements.
frompdx 6y ago
My cynical take is that the lack of accountability is exactly what makes software enabled bureaucracy so appealing. If this true, there is no incentive to change.
- rbanffy 6y ago
  It's a cost/benefit analysis. They'll be taken to court by a few inmates, have to pay them or settle out of court and bill the government for their stays. Since everything is a matter of money, what the company will do is what brings them the most profit after they paid their legal bills. It seems keeping inmates longer pays better than releasing them on time.

dustingetz 6y ago

software is a tool used by people. You are measuring America.

davidkhess 6y ago

What if the root cause is the ever increasing complexity that software is trying to manage? At all levels (legislature, management, bureaucrats, programming languages, developers, testers, users, subjects) we are creating more and more complex situations we ask software and the institutions that produce it to manage for us. But as the complexity goes up and the number of these complex situations increases, are we reaching a point where we outstrip the amount of money, talent and experience our institutions would need to deliver solutions to successfully manage them? With our resources and intelligence as a species being capped, it seems at some point this is inevitable.

PragmaticPulp 6y ago

> The computer is right in the sense that its decision cannot be undone, and solely to blame since no human can undo its edict or be held accountable, apparently. This is why penalties are such an important part of the feedback loop. Obviously we can't go back in time and restore someone's phone privileges, but we can award monetary damages for the mistake. Monetary damages alone won't discourage this behavior, though, as ultimately taxpayers foot the bill. There also must be some degree of accountability for those in charge of the system. Software can't become a tool for dodging accountability. Those in charge of implementing the software, providing the inputs, and managing the outputs must be held accountable for related mistakes. > There was an Ask HN question the other day where the poster asked if the software we are building is making the world a better place. There were hardly any replies at all. Few Ask HN questions get many responses. This is also a loaded question, as HN is notorious for nit-picking every response and putting too much emphasis on the downsides. For example, I know farmers who have increased their farm productivity massively using modern hardware and software. However, if I posted that it would inevitably draw concerns about replacing human jobs, right-to-repair issues, and other issues surrounding the space. The world is definitely better off for having more efficient and productive farming techniques, freeing most of us up to do things other than farm. However, all new advances bring a different set of problems. Instead of trying to force everything into broad categories of better or worse I think it's important to acknowledge that technology makes the world different. Different is a combination of better and worse. The modern world has different problems than we did 100 years ago, but given the choice I wouldn't choose to roll back to the pre-computer era. > It certainly seems that way reading articles like this. Both news and social media have a strong bias toward articles that spark anger or outrage. For me, the whole world stops feeling like a dumpster fire when I disconnect from news and social media for a while. I'm looking forward to the post-COVID era where we can get back to interacting with each other in person rather than gathering around a constant stream of negative stories on social media.

frompdx 6y ago
Both news and social media have a strong bias toward articles that spark anger or outrage. Absolutely, and I agree that disconnecting can have positive benefits. On the other hand, at least for me personally, covid has disrupted the mechanisms that normally prevent in depth observation. It has given me time to read books I normally would not have read because that time went to things like waiting for my car to warm up so I can get to work on time, commuting, going out to lunch with co-workers, and going out for drinks with co-workers, friends, and family. What is described in the article is outrageous. My concerns about bureaucracy and software's role in enabling it, on the other hand, have developed separately because I have the time to consider it.

majormajor 6y ago

The lack of desire to get things right throughout the bureaucracy is the problem. The software is just a mechanism. Other organizations that actually care figure out ways to get things right even when the software has issues. You can see in the film Brazil, from 35 years ago, that this was already a problem and concern even without modern software.

dTal 6y ago
Much older than that. The blackly humorous 1965 short story "Computers Don't Argue" by Gordon R. Dickson is pretty much the definitive "software as a bureaucracy" story. No spoilers - it's short and well worth it: https://www.atariarchives.org/bcc2/showpage.php?page=133 https://www.atariarchives.org/bcc2/showpage.php?page=133

6gvONxR4sf7o 6y ago

A big problem is that while we might improve the typical use with software, the failure mode is generally ignored and swept under the rug. See google's customer service. You can speed up and improve the average case a thousand times more, driving costs down by maybe a thousand times, or you can bring costs down like 2x but keep the benefits of manual, person centric failure recovery. Even then, non-automation doesn't make it "human" in the sense we all want. A rep in a call center who is only allowed to follow the playbook almost might as well be an automaton for all the freedom they have. Are faceless economies of scale and the bureaucracies they bring the root issue?

amelius 6y ago

There is plenty of useful software. For example: scientific software.

frompdx 6y ago
That's true, and I am not arguing that useful software does not exist. Instead, a lot of energy producing software is often not useful, or useful in perverse ways.

eunos 6y ago

I am not sure how software bug is the exclusive enabler since it is plausible as well for administration bug to occur with pen and paper along with the compliant warden.

frompdx 6y ago
It's not that software is the exclusive enabler. It is that software is the ideal enabler because of its ability to create a truly faceless entity that seems to exist outside the power of even those who administer it. Of course these issues were always possible without software. Software is just so much more efficient and useful for creating these kinds of issues because it can scale and because it can be the scapegoat.

austincheney 6y ago

The best solution to the problem is to hold developers personally liable for the software they write, as well as the owners. That could mean criminal penalties for negligent violations of industry standards and processes but will mostly result in civil penalties. The second and third order consequences is that developers will insulate themselves behind licensing and proofs of practice like every other industry. Until people actually advocate for real penalties for such harmful violations they don’t care. All their temporary whining and crying is just blowing smoke up our asses.

bondolo 6y ago

Little Britain's recurring bit Computer says "No", has always been a great illustration of this point. https://www.youtube.com/watch?v=0n_Ty_72Qds https://www.youtube.com/watch?v=0n_Ty_72Qds

mewpmewp2 6y ago

I'd argue that software like this has saved people from having to do millions of years worth of mundane work. This news is essentially like a traffic accident. Doesn't mean vehicles in general haven't benefitted the human experience. The fact that it is news worthy is evidence that it doesn't happen too often.

asdfasgasdgasdg 6y ago

> Is this because for the most part our efforts in producing software are actually doing the opposite? It certainly seems that way reading articles like this. I think the most likely explanation is just that people didn't see the question or weren't interested in having the discussion. Most people believe the work they're doing is at worst neutral. A less likely candidate for the reason (but still more likely than your guess) is that people didn't want to be subjected to unfounded criticism of their work from people who don't know anything about it.

hodgesrm 6y ago

> It is also a perfect example of how software is used to create increasingly more elaborate and faceless bureaucracies that force individuals to spend more and more time contending with them. You are attacking the wrong target. It's the government that's broken. This kind of outrage can happen just as easily with pencil and paper. The root cause is the lack of accountability and desire to make the government function better.

frompdx 6y ago
Except software allows a scale and efficiency that is impossible with pencil and paper while also creating an ideal scapegoat. Software is being used to avoid accountability at a scale much greater than what was possible with manual process.
- deleted 6y ago
  [deleted]
- Kalium 6y ago
  The old saw about computers is that they're designed to allow humans to make millions of mistakes, very quickly and very accurately.
- phone8675309 6y ago
  Isn't this the basis of the book "IBM and the Holocaust"[1] where the author lays out how IBM's technology helped facilitate the wide scale of Nazi genocide? [1] https://en.wikipedia.org/wiki/IBM_and_the_Holocaust https://en.wikipedia.org/wiki/IBM_and_the_Holocaust
- Kinrany 6y ago
  > Software is being used to avoid accountability Right, in the same way knives are used to rob people. This is not a new problem: an organization strategically builds an unmanageable bureacracy and then profits off the issue while claiming incompetence. Computers just make said bureaucracy cheaper to operate.
Tepix 6y ago

alephnil 6y ago

Yes, software can be used as a cover for abuse like in this case, but that happens because some people in power let it happen. For other pieces of software that have consequences for people with more power than prisoners, the society will not allow failures to happen. I only need to mention the MCAS software of Boeing 737 Max for a counterexample. Software does not have its own will. Software is only allowed to make decisions on our behalf because we let it do so.

frompdx 6y ago
It is interesting that you bring up the 737 Max. I was actually talking the 737 Max in the context of software being both infallible and the perfect scapegoat with someone last week. The 737 Max is an example of how software was believed to be infallible (it wasn't) and was ultimately the scapegoat for a design flaw. That doesn't mean the 737 Max was something that happened intentionally. However, when the time came to assign blame fingers began pointing at the software. I do agree that software has no will. It is a tool for facilitating our will for better or worse.

whack 6y ago

The following is very illuminating: > Instead of fixing the bug, department sources said employees are attempting to identify qualifying inmates manually... But sources say the department isn’t even scratching the surface of the entire number of eligible inmates. “The only prisoners that are getting into programming are the squeaky wheels,” a source said, “the ones who already know they qualify or people who have family members on the outside advocating for them.” > In the meantime, Lamoreaux confirmed the “data is being calculated manually and then entered into the system.” Department sources said this means “someone is sitting there crunching numbers with a calculator and interpreting how each of the new laws that have been passed would impact an inmate.” “It makes me sick,” one source said, noting that even the most diligent employees are capable of making math errors that could result in additional months or years in prison for an inmate. “What the hell are we doing here? People’s lives are at stake.” Comments like yours seem to glorify a pre-software world filled with manual entry. The reality is that manual entry is even more error-prone, bias-prone, with more people falling through the cracks. If nothing else, software can be uniformly applied at a mass scale, and audited for any and all bugs. And faulty software can be exposed through leaks like the above, to expose and fix systemic problems. Whereas a world of manual entry simply ignores vast numbers of errors and biases which are extremely hard to detect/prove, and even then, can simply be scapegoated with some unlucky individuals, without any effort to fix systemically.

UweSchmidt 6y ago
One of the issues is that laws are made on paper and then everyone needs to figure out how to map it to software. Instead, laws should be codified in software and legal APIs should be binding. This would do wonders for efficiency, but also force laws to be cleaned up, be consistent, simple and logical.
- jakelazaroff 6y ago
  I don't even know what this would entail. Reality is continuous and subjective; computers are not. And there's no reason that "legal APIs" would be any more cleaned up, consistent, simple or logical than our current legal system.

dmead 6y ago

most of us have jobs that service the revenue streams of rich owners. what did you think was going to happen?

rectang 6y ago

> making the world a better place For a large segment of the US electorate, anything that inflicts pain on "bad people" is "making the world a better place". If the software was causing prisoners to be released early, most US voters would be up in arms. But if they're being held too long, the calculus is different. In software terms, for many Americans, a "tough on crime" outcome is a "feature not a bug".

bitL 6y ago

> if the software we are building is making the world a better place No, it's now all about "extracting value", "rent seeking", "subscriptions", "censorship", "monopoly" and "control". We got bribed by FAANG and this is the consequence.

Bakary 6y ago
Bribed implies the natural course of the tech worker would be altruistic, which is quite the assumption

danielnixon 6y ago

You might like this https://www.berglas.org/Articles/ImportantThatSoftwareFails/ImportantThatSoftwareFails.html https://www.berglas.org/Articles/ImportantThatSoftwareFails/...