5 ms·

Suppose you have a system, but for legal liability reasons you cannot have this system be open to users (say it's a medical system, and having untrained people

by ezyang 14y ago

Suppose you have a system, but for legal liability reasons you cannot have this system be open to users (say it's a medical system, and having untrained people fuck around with it messes with certification.) However, you make it reasonably easy for users to override any interlocks (i.e. put in no protection). A user finds out about the override, hacks their device, misprograms it, and dies. Is the company that manufactured this device legally liable for this death? If they knew that it was easy to override the safety interlocks, are they then liable? How about if there was demonstrable evidence that the knew about the specific override? What if they, in an informal capacity, spread information about how to do the override? Here is a parallel situation: same company makes a device, and has locked it down so hard that it's not possible to actually get anything done with it. Nurses and other trained practitioners know that "the usual way of business" is to override some of the safeguards, because, of course, that's the only one anyone ever gets any work done. However, the company disclaims all legal liability in the case that someone does such a modification. Are they allowed to disclaim that? It's complicated.

sirrocco 14y ago
No it's not. At the doctor's office you should be told: "Don't fuck around with the machine. If you need to change the settings, come to me and I'll change it. You also have this hand-book of how to override, but if you do you might die." It's simple. You were warned.
- wickedchicken 14y ago
  > It's simple. You were warned. laws don't work like that
  - chao- 14y ago
    Not to nitpick, because I agree, but I think it's a more accurate to say "litigation doesn't work like that" or "lawyers don't work like that". I was under the impression that it's less a specific law which spells out the liability, but rather a patchwork of regulation and court cases that leave a company with so much uncertainty that the only intelligent choice is to cover your ass any-which-way that you can. My impression (and it is merely that) comes from today's earlier discussion on liability, regulations and compliance: http://news.ycombinator.com/item?id=3889512 http://news.ycombinator.com/item?id=3889512
    - Karunamon 14y ago
      I've been using this quote over and over the past month and this seems a good occasion as any: Lawyers fucking ruin everything. That really sums up pretty much everything wrong with this whole system.
      - JamesLeonis 14y ago
        While I don't necessarily disagree, that's like saying hackers ruin everything. What do lawyers (hackers) do but try to find weaknesses in the current law (program) and exploit them? We haven't figured out how to make perfectly secure systems that are still usable, so why do we expect our laws to be any more robust?
      - micks56 14y ago
        I'm not defending the legal field in its entirety, but your opinion would change if you or your loved one were harmed or killed.
        Karunamon 14y ago
        My loved ones are smart enough to not dick around with life sustaining hardware and software. On a less snarky note, it's absurd that I can't do what I want with something I bought and paid for, whether that be a game console or a breathing mask. It's not the law's job to protect me from myself. And I say lawyers ruin everything because the legal profession has a vested interest in keeping the lay person from being able to understand and use the law effectively. including such idiocy as the OP's scenario.
        Retric 14y ago
        Having actually been in that situation; I respectfully disagree.
        Cushman 14y ago
        > And I say lawyers ruin everything because the legal profession has a vested interest in keeping the lay person from being able to understand and use the law effectively. This may be true, but it is not the reason law is hard for the lay person to understand. Western legal theory has taken "justice" to mean that, given the same facts, and irrespective of the personal opinions of the judge or jury, the court should reach the same decision. Just think about that for a second-- it's not an easy problem. In fact it's a fabulously difficult problem, even before you add the condition that the system should remain fair over time in excess of a human lifespan. The (clearly sub-optimal) solution we've arrived at over the past few thousand years is basically to implement an enormous natural-language virtual machine in which individual agents (lawyers) execute programs (laws) using a very precisely-defined instruction set (legal jargon) which is very similar in structure to English. The reason the laws cannot be written in English is that English has natural ambiguities; this is fine for conversation, but is absolutely unacceptable when human freedom or livelihood is on the line. The obvious downside is that it means that average citizens can actually not learn the law simply by reading the law. To mitigate that, we've constructed an execution context in which a separate lawyer represents each interested party, with a judge acting as a neutral third party present to represent the interests of the law. The jury is asked only to establish the facts; understanding of the law is not required. It's really all pretty clever. Obviously it's not perfect. But, before you criticize the system wholesale, remember that law didn't always work this way. Time was you would just take your grievance before the king, and depending on if he was feeling merciful or surly or liked your family, you would either get what you wanted or not. If you were lucky, there would be rhyme or reason to the king's judgments, and you might be able to predict what he'd say. If not, well... sucks to be you. The complexity of the law, the cost of employing full-time legal scholars for every interaction, and the occasional abuses of the system-- these are the price we pay for fairness. I'm not saying I think lawyers are awesome, but a world without them would be a hell of a lot worse than most people think.
  - sirrocco 14y ago
    Yeap, I'll have to agree with Karunamon bellow : "Lawyers fucking ruin everything."
    - chao- 14y ago
      Except when they save you money? As in the famous "Fuck you. Pay me." video: http://vimeo.com/22053820 http://vimeo.com/22053820 I think it's more nuanced than simply "Lawyers are bad!" The cost of litigation, even when you are likely to win, is so high that it encourages whatever behaviors best avoid litigation, rather than whatever behavior fulfills the original goals of the company/sector/system. Companies and even entire sectors, once well-established, would rather optimize for continual, guaranteed success within a subpar ecosystem than risk losing big just to allow an environment of marginally more satisfied "informed consumers", or what-have-you. Edit: I am rambling about "the system" and such, without offering any solutions, partly because I'm at a loss for one. The best I can muster would be for legal fees and penalties to be in a range that does not warrant selecting such behavior. But that topic is a deep rabbit hole unto itself.
- micks56 14y ago
  Failure to warn is only one component of liability. There is design defect, manufacturing defect, negligence, and so on. "It's complicated" is very accurate.
  - chao- 14y ago
    Though I don't presume to have read his mind, I feel like sirrocco is falling prey to a line of thought that I often do with regards to law. As a programmer, I often want to build some sort of deterministic decision tree regarding matters. The instinct of "Once I have all possible behaviors and paths modeled, an optimal solution will present itself!" is a strong one. And then my friends in law school remind me, sometimes loudly, how the facts or circumstances can vary greatly, and how nothing in life is quite that simple. Alas.
overgard 14y ago
It may be complicated, but that doesn't make it any less outrageous. And it seems like "complicated" problems tend to go away rather quickly when enough people become sufficiently outraged. Or more to the point: we shouldn't be making excuses for people blatantly fucking us over. Let them make their own excuses. Even better, let them get out of the way.
shabble 14y ago
This has happened, to some extent: At the Instituto Oncologico Nacional in Panama, some doctors discovered that they could fudge things to circumvent some limitations in the software that calculated shielding block geometry for radiotherapy patients. In essence, the system could only handle 4 blocks, but it was possible (and sometimes correct) to add more than one block as a single shape. If this happened in the wrong order, however, it would screw up. The staff in charge were brought up on civil (and possibly criminal) negligence charges, from what I recall. Edit: Yep, according to http://www.scielosp.org/scielo.php?pid=S1020-49892006000800014&script=sci_arttext http://www.scielosp.org/scielo.php?pid=S1020-498920060008000... [search for "Legal actions" heading] "On 18 November 2004 it was announced that one of the physicists was acquitted, but the other two were found guilty, sentenced to four years in prison, and barred from practicing their profession for seven years (45). They have initiated an appeal process, but they are very worried, given the precedent established by the Costa Rica case. The [...] Costa Rican physicist [...] was accused of negligence in 30 homicides and 59 radiation injuries. He was also accused of falsifying documents and of using false documents. On 30 July 2001 he was absolved of the latter two charges, but he was found guilty of 14 homicides and 50 radiation injuries. He was sentenced to six years in prison and barred from practicing his profession for five years." [1] https://en.wikipedia.org/wiki/Instituto_Oncologico_Nacional#Accident https://en.wikipedia.org/wiki/Instituto_Oncologico_Nacional#...
- ars 14y ago
  It's quite different to do this for yourself vs for other people.
- Mohnican 14y ago
  This is a good example of the liability problems that come with modifying systems that can be potentially leathal. However, applying them to ALL medical devices under a blanketed set of terms seems absurd. Controlling radiation emitting devices? I can see where this would make sense. Adjusting an APAP or even O2 monitoring system? Not remotely.

by ezyang 14y ago

sirrocco 14y ago

No it's not. At the doctor's office you should be told: "Don't fuck around with the machine. If you need to change the settings, come to me and I'll change it. You also have this hand-book of how to override, but if you do you might die." It's simple. You were warned.

wickedchicken 14y ago
> It's simple. You were warned. laws don't work like that
- chao- 14y ago
  Not to nitpick, because I agree, but I think it's a more accurate to say "litigation doesn't work like that" or "lawyers don't work like that". I was under the impression that it's less a specific law which spells out the liability, but rather a patchwork of regulation and court cases that leave a company with so much uncertainty that the only intelligent choice is to cover your ass any-which-way that you can. My impression (and it is merely that) comes from today's earlier discussion on liability, regulations and compliance: http://news.ycombinator.com/item?id=3889512 http://news.ycombinator.com/item?id=3889512
  - Karunamon 14y ago
    I've been using this quote over and over the past month and this seems a good occasion as any: Lawyers fucking ruin everything. That really sums up pretty much everything wrong with this whole system.
    - JamesLeonis 14y ago
      While I don't necessarily disagree, that's like saying hackers ruin everything. What do lawyers (hackers) do but try to find weaknesses in the current law (program) and exploit them? We haven't figured out how to make perfectly secure systems that are still usable, so why do we expect our laws to be any more robust?
    - micks56 14y ago
      I'm not defending the legal field in its entirety, but your opinion would change if you or your loved one were harmed or killed.
      - Karunamon 14y ago
        My loved ones are smart enough to not dick around with life sustaining hardware and software. On a less snarky note, it's absurd that I can't do what I want with something I bought and paid for, whether that be a game console or a breathing mask. It's not the law's job to protect me from myself. And I say lawyers ruin everything because the legal profession has a vested interest in keeping the lay person from being able to understand and use the law effectively. including such idiocy as the OP's scenario.
        Retric 14y ago
        Having actually been in that situation; I respectfully disagree.
        Cushman 14y ago
        > And I say lawyers ruin everything because the legal profession has a vested interest in keeping the lay person from being able to understand and use the law effectively. This may be true, but it is not the reason law is hard for the lay person to understand. Western legal theory has taken "justice" to mean that, given the same facts, and irrespective of the personal opinions of the judge or jury, the court should reach the same decision. Just think about that for a second-- it's not an easy problem. In fact it's a fabulously difficult problem, even before you add the condition that the system should remain fair over time in excess of a human lifespan. The (clearly sub-optimal) solution we've arrived at over the past few thousand years is basically to implement an enormous natural-language virtual machine in which individual agents (lawyers) execute programs (laws) using a very precisely-defined instruction set (legal jargon) which is very similar in structure to English. The reason the laws cannot be written in English is that English has natural ambiguities; this is fine for conversation, but is absolutely unacceptable when human freedom or livelihood is on the line. The obvious downside is that it means that average citizens can actually not learn the law simply by reading the law. To mitigate that, we've constructed an execution context in which a separate lawyer represents each interested party, with a judge acting as a neutral third party present to represent the interests of the law. The jury is asked only to establish the facts; understanding of the law is not required. It's really all pretty clever. Obviously it's not perfect. But, before you criticize the system wholesale, remember that law didn't always work this way. Time was you would just take your grievance before the king, and depending on if he was feeling merciful or surly or liked your family, you would either get what you wanted or not. If you were lucky, there would be rhyme or reason to the king's judgments, and you might be able to predict what he'd say. If not, well... sucks to be you. The complexity of the law, the cost of employing full-time legal scholars for every interaction, and the occasional abuses of the system-- these are the price we pay for fairness. I'm not saying I think lawyers are awesome, but a world without them would be a hell of a lot worse than most people think.
- sirrocco 14y ago
  Yeap, I'll have to agree with Karunamon bellow : "Lawyers fucking ruin everything."
  - chao- 14y ago
    Except when they save you money? As in the famous "Fuck you. Pay me." video: http://vimeo.com/22053820 http://vimeo.com/22053820 I think it's more nuanced than simply "Lawyers are bad!" The cost of litigation, even when you are likely to win, is so high that it encourages whatever behaviors best avoid litigation, rather than whatever behavior fulfills the original goals of the company/sector/system. Companies and even entire sectors, once well-established, would rather optimize for continual, guaranteed success within a subpar ecosystem than risk losing big just to allow an environment of marginally more satisfied "informed consumers", or what-have-you. Edit: I am rambling about "the system" and such, without offering any solutions, partly because I'm at a loss for one. The best I can muster would be for legal fees and penalties to be in a range that does not warrant selecting such behavior. But that topic is a deep rabbit hole unto itself.
micks56 14y ago
Failure to warn is only one component of liability. There is design defect, manufacturing defect, negligence, and so on. "It's complicated" is very accurate.
- chao- 14y ago
  Though I don't presume to have read his mind, I feel like sirrocco is falling prey to a line of thought that I often do with regards to law. As a programmer, I often want to build some sort of deterministic decision tree regarding matters. The instinct of "Once I have all possible behaviors and paths modeled, an optimal solution will present itself!" is a strong one. And then my friends in law school remind me, sometimes loudly, how the facts or circumstances can vary greatly, and how nothing in life is quite that simple. Alas.

overgard 14y ago

It may be complicated, but that doesn't make it any less outrageous. And it seems like "complicated" problems tend to go away rather quickly when enough people become sufficiently outraged. Or more to the point: we shouldn't be making excuses for people blatantly fucking us over. Let them make their own excuses. Even better, let them get out of the way.

shabble 14y ago

This has happened, to some extent: At the Instituto Oncologico Nacional in Panama, some doctors discovered that they could fudge things to circumvent some limitations in the software that calculated shielding block geometry for radiotherapy patients. In essence, the system could only handle 4 blocks, but it was possible (and sometimes correct) to add more than one block as a single shape. If this happened in the wrong order, however, it would screw up. The staff in charge were brought up on civil (and possibly criminal) negligence charges, from what I recall. Edit: Yep, according to http://www.scielosp.org/scielo.php?pid=S1020-49892006000800014&script=sci_arttext http://www.scielosp.org/scielo.php?pid=S1020-498920060008000... [search for "Legal actions" heading] "On 18 November 2004 it was announced that one of the physicists was acquitted, but the other two were found guilty, sentenced to four years in prison, and barred from practicing their profession for seven years (45). They have initiated an appeal process, but they are very worried, given the precedent established by the Costa Rica case. The [...] Costa Rican physicist [...] was accused of negligence in 30 homicides and 59 radiation injuries. He was also accused of falsifying documents and of using false documents. On 30 July 2001 he was absolved of the latter two charges, but he was found guilty of 14 homicides and 50 radiation injuries. He was sentenced to six years in prison and barred from practicing his profession for five years." [1] https://en.wikipedia.org/wiki/Instituto_Oncologico_Nacional#Accident https://en.wikipedia.org/wiki/Instituto_Oncologico_Nacional#...

ars 14y ago
It's quite different to do this for yourself vs for other people.
Mohnican 14y ago
This is a good example of the liability problems that come with modifying systems that can be potentially leathal. However, applying them to ALL medical devices under a blanketed set of terms seems absurd. Controlling radiation emitting devices? I can see where this would make sense. Adjusting an APAP or even O2 monitoring system? Not remotely.