8 ms·

> I don't know if this is a GDPR violation or not (as someone else asked), but if it is, I hope we start actually seeing action of these sorts of things. Sound

> I don't know if this is a GDPR violation or not (as someone else asked), but if it is, I hope we start actually seeing action of these sorts of things. Sounds like you're suggesting that we criminalize software bugs.

inetknght 8y ago
We criminalize other actions which result in harm to people. Why not software bugs too?
- sdenton4 8y ago
  Well, there's two big differences... Planes have far fewer unknown unknowns than software: the specter bug in Intel chips is a great example of a place where the standard operating procedure was wrong, but no one ever knew it. It wasn't a case of negligence, though it had real world impact. The other big difference is that (for the most part) keeping a passenger plane in the air isn't an adversarial task. Actual breaches are the result of active bad actors, which is completely different from the problems you encounter in designing a plane. So criminal action seems crazy to me, though I can definitely see a great case for changing the incentives around storing user data. Could definitely see a good case for fines (and even an ongoing per-user tax, to make it an up front cost) for storing PII.
  - orhmeh09 8y ago
    I think it’s disputable that “no one ever knew” (or could have guessed) regarding Spectre and speculative execution generally. Intel took a risk for the sake of performance. This did not take long to find: Wang&already, 2006: “Information leakage through covert channels and side channels is becoming a serious problem, especially when these are enhanced by modern processor architecture features. We show how processor architecture features such as simultaneous multithreading, control speculation and shared caches can inadvertently accelerate such covert channels or enable new covert channels and side channels. We first illustrate the reality and severity of this problem by describing concrete attacks. We identify two new covert channels. We show orders of magnitude increases in covert channel capacities. We then present two solutions, Selective Partitioning and the novel Random Permutation Cache (RPCache). The RPCache can thwart most cache-based software side channel attacks, with minimal hardware costs and negligible performance impact.” http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.190.1003&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.190...
lalaithion 8y ago
If a plane crashed, and the company that manufactured the plane was fined because they had an engineering bug, no one would blink an eye.
- canttestthis 8y ago
  The analogy doesn't work. Barring malicious intent or negligence leading to death I cannot imagine (or remember) a situation where the company would be fined for a software bug.
  - vanderZwan 8y ago
    That is more of a failure of the imagination then. It's not like programming would cease to exist if fines were introduced.
  - EpicEng 8y ago
    >or negligence Right. Companies like FB are entrusted with the private information of hundreds of millions of people. There should be investigations as there would be in a plane crash. If negligence is found, there should be appropriate punishment doled out.
  - burkaman 8y ago
    Financial regulators are happy to do it. https://www.networkworld.com/article/2225633/software/knight-capital-fined-a-measly--12m-for-a-software-bug-that-cost--460m.html https://www.networkworld.com/article/2225633/software/knight... https://www.qa-financial.com/articles/cftc-fines-societe-generale-450000-software-bug https://www.qa-financial.com/articles/cftc-fines-societe-gen... https://www.ibtimes.co.uk/citigroup-fined-7m-over-software-bug-that-misreported-trading-data-us-regulator-15-years-1570635 https://www.ibtimes.co.uk/citigroup-fined-7m-over-software-b... https://www.bbc.com/news/business-30125728 https://www.bbc.com/news/business-30125728 https://www.theguardian.com/technology/2013/mar/06/microsoft-fined-browser-error https://www.theguardian.com/technology/2013/mar/06/microsoft...
  - elliekelly 8y ago
    This analogy does work. Boeing's software caused a plane to crash: https://www.reuters.com/article/us-indonesia-crash-boeing/boeing-to-hold-airline-call-on-737-max-systems-after-indonesia-crash-sources-idUSKCN1NP09V https://www.reuters.com/article/us-indonesia-crash-boeing/bo...
ColinDabritz 8y ago
software bugs cause mass harm. We can't ask people to never make mistakes, but we can ask that they have appropriate practices, standards, quality controls, and care. Not taking appropriate measures to mitigate risks that can significantly affect millions of users is willful negligence, and should be a crime.
bluetidepro 8y ago
Yes, I am suggesting that. I don't necessarily think jail time is the right thing, but I do think something like meaningful fines are more than reasonable for major software bugs that cause these kinds of breaches of privacy. It will make larger companies like this be much more careful when money is on the table for them to lose. To me, if we can criminalize something like a major oil spill such as BP/Deepwater Horizon, how is this much different? It's not like they did the oil spill on purpose, but they still need had consequences for those risks that they were taking. Software companies, esp larger ones like Facebook, should have the same kind of consequences for their risks of software bugs that cause these kinds of privacy breaches. Also, as someone else below pointed out to someone else with a similar tone as your phrasing of "criminalize software bugs": "intentionally obscuring the debate. Gross negligence is an entirely different standard than just software bugs."
- newsopt 8y ago
  Just a quick question, do you write software? Do you have a legal or economic background? It seems pretty clear to me that anyone suggesting that software bugs in applications that have no risk of causing physical harm should have criminal liability has no idea what they are talking about and what damage such a law would cause. Case in point look at the quality of medical software today. Hospitals still use windows xp and other completely insecure and outdated software. Because absolutely nobody wants to deal with the nightmare that is HIPAA.
  - elliekelly 8y ago
    HIPAA only carries criminal penalties when someone knowingly discloses covered information - not a software bug. Until the bug is identified at least. For the most part HIPAA is enforced with civil penalties. And your "nightmare" scenario of (civil) liability flowing from programming bugs already exists in the investment world and it hasn't come apart at the seams. Google Axa Rosenberg. A coding error in their trading algorithm went undiscovered for two years. Negligent for sure, but not why the SEC went after them. The problem was they didn't promptly disclose the error to investors and they didn't promptly correct it. Algorithmic trading firms should have mechanisms to catch errors, correct errors, and disclose those errors to investors. And after seeing Axa Rosenberg's $250 million fine and Rosenberg's lifetime ban from the industry guess what they all implemented?
ProAm 8y ago
> Sounds like you're suggesting that we criminalize software bugs. When there is irreparable damage I believe it should be criminalized. You cannot regain privacy after an incident such as this, it is irrevocably taken from you against your will.
- UncleMeat 8y ago
  Suppose there is a bug in the Linux kernel. Some business runs their webservers on Linux. They have user email addresses (PII). Is Linus responsible for breaches? If so, then OSS dies. If not, then how do you intend to prove that their are no vulns in any of your dependencies for the rest of time?
  - AlexandrB 8y ago
    This is silly. If I build my bridge with equations I find on mathoverflow, the forum is not responsible for my bridge collapsing. If you’re using OSS for mission-critical software you must either ensure that it’s fit for purpose or pay someone to do it for you. Nothing in the Linux Kernel documentation suggests that it can/should be used for flying airplanes of securing PII without doing additional due diligence.
  - pixl97 8y ago
    The person storing the data is the one responsible for securing the data. Everyone keeps trying to push data security up the stack, but the company/ individual collecting it is the responsible party.
decebalus1 8y ago
> Sounds like you're suggesting that we criminalize software bugs. When my dad went to college, a very old and bitter professor (this was Civil Engineering, communist Eastern Europe) told the students on the first day in class something along the lines of: "If you know you're stupid or don't give a shit about your work, just go home and save everyone the trouble of dealing with your future fuckups. Mistakes here can cause deaths or losses of huge amounts of money". I believe we've reached the point in which negligence in the software world can cause loss of lives, even when the software is not operating a crane or an airplane (think Grindr leaking account data over http in Saudi Arabia). So you're minimizing the issue by asking if we should criminalize software bugs. We should and currently do criminalize negligence. If bugs are a result of negligence (you know, 'move fast and break things', 'better to ask for forgiveness than for permission') then fines, jailtime and criminal records should be a'coming. This is no longer child-play, this is the new world which runs on software.
pjmorris 8y ago
Hammurabi's code (~1700 BC) includes this about building: Building Code 229. If a builder builds a house for a man and does not make its construction sound, and the house which he has built collapses and causes the death of the owner of the house, the builder shall be put to death. 233. If a builder builds a house for a man and does not make its construction sound, and a wall cracks, that builder shall strengthen that wall at his own expense. Bugs in houses have been criminalized for a very long time. Online data may be less fundamental than safe housing, but housing our data safely becomes proportionally more important as more of modern life depends on it. [0] http://www.wright.edu/~christopher.oldstone-moore/Hamm.htm http://www.wright.edu/~christopher.oldstone-moore/Hamm.htm
- dkrich 8y ago
  But they aren’t. Most home sales in the US follow caveat emptor. If you buy a house from a private seller and then later discover mold in the walls or a crack in the foundation I wish you luck in getting the seller to pay for the repair.
  - perfmode 8y ago
    The parent was establishing precedent.
    - dkrich 8y ago
      And I'm establishing that the precedent cited doesn't exist, not in modern times anyway. If the argument is that home sellers are punished if they don't protect the buyer and so data sellers should be punished if they don't protect the data, that doesn't really hold up because home sellers aren't typically punished.
      - pjmorris 8y ago
        Note that I (OP) am referring to builders, not sellers. With respect to the systems that hold FB's data, I'd argue they are more like builders than sellers.
JKCalhoun 8y ago
Do we have anything more than the company's word for it that it was a bug?
alxlaz 8y ago
But it's not the mere existence of software bugs that is at issue here. Everyone's first attempt at solving a problem in software is going to have bugs. Everyone's last attempt at solving a problem in software is likely to have bugs -- that's why we design systems with safelocks in place. There is risk in any human endeavour that touches upon someone else's life, in every domain. But, for example, only some of the deaths that occur in a hospital are the result of malpractice. That is the type of mistake for which we hold others accountable: not the mere act of providing insufficient care, but the act of providing insufficient care as a result of a dereliction of professional duty or a failure to exercise an ordinary degree of professional skill or learning. IMHO: 1. If this was a novel, or very complicated breach, that Facebook did everything possible to avoid, but avoiding it was beyond the knowledge and skills of their security, engineering and QA teams, who otherwise did their absolute best, then it's at the very least defensible. One could argue that you shouldn't handle private data if you can't do it securely, but risk is inherent to anything, and perhaps worth it under the right circumstances. 2. If this was just "move fast and break things" policy, then a big fine is in order, and if no insurance is in place, whoever approved it should get to pay it out of their own pocket. This is the equivalent of a civil engineering company designing a collapsing bridge because everyone showed up at work hungover, or skipped safety calculations because they just take too damn long and time to market is critical. If you think gee, this was just a bunch of photos, man, it's not like a bridge collapsed, how certain are you they didn't end up traded on the black market, or used for blackmail? Bet-your-company's-profits certain they weren't? 3. If this was deliberate policy -- not just accident, but a conscious business decision that was then reverted and declared a breach -- then whoever came up with it and/or approved it should be facing jail time. Edit: also, it pisses me off that people are trying to decide how responsible we should about what we do based on other fields. They don't fine companies that write crashing firwmares for planes or cars or they fine it X amount, clearly we're only doing computer stuff so we should be fined less, no? What the hell? First, they are fined (see, for instance, Toyota, who were fined 1.2B for their infamous acceleration firmware bug). And second, even if they weren't, we shouldn't be aspiring to do the worst thing that's still acceptable! We should be striving for better than anything else, not for well, at least we're not worse than civil engineers...
dudul 8y ago
When we're entering an era where everything will involve software, yes, bad software and careless bugs should be severely punished.
village-idiot 8y ago
The alternative is to allow the least competent actors continue to lose user data without any consequences.
pasxizeis 8y ago
Sounds like you're suggesting we take no action when law violations happen.
joering2 8y ago
Why shouldnt we? If you are proven in court of law you knew about certain bugs or you underspend on engineering while your public outreach grew expenantially, how come you shouldnt be liable? Anywhere else its a plain case of malpractice whether its law or medicine, etc.
fiblye 8y ago
Facebook sure does have a lot of "bugs" that grant them access to things they shouldn't have. Things that allow them to profit immensely. It happens with such regularity that I'm amazed anyone here would be kind enough to accept their fake apologies for clearly malicious actions.
rhizome 8y ago
I think criminalizing commercial data leaks is a more workable idea.

inetknght 8y ago

We criminalize other actions which result in harm to people. Why not software bugs too?

sdenton4 8y ago
Well, there's two big differences... Planes have far fewer unknown unknowns than software: the specter bug in Intel chips is a great example of a place where the standard operating procedure was wrong, but no one ever knew it. It wasn't a case of negligence, though it had real world impact. The other big difference is that (for the most part) keeping a passenger plane in the air isn't an adversarial task. Actual breaches are the result of active bad actors, which is completely different from the problems you encounter in designing a plane. So criminal action seems crazy to me, though I can definitely see a great case for changing the incentives around storing user data. Could definitely see a good case for fines (and even an ongoing per-user tax, to make it an up front cost) for storing PII.
- orhmeh09 8y ago
  I think it’s disputable that “no one ever knew” (or could have guessed) regarding Spectre and speculative execution generally. Intel took a risk for the sake of performance. This did not take long to find: Wang&already, 2006: “Information leakage through covert channels and side channels is becoming a serious problem, especially when these are enhanced by modern processor architecture features. We show how processor architecture features such as simultaneous multithreading, control speculation and shared caches can inadvertently accelerate such covert channels or enable new covert channels and side channels. We first illustrate the reality and severity of this problem by describing concrete attacks. We identify two new covert channels. We show orders of magnitude increases in covert channel capacities. We then present two solutions, Selective Partitioning and the novel Random Permutation Cache (RPCache). The RPCache can thwart most cache-based software side channel attacks, with minimal hardware costs and negligible performance impact.” http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.190.1003&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.190...

lalaithion 8y ago

If a plane crashed, and the company that manufactured the plane was fined because they had an engineering bug, no one would blink an eye.

canttestthis 8y ago
The analogy doesn't work. Barring malicious intent or negligence leading to death I cannot imagine (or remember) a situation where the company would be fined for a software bug.
- vanderZwan 8y ago
  That is more of a failure of the imagination then. It's not like programming would cease to exist if fines were introduced.
- EpicEng 8y ago
  >or negligence Right. Companies like FB are entrusted with the private information of hundreds of millions of people. There should be investigations as there would be in a plane crash. If negligence is found, there should be appropriate punishment doled out.
- burkaman 8y ago
  Financial regulators are happy to do it. https://www.networkworld.com/article/2225633/software/knight-capital-fined-a-measly--12m-for-a-software-bug-that-cost--460m.html https://www.networkworld.com/article/2225633/software/knight... https://www.qa-financial.com/articles/cftc-fines-societe-generale-450000-software-bug https://www.qa-financial.com/articles/cftc-fines-societe-gen... https://www.ibtimes.co.uk/citigroup-fined-7m-over-software-bug-that-misreported-trading-data-us-regulator-15-years-1570635 https://www.ibtimes.co.uk/citigroup-fined-7m-over-software-b... https://www.bbc.com/news/business-30125728 https://www.bbc.com/news/business-30125728 https://www.theguardian.com/technology/2013/mar/06/microsoft-fined-browser-error https://www.theguardian.com/technology/2013/mar/06/microsoft...
- elliekelly 8y ago
  This analogy does work. Boeing's software caused a plane to crash: https://www.reuters.com/article/us-indonesia-crash-boeing/boeing-to-hold-airline-call-on-737-max-systems-after-indonesia-crash-sources-idUSKCN1NP09V https://www.reuters.com/article/us-indonesia-crash-boeing/bo...

ColinDabritz 8y ago

software bugs cause mass harm. We can't ask people to never make mistakes, but we can ask that they have appropriate practices, standards, quality controls, and care. Not taking appropriate measures to mitigate risks that can significantly affect millions of users is willful negligence, and should be a crime.

bluetidepro 8y ago

Yes, I am suggesting that. I don't necessarily think jail time is the right thing, but I do think something like meaningful fines are more than reasonable for major software bugs that cause these kinds of breaches of privacy. It will make larger companies like this be much more careful when money is on the table for them to lose. To me, if we can criminalize something like a major oil spill such as BP/Deepwater Horizon, how is this much different? It's not like they did the oil spill on purpose, but they still need had consequences for those risks that they were taking. Software companies, esp larger ones like Facebook, should have the same kind of consequences for their risks of software bugs that cause these kinds of privacy breaches. Also, as someone else below pointed out to someone else with a similar tone as your phrasing of "criminalize software bugs": "intentionally obscuring the debate. Gross negligence is an entirely different standard than just software bugs."

newsopt 8y ago
Just a quick question, do you write software? Do you have a legal or economic background? It seems pretty clear to me that anyone suggesting that software bugs in applications that have no risk of causing physical harm should have criminal liability has no idea what they are talking about and what damage such a law would cause. Case in point look at the quality of medical software today. Hospitals still use windows xp and other completely insecure and outdated software. Because absolutely nobody wants to deal with the nightmare that is HIPAA.
- elliekelly 8y ago
  HIPAA only carries criminal penalties when someone knowingly discloses covered information - not a software bug. Until the bug is identified at least. For the most part HIPAA is enforced with civil penalties. And your "nightmare" scenario of (civil) liability flowing from programming bugs already exists in the investment world and it hasn't come apart at the seams. Google Axa Rosenberg. A coding error in their trading algorithm went undiscovered for two years. Negligent for sure, but not why the SEC went after them. The problem was they didn't promptly disclose the error to investors and they didn't promptly correct it. Algorithmic trading firms should have mechanisms to catch errors, correct errors, and disclose those errors to investors. And after seeing Axa Rosenberg's $250 million fine and Rosenberg's lifetime ban from the industry guess what they all implemented?

ProAm 8y ago

> Sounds like you're suggesting that we criminalize software bugs. When there is irreparable damage I believe it should be criminalized. You cannot regain privacy after an incident such as this, it is irrevocably taken from you against your will.

UncleMeat 8y ago
Suppose there is a bug in the Linux kernel. Some business runs their webservers on Linux. They have user email addresses (PII). Is Linus responsible for breaches? If so, then OSS dies. If not, then how do you intend to prove that their are no vulns in any of your dependencies for the rest of time?
- AlexandrB 8y ago
  This is silly. If I build my bridge with equations I find on mathoverflow, the forum is not responsible for my bridge collapsing. If you’re using OSS for mission-critical software you must either ensure that it’s fit for purpose or pay someone to do it for you. Nothing in the Linux Kernel documentation suggests that it can/should be used for flying airplanes of securing PII without doing additional due diligence.
- pixl97 8y ago
  The person storing the data is the one responsible for securing the data. Everyone keeps trying to push data security up the stack, but the company/ individual collecting it is the responsible party.

decebalus1 8y ago

> Sounds like you're suggesting that we criminalize software bugs. When my dad went to college, a very old and bitter professor (this was Civil Engineering, communist Eastern Europe) told the students on the first day in class something along the lines of: "If you know you're stupid or don't give a shit about your work, just go home and save everyone the trouble of dealing with your future fuckups. Mistakes here can cause deaths or losses of huge amounts of money". I believe we've reached the point in which negligence in the software world can cause loss of lives, even when the software is not operating a crane or an airplane (think Grindr leaking account data over http in Saudi Arabia). So you're minimizing the issue by asking if we should criminalize software bugs. We should and currently do criminalize negligence. If bugs are a result of negligence (you know, 'move fast and break things', 'better to ask for forgiveness than for permission') then fines, jailtime and criminal records should be a'coming. This is no longer child-play, this is the new world which runs on software.

pjmorris 8y ago

Hammurabi's code (~1700 BC) includes this about building: Building Code 229. If a builder builds a house for a man and does not make its construction sound, and the house which he has built collapses and causes the death of the owner of the house, the builder shall be put to death. 233. If a builder builds a house for a man and does not make its construction sound, and a wall cracks, that builder shall strengthen that wall at his own expense. Bugs in houses have been criminalized for a very long time. Online data may be less fundamental than safe housing, but housing our data safely becomes proportionally more important as more of modern life depends on it. [0] http://www.wright.edu/~christopher.oldstone-moore/Hamm.htm http://www.wright.edu/~christopher.oldstone-moore/Hamm.htm

dkrich 8y ago
But they aren’t. Most home sales in the US follow caveat emptor. If you buy a house from a private seller and then later discover mold in the walls or a crack in the foundation I wish you luck in getting the seller to pay for the repair.
- perfmode 8y ago
  The parent was establishing precedent.
  - dkrich 8y ago
    And I'm establishing that the precedent cited doesn't exist, not in modern times anyway. If the argument is that home sellers are punished if they don't protect the buyer and so data sellers should be punished if they don't protect the data, that doesn't really hold up because home sellers aren't typically punished.
    - pjmorris 8y ago
      Note that I (OP) am referring to builders, not sellers. With respect to the systems that hold FB's data, I'd argue they are more like builders than sellers.

JKCalhoun 8y ago

Do we have anything more than the company's word for it that it was a bug?

alxlaz 8y ago

But it's not the mere existence of software bugs that is at issue here. Everyone's first attempt at solving a problem in software is going to have bugs. Everyone's last attempt at solving a problem in software is likely to have bugs -- that's why we design systems with safelocks in place. There is risk in any human endeavour that touches upon someone else's life, in every domain. But, for example, only some of the deaths that occur in a hospital are the result of malpractice. That is the type of mistake for which we hold others accountable: not the mere act of providing insufficient care, but the act of providing insufficient care as a result of a dereliction of professional duty or a failure to exercise an ordinary degree of professional skill or learning. IMHO: 1. If this was a novel, or very complicated breach, that Facebook did everything possible to avoid, but avoiding it was beyond the knowledge and skills of their security, engineering and QA teams, who otherwise did their absolute best, then it's at the very least defensible. One could argue that you shouldn't handle private data if you can't do it securely, but risk is inherent to anything, and perhaps worth it under the right circumstances. 2. If this was just "move fast and break things" policy, then a big fine is in order, and if no insurance is in place, whoever approved it should get to pay it out of their own pocket. This is the equivalent of a civil engineering company designing a collapsing bridge because everyone showed up at work hungover, or skipped safety calculations because they just take too damn long and time to market is critical. If you think gee, this was just a bunch of photos, man, it's not like a bridge collapsed, how certain are you they didn't end up traded on the black market, or used for blackmail? Bet-your-company's-profits certain they weren't? 3. If this was deliberate policy -- not just accident, but a conscious business decision that was then reverted and declared a breach -- then whoever came up with it and/or approved it should be facing jail time. Edit: also, it pisses me off that people are trying to decide how responsible we should about what we do based on other fields. They don't fine companies that write crashing firwmares for planes or cars or they fine it X amount, clearly we're only doing computer stuff so we should be fined less, no? What the hell? First, they are fined (see, for instance, Toyota, who were fined 1.2B for their infamous acceleration firmware bug). And second, even if they weren't, we shouldn't be aspiring to do the worst thing that's still acceptable! We should be striving for better than anything else, not for well, at least we're not worse than civil engineers...

dudul 8y ago

When we're entering an era where everything will involve software, yes, bad software and careless bugs should be severely punished.

village-idiot 8y ago

The alternative is to allow the least competent actors continue to lose user data without any consequences.

pasxizeis 8y ago

Sounds like you're suggesting we take no action when law violations happen.

joering2 8y ago

Why shouldnt we? If you are proven in court of law you knew about certain bugs or you underspend on engineering while your public outreach grew expenantially, how come you shouldnt be liable? Anywhere else its a plain case of malpractice whether its law or medicine, etc.

fiblye 8y ago

Facebook sure does have a lot of "bugs" that grant them access to things they shouldn't have. Things that allow them to profit immensely. It happens with such regularity that I'm amazed anyone here would be kind enough to accept their fake apologies for clearly malicious actions.

rhizome 8y ago

I think criminalizing commercial data leaks is a more workable idea.