5 ms·

> The device did not audibly advise that it was preparing to send the recording, something it’s programmed to do. Apparently it's not programmed to do that. Un

> The device did not audibly advise that it was preparing to send the recording, something it’s programmed to do. Apparently it's not programmed to do that. Unless this was a hardware glitch or cosmic-ray event.

cortesoft 8y ago
As developers, we use 'programmed' in that manner all the time, to mean what we were trying to get it to do rather than what it actually does.
- DoofusOfDeath 8y ago
  I don't. And in this particular scenario, it's an important distinction to make because there's a risk of Amazon using equivocation to deflect responsibility.
  - cortesoft 8y ago
    If this was intentional, then yes they shouldn't be able to deflect responsibility. However, if it was just a bug, I think it is a bit unfair to vilify either the company or the developers. Bugs happen, and hopefully they can learn what caused this and prevent this class of bugs in the future.
    - toss1 8y ago
      And this is where "software" diverges from "engineering". Bugs happen in architecture, aircraft, etc. too. the difference is that the actual engineers are paid to have a precautionary approach and spend significant resources to actively prevent bugs from making it into the final product. In contrast, software is often written to "ship first", be "agile", and "move fast and break things". Yet when it causes problems, they just say "bugs happen", and "it is unfair to vilify them". Features are not better than reliable security. And yes, negligence is less bad than malice, but it is still damaging and developers and managers need to be held to account. If your manager is pressing you to do unsafe crap in too big a hurry, it's your responsibility to push back, and if unsuccessful, leave for saner pastures and make it more difficult for that management to proceed.
      - cortesoft 8y ago
        > Features are not better than reliable security. Well, yes and no. Bugs, security, reliability, etc are all important things to consider, but they can't be the only focus. Security doesn't matter if the thing you are creating has no features; it would be the same as if it didn't exist at all. Instead, we must manage risk; the risk of bugs, the risk of security vulnerabilities, etc. Nothing we do is risk free. Even walking across the bathroom floor has SOME risk; we might slip and fall. Does that mean we should just stay in bed all day to avoid any risk? No, we measure risk by factoring the chance of the bad thing happening and the consequence of the bad thing happening. We then determine how much effort we should spend on that risk, since there are infinite risks and only a finite amount of effort we can expend. If the consequence of a risk is death, then we should absolutely put a lot of effort into minimizing that risk. If the consequence of a risk is that a private conversation is sent to a contact, we should definitely put a lot of effort into minimizing that risk, but probably not quite as much as you would into something that has the consequence of death. Even when the risk is death, however, we don't put infinite effort into avoiding it. We choose to cross the street, even when we know there is a risk of death when we do it. We drive cars that have chances of mechanical failures that could cause our death, but we don't bring the car to the shop every day to check for failures. Things are not so black and white as to say "security is always the most important thing"
        toss1 8y ago
        I did not say >"security is always the most important thing". I did not suggest that anyone should develop such that >" the thing you are creating has no features". I certainly never suggested that no one get out of bed because they might slip in the bath. These are strawman arguments. You do not need to lecture me about risk. I've had a career in international downhill ski racing, have won auto racing championships, and enjoyed lots of technical rock climbing, all of which require a high degree of risk assessment, both in extended preparation phases and at split-second time scales. I've also run risk analysis for UAV flight systems I understand well the difference between smart-crazy and dumb-crazy, and where the pseudo-mathematical risk models like yours break down. Your 'analysis' to "...measure risk by factoring the chance..." would have fit right in at the meetings where Ford decided to just go ahead with the design of the Pinto/Bobcat because the lawsuits would cost less than the fix -- they wound up killing dozens of people. Your 'analysis' would have fit right in where the trading algorithms were being designed, which worked fantastically profitably, until they didn't and ended up crashing the global economy in 2007-8. You cannot simply multiply the cost of the consequences by the expected probability and get an allocation of resources. That is what you do to see if the lottery jackpot is big enough for you to want to buy a $2 ticket this week. You must instead 1) fully examine the system for potential critical failure points/modes and then 2) allocate WHATEVER resources are necessary to account for preventing those critical failures, then implement those remedies along with the features. These preventative measures may involve installing redundant systems around the critical points, redesigning the points so they fail in a safe mode (e.g., fail to send the data vs sending it off, shut down vs, explode, etc.), adding check procedures around the potential critical failure, etc. Note that NONE of these measures involve not implementing the feature. They involve 1) checking for critical failure modes, 2) allocating R&D to develop preventative & fail-safe measures, 3) implementing the measures, 4) testing, and 5) field monitoring. This is what you do if you are serious about risk.
- drb91 8y ago
  Really? I would find that pretty confusing as a developer--a computer famously only does what it is programmed to do.
- SilasX 8y ago
  Right, or "the current source code says not to do that, but some discrepant, previous, buggy build of the machine code made it into the product you actually bought."
- squeaky-clean 8y ago
  I think there's an implicit "something it’s programmed to do under normal circumstances" in there.
lucisferre 8y ago
Interesting. I had heard that cosmic rays are how "real" programmers do it. Perhaps there is something to that after all. https://xkcd.com/378/ https://xkcd.com/378/
jordigh 8y ago
On the topic of cosmic rays... it's more likely than most people think, at least for 2007 hardware. https://lwn.net/Articles/219983/ https://lwn.net/Articles/219983/
- setquk 8y ago
  It’s even worse. Sometimes the packages that the DRAM ICs are encapsulated in have minor radioactive elements in and alpha particles emitted can flip bits. Interesting paper: https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinaburg_Bitsquatting_WP.pdf https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu...
grkvlt 8y ago
Or just a bug.
r00fus 8y ago
cosmic ray - is that an exploit name or the secret government program that mandates Amazon provide this information on request?
- mykull 8y ago
  If by chance this is not a joke, you should look into ECC memory and why it exists.

cortesoft 8y ago

As developers, we use 'programmed' in that manner all the time, to mean what we were trying to get it to do rather than what it actually does.

DoofusOfDeath 8y ago
I don't. And in this particular scenario, it's an important distinction to make because there's a risk of Amazon using equivocation to deflect responsibility.
- cortesoft 8y ago
  If this was intentional, then yes they shouldn't be able to deflect responsibility. However, if it was just a bug, I think it is a bit unfair to vilify either the company or the developers. Bugs happen, and hopefully they can learn what caused this and prevent this class of bugs in the future.
  - toss1 8y ago
    And this is where "software" diverges from "engineering". Bugs happen in architecture, aircraft, etc. too. the difference is that the actual engineers are paid to have a precautionary approach and spend significant resources to actively prevent bugs from making it into the final product. In contrast, software is often written to "ship first", be "agile", and "move fast and break things". Yet when it causes problems, they just say "bugs happen", and "it is unfair to vilify them". Features are not better than reliable security. And yes, negligence is less bad than malice, but it is still damaging and developers and managers need to be held to account. If your manager is pressing you to do unsafe crap in too big a hurry, it's your responsibility to push back, and if unsuccessful, leave for saner pastures and make it more difficult for that management to proceed.
    - cortesoft 8y ago
      > Features are not better than reliable security. Well, yes and no. Bugs, security, reliability, etc are all important things to consider, but they can't be the only focus. Security doesn't matter if the thing you are creating has no features; it would be the same as if it didn't exist at all. Instead, we must manage risk; the risk of bugs, the risk of security vulnerabilities, etc. Nothing we do is risk free. Even walking across the bathroom floor has SOME risk; we might slip and fall. Does that mean we should just stay in bed all day to avoid any risk? No, we measure risk by factoring the chance of the bad thing happening and the consequence of the bad thing happening. We then determine how much effort we should spend on that risk, since there are infinite risks and only a finite amount of effort we can expend. If the consequence of a risk is death, then we should absolutely put a lot of effort into minimizing that risk. If the consequence of a risk is that a private conversation is sent to a contact, we should definitely put a lot of effort into minimizing that risk, but probably not quite as much as you would into something that has the consequence of death. Even when the risk is death, however, we don't put infinite effort into avoiding it. We choose to cross the street, even when we know there is a risk of death when we do it. We drive cars that have chances of mechanical failures that could cause our death, but we don't bring the car to the shop every day to check for failures. Things are not so black and white as to say "security is always the most important thing"
      - toss1 8y ago
        I did not say >"security is always the most important thing". I did not suggest that anyone should develop such that >" the thing you are creating has no features". I certainly never suggested that no one get out of bed because they might slip in the bath. These are strawman arguments. You do not need to lecture me about risk. I've had a career in international downhill ski racing, have won auto racing championships, and enjoyed lots of technical rock climbing, all of which require a high degree of risk assessment, both in extended preparation phases and at split-second time scales. I've also run risk analysis for UAV flight systems I understand well the difference between smart-crazy and dumb-crazy, and where the pseudo-mathematical risk models like yours break down. Your 'analysis' to "...measure risk by factoring the chance..." would have fit right in at the meetings where Ford decided to just go ahead with the design of the Pinto/Bobcat because the lawsuits would cost less than the fix -- they wound up killing dozens of people. Your 'analysis' would have fit right in where the trading algorithms were being designed, which worked fantastically profitably, until they didn't and ended up crashing the global economy in 2007-8. You cannot simply multiply the cost of the consequences by the expected probability and get an allocation of resources. That is what you do to see if the lottery jackpot is big enough for you to want to buy a $2 ticket this week. You must instead 1) fully examine the system for potential critical failure points/modes and then 2) allocate WHATEVER resources are necessary to account for preventing those critical failures, then implement those remedies along with the features. These preventative measures may involve installing redundant systems around the critical points, redesigning the points so they fail in a safe mode (e.g., fail to send the data vs sending it off, shut down vs, explode, etc.), adding check procedures around the potential critical failure, etc. Note that NONE of these measures involve not implementing the feature. They involve 1) checking for critical failure modes, 2) allocating R&D to develop preventative & fail-safe measures, 3) implementing the measures, 4) testing, and 5) field monitoring. This is what you do if you are serious about risk.
drb91 8y ago
Really? I would find that pretty confusing as a developer--a computer famously only does what it is programmed to do.
SilasX 8y ago
Right, or "the current source code says not to do that, but some discrepant, previous, buggy build of the machine code made it into the product you actually bought."
squeaky-clean 8y ago
I think there's an implicit "something it’s programmed to do under normal circumstances" in there.

lucisferre 8y ago

Interesting. I had heard that cosmic rays are how "real" programmers do it. Perhaps there is something to that after all. https://xkcd.com/378/ https://xkcd.com/378/

jordigh 8y ago

On the topic of cosmic rays... it's more likely than most people think, at least for 2007 hardware. https://lwn.net/Articles/219983/ https://lwn.net/Articles/219983/

setquk 8y ago
It’s even worse. Sometimes the packages that the DRAM ICs are encapsulated in have minor radioactive elements in and alpha particles emitted can flip bits. Interesting paper: https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinaburg_Bitsquatting_WP.pdf https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu...

grkvlt 8y ago

Or just a bug.

r00fus 8y ago

cosmic ray - is that an exploit name or the secret government program that mandates Amazon provide this information on request?

mykull 8y ago
If by chance this is not a joke, you should look into ECC memory and why it exists.