7 ms·

Summary of vague technical details (which may be all we hear about this): > an Alexa engineer investigated ... they said 'our engineers went through your logs,

by asn0 8y ago

Summary of vague technical details (which may be all we hear about this): > an Alexa engineer investigated ... they said 'our engineers went through your logs, and they saw exactly what you told us, they saw exactly what you said happened, and we're sorry.' He apologized like 15 times in a matter of 30 minutes and he said we really appreciate you bringing this to our attention, this is something we need to fix!" > the engineer did not provide specifics about why it happened, or if it's a widespread issue. > "He told us that the device just guessed what we were saying" The device did not audibly advise that it was preparing to send the recording, something it’s programmed to do.

DoofusOfDeath 8y ago
> The device did not audibly advise that it was preparing to send the recording, something it’s programmed to do. Apparently it's not programmed to do that. Unless this was a hardware glitch or cosmic-ray event.
- cortesoft 8y ago
  As developers, we use 'programmed' in that manner all the time, to mean what we were trying to get it to do rather than what it actually does.
  - DoofusOfDeath 8y ago
    I don't. And in this particular scenario, it's an important distinction to make because there's a risk of Amazon using equivocation to deflect responsibility.
    - cortesoft 8y ago
      If this was intentional, then yes they shouldn't be able to deflect responsibility. However, if it was just a bug, I think it is a bit unfair to vilify either the company or the developers. Bugs happen, and hopefully they can learn what caused this and prevent this class of bugs in the future.
      - toss1 8y ago
        And this is where "software" diverges from "engineering". Bugs happen in architecture, aircraft, etc. too. the difference is that the actual engineers are paid to have a precautionary approach and spend significant resources to actively prevent bugs from making it into the final product. In contrast, software is often written to "ship first", be "agile", and "move fast and break things". Yet when it causes problems, they just say "bugs happen", and "it is unfair to vilify them". Features are not better than reliable security. And yes, negligence is less bad than malice, but it is still damaging and developers and managers need to be held to account. If your manager is pressing you to do unsafe crap in too big a hurry, it's your responsibility to push back, and if unsuccessful, leave for saner pastures and make it more difficult for that management to proceed.
        cortesoft 8y ago
        > Features are not better than reliable security. Well, yes and no. Bugs, security, reliability, etc are all important things to consider, but they can't be the only focus. Security doesn't matter if the thing you are creating has no features; it would be the same as if it didn't exist at all. Instead, we must manage risk; the risk of bugs, the risk of security vulnerabilities, etc. Nothing we do is risk free. Even walking across the bathroom floor has SOME risk; we might slip and fall. Does that mean we should just stay in bed all day to avoid any risk? No, we measure risk by factoring the chance of the bad thing happening and the consequence of the bad thing happening. We then determine how much effort we should spend on that risk, since there are infinite risks and only a finite amount of effort we can expend. If the consequence of a risk is death, then we should absolutely put a lot of effort into minimizing that risk. If the consequence of a risk is that a private conversation is sent to a contact, we should definitely put a lot of effort into minimizing that risk, but probably not quite as much as you would into something that has the consequence of death. Even when the risk is death, however, we don't put infinite effort into avoiding it. We choose to cross the street, even when we know there is a risk of death when we do it. We drive cars that have chances of mechanical failures that could cause our death, but we don't bring the car to the shop every day to check for failures. Things are not so black and white as to say "security is always the most important thing"
  - drb91 8y ago
    Really? I would find that pretty confusing as a developer--a computer famously only does what it is programmed to do.
  - SilasX 8y ago
    Right, or "the current source code says not to do that, but some discrepant, previous, buggy build of the machine code made it into the product you actually bought."
  - squeaky-clean 8y ago
    I think there's an implicit "something it’s programmed to do under normal circumstances" in there.
- lucisferre 8y ago
  Interesting. I had heard that cosmic rays are how "real" programmers do it. Perhaps there is something to that after all. https://xkcd.com/378/ https://xkcd.com/378/
- jordigh 8y ago
  On the topic of cosmic rays... it's more likely than most people think, at least for 2007 hardware. https://lwn.net/Articles/219983/ https://lwn.net/Articles/219983/
  - setquk 8y ago
    It’s even worse. Sometimes the packages that the DRAM ICs are encapsulated in have minor radioactive elements in and alpha particles emitted can flip bits. Interesting paper: https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinaburg_Bitsquatting_WP.pdf https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu...
- grkvlt 8y ago
  Or just a bug.
- r00fus 8y ago
  cosmic ray - is that an exploit name or the secret government program that mandates Amazon provide this information on request?
  - mykull 8y ago
    If by chance this is not a joke, you should look into ECC memory and why it exists.
btgeekboy 8y ago
It sounds like she had the voice-activated equivalent of butt-dialing.
- ajross 8y ago
  Yeah, pretty much. It's likely that the device heard something like "Alexa, send a note to Enrique" and then recorded and sent the subsequent note to the matching contact that it found. The microfailure is that it either failed to respond to the user in a way that was clear, or did it in a way that the user didn't notice (I dunno, volume all the way down? Output to a bluetooth speaker or headphone jack that wasn't audible?). And that can be fixed. The macrofailure, though, is that you have a device just guessing at what you want and doing it. That can't be fixed without going back to manually constructed (or at least manually triggered) notifications. Which for "send a note" might be an obvious fix, sure, but there is a lot of gray area in what constitutes "privacy" and Alexa would be pretty useless if you had to affirmatively OK everything it did.
  - sqlacid 8y ago
    >Output to a bluetooth speaker or headphone jack that wasn't audible?). And that can be fixed. Not sure how to fix that, if you use the aux out it has no way of knowing what happens on the other end; a confirmation prompt that isn't heard could still pick up an errant "Ok" response. Perhaps require a PIN like the shopping confirmation that would make it much much more unlikely? Or better yet, disable messaging as an option.
SilasX 8y ago
I'm just thankful that the response is more satisfying than the typical, "nope, couldn't have happened, can't reproduce".
- tfandango 8y ago
  I thought the response was pretty disturbing. They have logs of everything you say to Alexa? I guess I assumed the private conversations were not kept beyond some sort of metadata. No thanks.
  - ahumm 8y ago
    That response could easily mean that they saw that there was an interaction where a message was recorded and sent without the device getting confirmation beforehand which matched what was reported to them, not necessarily that they kept the content of the conversation and could play it back. That's also a paraphrased conversation reported by a possibly non-technical person and likely passed through a customer service representative, so I'd take any technical details gleaned from it with a grain of salt.
PostOnce 8y ago
Fortunately we recorded everything you've ever said, so we'll just check these Alexa logs and figure it out...

by asn0 8y ago

DoofusOfDeath 8y ago

> The device did not audibly advise that it was preparing to send the recording, something it’s programmed to do. Apparently it's not programmed to do that. Unless this was a hardware glitch or cosmic-ray event.

cortesoft 8y ago
As developers, we use 'programmed' in that manner all the time, to mean what we were trying to get it to do rather than what it actually does.
- DoofusOfDeath 8y ago
  I don't. And in this particular scenario, it's an important distinction to make because there's a risk of Amazon using equivocation to deflect responsibility.
  - cortesoft 8y ago
    If this was intentional, then yes they shouldn't be able to deflect responsibility. However, if it was just a bug, I think it is a bit unfair to vilify either the company or the developers. Bugs happen, and hopefully they can learn what caused this and prevent this class of bugs in the future.
    - toss1 8y ago
      And this is where "software" diverges from "engineering". Bugs happen in architecture, aircraft, etc. too. the difference is that the actual engineers are paid to have a precautionary approach and spend significant resources to actively prevent bugs from making it into the final product. In contrast, software is often written to "ship first", be "agile", and "move fast and break things". Yet when it causes problems, they just say "bugs happen", and "it is unfair to vilify them". Features are not better than reliable security. And yes, negligence is less bad than malice, but it is still damaging and developers and managers need to be held to account. If your manager is pressing you to do unsafe crap in too big a hurry, it's your responsibility to push back, and if unsuccessful, leave for saner pastures and make it more difficult for that management to proceed.
      - cortesoft 8y ago
        > Features are not better than reliable security. Well, yes and no. Bugs, security, reliability, etc are all important things to consider, but they can't be the only focus. Security doesn't matter if the thing you are creating has no features; it would be the same as if it didn't exist at all. Instead, we must manage risk; the risk of bugs, the risk of security vulnerabilities, etc. Nothing we do is risk free. Even walking across the bathroom floor has SOME risk; we might slip and fall. Does that mean we should just stay in bed all day to avoid any risk? No, we measure risk by factoring the chance of the bad thing happening and the consequence of the bad thing happening. We then determine how much effort we should spend on that risk, since there are infinite risks and only a finite amount of effort we can expend. If the consequence of a risk is death, then we should absolutely put a lot of effort into minimizing that risk. If the consequence of a risk is that a private conversation is sent to a contact, we should definitely put a lot of effort into minimizing that risk, but probably not quite as much as you would into something that has the consequence of death. Even when the risk is death, however, we don't put infinite effort into avoiding it. We choose to cross the street, even when we know there is a risk of death when we do it. We drive cars that have chances of mechanical failures that could cause our death, but we don't bring the car to the shop every day to check for failures. Things are not so black and white as to say "security is always the most important thing"
- drb91 8y ago
  Really? I would find that pretty confusing as a developer--a computer famously only does what it is programmed to do.
- SilasX 8y ago
  Right, or "the current source code says not to do that, but some discrepant, previous, buggy build of the machine code made it into the product you actually bought."
- squeaky-clean 8y ago
  I think there's an implicit "something it’s programmed to do under normal circumstances" in there.
lucisferre 8y ago
Interesting. I had heard that cosmic rays are how "real" programmers do it. Perhaps there is something to that after all. https://xkcd.com/378/ https://xkcd.com/378/
jordigh 8y ago
On the topic of cosmic rays... it's more likely than most people think, at least for 2007 hardware. https://lwn.net/Articles/219983/ https://lwn.net/Articles/219983/
- setquk 8y ago
  It’s even worse. Sometimes the packages that the DRAM ICs are encapsulated in have minor radioactive elements in and alpha particles emitted can flip bits. Interesting paper: https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinaburg_Bitsquatting_WP.pdf https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu...
grkvlt 8y ago
Or just a bug.
r00fus 8y ago
cosmic ray - is that an exploit name or the secret government program that mandates Amazon provide this information on request?
- mykull 8y ago
  If by chance this is not a joke, you should look into ECC memory and why it exists.

btgeekboy 8y ago

It sounds like she had the voice-activated equivalent of butt-dialing.

ajross 8y ago
Yeah, pretty much. It's likely that the device heard something like "Alexa, send a note to Enrique" and then recorded and sent the subsequent note to the matching contact that it found. The microfailure is that it either failed to respond to the user in a way that was clear, or did it in a way that the user didn't notice (I dunno, volume all the way down? Output to a bluetooth speaker or headphone jack that wasn't audible?). And that can be fixed. The macrofailure, though, is that you have a device just guessing at what you want and doing it. That can't be fixed without going back to manually constructed (or at least manually triggered) notifications. Which for "send a note" might be an obvious fix, sure, but there is a lot of gray area in what constitutes "privacy" and Alexa would be pretty useless if you had to affirmatively OK everything it did.
- sqlacid 8y ago
  >Output to a bluetooth speaker or headphone jack that wasn't audible?). And that can be fixed. Not sure how to fix that, if you use the aux out it has no way of knowing what happens on the other end; a confirmation prompt that isn't heard could still pick up an errant "Ok" response. Perhaps require a PIN like the shopping confirmation that would make it much much more unlikely? Or better yet, disable messaging as an option.

SilasX 8y ago

I'm just thankful that the response is more satisfying than the typical, "nope, couldn't have happened, can't reproduce".

tfandango 8y ago
I thought the response was pretty disturbing. They have logs of everything you say to Alexa? I guess I assumed the private conversations were not kept beyond some sort of metadata. No thanks.
- ahumm 8y ago
  That response could easily mean that they saw that there was an interaction where a message was recorded and sent without the device getting confirmation beforehand which matched what was reported to them, not necessarily that they kept the content of the conversation and could play it back. That's also a paraphrased conversation reported by a possibly non-technical person and likely passed through a customer service representative, so I'd take any technical details gleaned from it with a grain of salt.

PostOnce 8y ago

Fortunately we recorded everything you've ever said, so we'll just check these Alexa logs and figure it out...