47 ms·

> nobody should be deploying software in production that is capable of this kind of harm. This "kind" of harm seems to hide a huge amount of complexity. Let's

> nobody should be deploying software in production that is capable of this kind of harm. This "kind" of harm seems to hide a huge amount of complexity. Let's assume the algorithm starts off on day 1 just looking for purchases that tend to appear in clusters. There's a potentially unbounded number of clusters that could be regarded as harmful. The options are surely: 1. Be vigilant and special case items one by one. I presume they are doing this. Maybe they aren't putting enough resources behind it or not reacting quickly enough but even if they were then things would always slip through. 2. Never use any kind of bundling algorithm ever. (1) is a forever ongoing task so it doesn't sound like you're referring to that as that's not something where you can "iron out the kinks". However it also doesn't sound like you mean (2) as you seem to be saying there's a way to fix this once and for all. Is there a third option I haven't thought of? EDIT - I guess maybe all "bundles" could require human curation before they are allowed to be displayed. It would be a gargantuan task but it's a valid third option.

CPLX 4y ago
Nothing slipped through. Amazon has been fully informed about it, is doing it on purpose, and is engaging in expensive legal battles to be allowed to keep doing it. This story is so much worse than the headline suggests, despite that seeming impossible.
- iancmceachern 4y ago
  Why on earth would they do so? I believe you 100%, but why Jeff, why?
  - bombcar 4y ago
    Precedent. If they artificially limit their "AI" from outside pressure, then they will have admitted that they can do it, and they'll get pressured to do it more and more. They don't want to deal with that, and would rather wait until a law is passed "forcing" them to do so - coincidentally Amazon will be best positioned to comply with that law and for some reason it'll only ever be used to go after smaller competitors.
  - CPLX 4y ago
    Because someone decided admitting they fucked up profoundly here is too scary from a business perspective. They are obviously afraid of a precedent of having to answer for what they sell on their site and what happens next. It’s understandable but no less sociopathic once understood.
  - goatherd 4y ago
    The type of people who run Amazon have zero social conscience. As another example, just look at how badly they treat their warehouse workers.
  - michaelmrose 4y ago
    They don't want to establish the precedent that recommending a harmful item makes them liable for that harm at any cost even at the cost of a few dead children. Before this in a previous scandal they were promoting "literature" that suggested parents of autistic children cure them with bleach and recommended the bleach they could inadvertently murder their children with.
- FearlessNebula 4y ago
  There’s no way they’re doing this on purpose. They can’t possibly be selling enough of these to offset legal fees and bad press
  - ectopod 4y ago
    When you say you are doing nothing wrong and continue with an action then you are doing it on purpose.
Lightbody 4y ago
It does not seem like they are doing 1 in any sufficient way: its pretty easy to argue that this is by far the most important special case (it’s literally life and death stakes), and they are failing at it even after a growing PR campaign and advocacy made them aware months (years?) ago. Amazon is a $1T company. They have the resources. This is willful negligence.
- andybak 4y ago
  I entirely agree. My comment was directed towards the specific comment I was replying to.
bastawhiz 4y ago
There's a hell of a difference between recommending pairing a cute shirt with some cute earrings and pairing an industrial chemical with an item from the pharmaceuticals category. There are extremely basic and straightforward safeguards that could have been put in place that simply do not exist.
- catiopatio 4y ago
  Why is it amazon’s responsibility to safeguard your children? That’s your responsibility. They can still figure out how to buy the items separately, and Amazon isn’t inducing suicidal ideation. Perhaps your kids shouldn’t have unrestricted and unmonitored access to order things from Amazon? Start there, instead of insisting that Amazon do the parenting for you.
  - CPLX 4y ago
    > Why is it amazon’s responsibility to safeguard your children? That’s your responsibility. Indeed. And we intend to exercise that responsibility to safeguard our children by stopping Amazon from mixing suicide kits in with shipments of innocuous household goods.
    - catiopatio 4y ago
      You intend to exercise your responsibility by abdicating it, and instead insisting that Amazon and other retailers do it for you, and society at large bear the cost? I think we disagree on the definition of “responsibility”.
      - CPLX 4y ago
        Yes. Society can bear the cost of having to jump through a few extra steps to buy pure concentrated chemicals that have been demonstrated to repeatedly kill teenagers. So the people who have legitimate use for those chemicals can jump through a few hoops to get them. And Amazon, a trillion dollar company, can do a better job with its algorithms and stop recommending them and selling them to teenagers. I am super comfortable inflicting those costs on the world.
mrcus 4y ago
This is a good example showing that all work is not removed by automating a task, such as selecting which products are related and displaying that on an online store’s product page. Some work may be removed, but new work is added to compensate for what the algorithms can not trivially do, such as considering whether the links between these products are ethical to present to users. It seems to me that companies massively underestimate (or willingly don’t compensate for) the importance of this newly introduced work, only cashing in on the savings from the work removed. Other examples of similar behavior include automatic moderation, which also massively fails to remove actual dangerous content. This kind of behavior looks like greed to me, but I guess it’s legal, and needs more regulation.
foepys 4y ago
> EDIT - I guess maybe all "bundles" could require human curation before they are allowed to be displayed. It would be a gargantuan task but it's a valid third option. This is the correct way to handle it. It being a "gargantuan task" doesn't matter. Everybody would freak out if the local supermarket advertised this kind of product but somehow "tech" (read: online retailers) can get away with this because they are big? Amazon posts billions in profits, it can certainly afford to control their algorithms. If it's impossible, then stop using algorithms and use human curation like literally every brick and mortar business.
akudha 4y ago
How about not using bundling algorithms for chemicals and pharmaceuticals, the entire category? And stick to recommending cute t-shirts and such? Amazon’s recommendations are mostly crap anyways, even for mundane, straightforward stuff. They shouldn’t be recommending anything dangerous
- Comevius 4y ago
  Exactly this, Amazon's recommendation engine doesn't have the domain knowledge about chemicals and such, what can be mixed with what. Their recommendation may mislead people.
- andybak 4y ago
  The only results I get for "sodium nitrate" is in "Herbs, Spices & Seasonings" (at low concentrations) as part of a salt curing kit. In fact in the UK I'm not seeing any results that could be described as a "suicide kit". The US site does seem to stock chemicals but from a quick skim I don't seem to get any sodium nitrate in the results.
- Ferret7446 4y ago
  How would that even address the problem? Someone buying these things to commit suicide will still buy these things even if they need to do a few more clicks and searches because the items aren't conveniently bundled or recommended.
  - akudha 4y ago
    Yes, people who want to commit suicide might still succeed. That doesn't mean Amazon or you or I need to make it easier for them or facilitate them.
    - catiopatio 4y ago
      That also doesn’t mean Amazon or you or I need to make life harder for the 99%+ of the population that isn’t going to top themselves.
deleted 4y ago
[deleted]

CPLX 4y ago

Nothing slipped through. Amazon has been fully informed about it, is doing it on purpose, and is engaging in expensive legal battles to be allowed to keep doing it. This story is so much worse than the headline suggests, despite that seeming impossible.

iancmceachern 4y ago
Why on earth would they do so? I believe you 100%, but why Jeff, why?
- bombcar 4y ago
  Precedent. If they artificially limit their "AI" from outside pressure, then they will have admitted that they can do it, and they'll get pressured to do it more and more. They don't want to deal with that, and would rather wait until a law is passed "forcing" them to do so - coincidentally Amazon will be best positioned to comply with that law and for some reason it'll only ever be used to go after smaller competitors.
- CPLX 4y ago
  Because someone decided admitting they fucked up profoundly here is too scary from a business perspective. They are obviously afraid of a precedent of having to answer for what they sell on their site and what happens next. It’s understandable but no less sociopathic once understood.
- goatherd 4y ago
  The type of people who run Amazon have zero social conscience. As another example, just look at how badly they treat their warehouse workers.
- michaelmrose 4y ago
  They don't want to establish the precedent that recommending a harmful item makes them liable for that harm at any cost even at the cost of a few dead children. Before this in a previous scandal they were promoting "literature" that suggested parents of autistic children cure them with bleach and recommended the bleach they could inadvertently murder their children with.
FearlessNebula 4y ago
There’s no way they’re doing this on purpose. They can’t possibly be selling enough of these to offset legal fees and bad press
- ectopod 4y ago
  When you say you are doing nothing wrong and continue with an action then you are doing it on purpose.

Lightbody 4y ago

It does not seem like they are doing 1 in any sufficient way: its pretty easy to argue that this is by far the most important special case (it’s literally life and death stakes), and they are failing at it even after a growing PR campaign and advocacy made them aware months (years?) ago. Amazon is a $1T company. They have the resources. This is willful negligence.

andybak 4y ago
I entirely agree. My comment was directed towards the specific comment I was replying to.

bastawhiz 4y ago

There's a hell of a difference between recommending pairing a cute shirt with some cute earrings and pairing an industrial chemical with an item from the pharmaceuticals category. There are extremely basic and straightforward safeguards that could have been put in place that simply do not exist.

catiopatio 4y ago
Why is it amazon’s responsibility to safeguard your children? That’s your responsibility. They can still figure out how to buy the items separately, and Amazon isn’t inducing suicidal ideation. Perhaps your kids shouldn’t have unrestricted and unmonitored access to order things from Amazon? Start there, instead of insisting that Amazon do the parenting for you.
- CPLX 4y ago
  > Why is it amazon’s responsibility to safeguard your children? That’s your responsibility. Indeed. And we intend to exercise that responsibility to safeguard our children by stopping Amazon from mixing suicide kits in with shipments of innocuous household goods.
  - catiopatio 4y ago
    You intend to exercise your responsibility by abdicating it, and instead insisting that Amazon and other retailers do it for you, and society at large bear the cost? I think we disagree on the definition of “responsibility”.
    - CPLX 4y ago
      Yes. Society can bear the cost of having to jump through a few extra steps to buy pure concentrated chemicals that have been demonstrated to repeatedly kill teenagers. So the people who have legitimate use for those chemicals can jump through a few hoops to get them. And Amazon, a trillion dollar company, can do a better job with its algorithms and stop recommending them and selling them to teenagers. I am super comfortable inflicting those costs on the world.

mrcus 4y ago

This is a good example showing that all work is not removed by automating a task, such as selecting which products are related and displaying that on an online store’s product page. Some work may be removed, but new work is added to compensate for what the algorithms can not trivially do, such as considering whether the links between these products are ethical to present to users. It seems to me that companies massively underestimate (or willingly don’t compensate for) the importance of this newly introduced work, only cashing in on the savings from the work removed. Other examples of similar behavior include automatic moderation, which also massively fails to remove actual dangerous content. This kind of behavior looks like greed to me, but I guess it’s legal, and needs more regulation.

foepys 4y ago

> EDIT - I guess maybe all "bundles" could require human curation before they are allowed to be displayed. It would be a gargantuan task but it's a valid third option. This is the correct way to handle it. It being a "gargantuan task" doesn't matter. Everybody would freak out if the local supermarket advertised this kind of product but somehow "tech" (read: online retailers) can get away with this because they are big? Amazon posts billions in profits, it can certainly afford to control their algorithms. If it's impossible, then stop using algorithms and use human curation like literally every brick and mortar business.

akudha 4y ago

How about not using bundling algorithms for chemicals and pharmaceuticals, the entire category? And stick to recommending cute t-shirts and such? Amazon’s recommendations are mostly crap anyways, even for mundane, straightforward stuff. They shouldn’t be recommending anything dangerous

Comevius 4y ago
Exactly this, Amazon's recommendation engine doesn't have the domain knowledge about chemicals and such, what can be mixed with what. Their recommendation may mislead people.
andybak 4y ago
The only results I get for "sodium nitrate" is in "Herbs, Spices & Seasonings" (at low concentrations) as part of a salt curing kit. In fact in the UK I'm not seeing any results that could be described as a "suicide kit". The US site does seem to stock chemicals but from a quick skim I don't seem to get any sodium nitrate in the results.
Ferret7446 4y ago
How would that even address the problem? Someone buying these things to commit suicide will still buy these things even if they need to do a few more clicks and searches because the items aren't conveniently bundled or recommended.
- akudha 4y ago
  Yes, people who want to commit suicide might still succeed. That doesn't mean Amazon or you or I need to make it easier for them or facilitate them.
  - catiopatio 4y ago
    That also doesn’t mean Amazon or you or I need to make life harder for the 99%+ of the population that isn’t going to top themselves.

deleted 4y ago

[deleted]