5 ms·

TikTok, Inc., via its algorithm, recommended and promoted videos posted by third parties to ten-year-old Nylah Anderson on her uniquely curated “For You Page.”

TikTok, Inc., via its algorithm, recommended and promoted videos posted by third parties to ten-year-old Nylah Anderson on her uniquely curated “For You Page.” One video depicted the “Blackout Challenge,” which encourages viewers to record themselves engaging in acts of self-asphyxiation. After watching the video, Nylah attempted the conduct depicted in the challenge and unintentionally hanged herself. -- https://cases.justia.com/federal/appellate-courts/ca3/22-3061/22-3061-2024-08-27.pdf?ts=1724792413 An algorithm accidentally enticed a child to hang herself. I've got code running on dozens of websites that recommends articles to read based on user demographics. There's nothing in that code that would or could prevent an article about self-asphyxiation being recommended to a child. It just depends on the clients that use the software not posting that kind of content, people with similar demographics to the child not reading it, and a child who gets the recommendation not reading it and acting it out. If those assumptions fail should I or my employer be liable?

discardedrefuse 2y ago
Isn't it usually the case that when someone builds a shitty thing and people get hurt, the builder is liable?
- ineedaj0b 2y ago
  Yeah, but buying a hammer and hitting yourself with it is different. The dangers of social media are unknown to most still.
  - discardedrefuse 2y ago
    Yes. Buying a hammer and hitting yourself with it IS different.
  - spacemadness 2y ago
    Yes, because a mechanical tool made of solid metal is the same thing as software that can change its behavior at any time and is controlled live by some company with its own motives.
  - ThunderSizzle 2y ago
    It's be more akin to buying a hammer and then the hammer starts morphing into a screw driver without you noticing. Then when you accidentally hit your hand with the hammer, you actually stabbed yourself. And that's when you realized your hammer is now a screwdriver.
    - ineedaj0b 2y ago
      Yes, I thought that’s what I said - no one knows the shape of danger social media is currently. It’s like trying to draw tiger but you’ve never seen an animal. We only have the faintest clue what social media is right now. It will change in the next 25+ years as well. Sure we know some dangers but… I think we need more time to know them all.
      - BlueTemplar 2y ago
        We already know that broadcasting (aka the 4th power) comes with responsibility. We've known that for several hundred years by now. And social media (with editorialized or even curated-for-specific-user feeds) have been around for more than a decade now (Facebook's big Timeline update for instance dates to 2011).
- x0x0 2y ago
  You would think so, wouldn't you? Except right now youtube have a self advertisement in the middle of the page warning people not to trust the content on youtube. A company warning people not to trust the product they built and the videos they choose to show you... we need to rethink 230. We've gone seriously awry.
  - tines 2y ago
    It's more nuanced than that. If I sent a hateful letter through the mail and someone gets hurt by it (even physically), who is responsible, me or the post office? I know youtube is different in important ways than the post, but it's also different in important ways from e.g. somebody who builds a building that falls down.
    - cmrdporcupine 2y ago
      The Post Office just delivers your mail, it doesn't do any curation. YouTube, TikTok, etc. differ by applying an algorithm to "decide" what to show you. Those algorithms have all sorts of weights and measures, but they're ultimately personalized to you. And if they're making personalized recommendations that include "how to kill yourself"... I think we have a problem? It's simply not just a FIFO of content in content out, and in many cases (Facebook & Instagram especially) the user barely gets to a choice in what is shown in the feed... Contrast with e.g. Mastodon where there is no algorithm and it only shows you what you explicitly followed, and in the exact linear order it was posted. (Which is actually how Facebook used to be)
      - anticensor 2y ago
        > (Which is actually how Facebook used to be) There is now too many posts on Facebook to do that.
    - ang_cire 2y ago
      If the post office opened your letter, read it, and then decided to copy it and send it to a bunch of kids, you would be responsible for your part in creating it, and they would be responsible for their part in disseminating it.
- deleted 2y ago
  [deleted]
mihaaly 2y ago
Yes. Or you do things that gives you rewards - and do not care what it will result otherwise - but you want to be saved from any responsibility (automatically!) for what it causes just because it is an algorithm? The enjoying the benefits but running away from responsibility is a cowardly and childish act. Childish acts need supervision from adults.
- EasyMark 2y ago
  What happened to that child is on the parents not some programmer who coded an optimization algorithm. It’s really as simple as that. No 10 year old should be on TikTok, I’m not sure anyone under 18 should be given the garbage, dangerous misinformation, intentional disinformation, and lack of any ability to control what your child sees.
  - itishappy 2y ago
    Do you feel the same way about the sale of alcohol? I do see the argument for parental responsibility, but I'm not sure how parents will enforce that if the law allows people to sell kids alcohol free from liability.
    - tines 2y ago
      This is a good argument I didn't think of before. What's the response to it?
      - Flozzin 2y ago
        We regulate the sale of all sorts of things that can do damage but also have other uses. You can't buy large amounts of certain cold medicines, and you need to be an adult to do so. You can't buy fireworks if you are a minor in most places. In some countries they won't even sell you a set of steak knives if you are underage. Someone else's response was that a 10 year old should not be on ticktoc. Well then how did they get past the age restrictions?(I'm guessing its a check box at best). So its inadequately gated. But really, I don't think its the sort of thing that needs an age gate. They are responsible for a product that is actively targeting harmful behavior at children and adults. It's not ok in either situation. You cannot allow your platform to be hijacked for content like this. Full stop. These 'services' need better ways to moderate content. If that is more controls that allow them to delete certain posts and videos or some other method to contain videos like this. You cannot just allow users to upload and share whatever they want. And further, have your own systems promote these videos. Everyone who makes a product(especially for mass consumption), has a responsibility to make sure their product is safe. If your product is so complicated that you can't control it, then you need to step back and re-evaluate how it's functioning. Not just plow ahead, making money, letting it harm people.
itishappy 2y ago
It sounds like your algorithm targets children with unmoderated content. That feels like a dangerous position with potential for strong arguments in either direction. I think the only reasonable advice here is to keep close tabs on this case.
- Sohcahtoa82 2y ago
  Does it specifically target children or does it simply target people and children happen to be some of the people using it? If a child searches Google for "boobs", it's not fair to accuse Google of showing naked women to children, and definitely not fair to even say Google was targeting children.
  - BlueTemplar 2y ago
    You're getting off topic here though ? «Had Nylah viewed a Blackout Challenge video through TikTok’s search function, rather than through her FYP, then TikTok may be viewed more like a repository of third-party content than an affirmative promoter of such content.»
    - Sohcahtoa82 2y ago
      You must be responding to the wrong person. It was other people claiming that the algorithm targets children.
  - itishappy 2y ago
    I'd say explicitly allowing children on your site (such as TikTok and Google) is targeting children. Any company doing so should be careful with the content on their front pages and feeds.
drpossum 2y ago
You sure are if you knew about it (like tiktok was)
- yadaeno 2y ago
  Even if they didn’t know it should be their responsibility to know regardless
troyvit 2y ago
Right? Like if I'm a cement company, and I build a sidewalk that's really good and stable, stable enough for a person to plant a milk crate on it, and stand on that milk crate, and hold up a big sign that gives clear instructions on self-asphyxiation, and a child reads that sign, tries it out and dies, am I going to get sued? All I did was build the foundation for a platform.
- averageRoyalty 2y ago
  That's not a fair analogy though. To be fairer, you'd have to monitor said footpath 24/7 and have a robot and/or a number of people removing milk crate signs that you deemed inappropriate for your foothpath. They'd also move various milk crate signs in front of people as they walked and hide others. If you were indeed monitoring the footpath for milk crate signs and moving them, yes you may be liable for showing or not removing one to someone it wouldn't be appropriate for.
  - troyvit 2y ago
    That's a good point, and actually the heart of the issue, and what I missed. In my analogy the stable sidewalk that can hold the milk crate is both the platform and the optimization algorithm. But to your point there's actually a lot more going on with the optimization than just building a place where any rando can market self-asphyxiation. It's about how they willfully targeted people with that content.
awongh 2y ago
I think it depends on some technical specifics, like which meta data was associated with that content, and the degree to which that content was surfaced to users that fit the demographic profile of a ten year old child. If your algorithm decides that things in the 90th percentile of shock value will boost engagement to a user profile that can also include users who are ten years old then you maybe have built a negligent algorithm. Maybe that’s not the case in this particular instance but it could be possible.
thinkingtoilet 2y ago
Of course you should be. Just because an algorithm gave you an output doesn't absolve you from using it. It's some magical mystical thing. It's something you created and you are 100% responsible for what you do with the output of it.
thih9 2y ago
Yes, if a product actively contributes to child fatalities then the manufacturer should be liable. Then again, I guess your platform is about article recommendation and not about recording yourself doing popular trends. And perhaps children are not your target audience, or an audience at all. In many ways the situation was different for TikTok.
plandis 2y ago
Part of the claim is that TikTok knew about this content being promoted and other cases where children had died as a result. > But by the time Nylah viewed these videos, TikTok knew that: 1) "the deadly Blackout Challenge was spreading through its app," 2) "its algorithm was specifically feeding the Blackout Challenge to children," and 3) several children had died while attempting the Blackout Challenge after viewing videos of the Challenge on their For You Pages. App. 31-32. Yet TikTok "took no and/or completely inadequate action to extinguish and prevent the spread of the Blackout Challenge and specifically to prevent the Blackout Challenge from being shown to children on their [For You Pages]." App. 32-33. Instead, TikTok continued to recommend these videos to children like Nylah. Do you think this should be legal? Would you do nothing if you knew children were dying directly because of the content you were feeding them?
ang_cire 2y ago
"I have a catapult that launches loosely demo-targeted things, without me checking what is being loaded into it. I only intend for harmless things to be loaded. Should I be liable if someone loads a boulder and it hurts someone?"

discardedrefuse 2y ago

Isn't it usually the case that when someone builds a shitty thing and people get hurt, the builder is liable?

ineedaj0b 2y ago
Yeah, but buying a hammer and hitting yourself with it is different. The dangers of social media are unknown to most still.
- discardedrefuse 2y ago
  Yes. Buying a hammer and hitting yourself with it IS different.
- spacemadness 2y ago
  Yes, because a mechanical tool made of solid metal is the same thing as software that can change its behavior at any time and is controlled live by some company with its own motives.
- ThunderSizzle 2y ago
  It's be more akin to buying a hammer and then the hammer starts morphing into a screw driver without you noticing. Then when you accidentally hit your hand with the hammer, you actually stabbed yourself. And that's when you realized your hammer is now a screwdriver.
  - ineedaj0b 2y ago
    Yes, I thought that’s what I said - no one knows the shape of danger social media is currently. It’s like trying to draw tiger but you’ve never seen an animal. We only have the faintest clue what social media is right now. It will change in the next 25+ years as well. Sure we know some dangers but… I think we need more time to know them all.
    - BlueTemplar 2y ago
      We already know that broadcasting (aka the 4th power) comes with responsibility. We've known that for several hundred years by now. And social media (with editorialized or even curated-for-specific-user feeds) have been around for more than a decade now (Facebook's big Timeline update for instance dates to 2011).
x0x0 2y ago
You would think so, wouldn't you? Except right now youtube have a self advertisement in the middle of the page warning people not to trust the content on youtube. A company warning people not to trust the product they built and the videos they choose to show you... we need to rethink 230. We've gone seriously awry.
- tines 2y ago
  It's more nuanced than that. If I sent a hateful letter through the mail and someone gets hurt by it (even physically), who is responsible, me or the post office? I know youtube is different in important ways than the post, but it's also different in important ways from e.g. somebody who builds a building that falls down.
  - cmrdporcupine 2y ago
    The Post Office just delivers your mail, it doesn't do any curation. YouTube, TikTok, etc. differ by applying an algorithm to "decide" what to show you. Those algorithms have all sorts of weights and measures, but they're ultimately personalized to you. And if they're making personalized recommendations that include "how to kill yourself"... I think we have a problem? It's simply not just a FIFO of content in content out, and in many cases (Facebook & Instagram especially) the user barely gets to a choice in what is shown in the feed... Contrast with e.g. Mastodon where there is no algorithm and it only shows you what you explicitly followed, and in the exact linear order it was posted. (Which is actually how Facebook used to be)
    - anticensor 2y ago
      > (Which is actually how Facebook used to be) There is now too many posts on Facebook to do that.
  - ang_cire 2y ago
    If the post office opened your letter, read it, and then decided to copy it and send it to a bunch of kids, you would be responsible for your part in creating it, and they would be responsible for their part in disseminating it.
deleted 2y ago
[deleted]

mihaaly 2y ago

Yes. Or you do things that gives you rewards - and do not care what it will result otherwise - but you want to be saved from any responsibility (automatically!) for what it causes just because it is an algorithm? The enjoying the benefits but running away from responsibility is a cowardly and childish act. Childish acts need supervision from adults.

EasyMark 2y ago
What happened to that child is on the parents not some programmer who coded an optimization algorithm. It’s really as simple as that. No 10 year old should be on TikTok, I’m not sure anyone under 18 should be given the garbage, dangerous misinformation, intentional disinformation, and lack of any ability to control what your child sees.
- itishappy 2y ago
  Do you feel the same way about the sale of alcohol? I do see the argument for parental responsibility, but I'm not sure how parents will enforce that if the law allows people to sell kids alcohol free from liability.
  - tines 2y ago
    This is a good argument I didn't think of before. What's the response to it?
    - Flozzin 2y ago
      We regulate the sale of all sorts of things that can do damage but also have other uses. You can't buy large amounts of certain cold medicines, and you need to be an adult to do so. You can't buy fireworks if you are a minor in most places. In some countries they won't even sell you a set of steak knives if you are underage. Someone else's response was that a 10 year old should not be on ticktoc. Well then how did they get past the age restrictions?(I'm guessing its a check box at best). So its inadequately gated. But really, I don't think its the sort of thing that needs an age gate. They are responsible for a product that is actively targeting harmful behavior at children and adults. It's not ok in either situation. You cannot allow your platform to be hijacked for content like this. Full stop. These 'services' need better ways to moderate content. If that is more controls that allow them to delete certain posts and videos or some other method to contain videos like this. You cannot just allow users to upload and share whatever they want. And further, have your own systems promote these videos. Everyone who makes a product(especially for mass consumption), has a responsibility to make sure their product is safe. If your product is so complicated that you can't control it, then you need to step back and re-evaluate how it's functioning. Not just plow ahead, making money, letting it harm people.

itishappy 2y ago

It sounds like your algorithm targets children with unmoderated content. That feels like a dangerous position with potential for strong arguments in either direction. I think the only reasonable advice here is to keep close tabs on this case.

Sohcahtoa82 2y ago
Does it specifically target children or does it simply target people and children happen to be some of the people using it? If a child searches Google for "boobs", it's not fair to accuse Google of showing naked women to children, and definitely not fair to even say Google was targeting children.
- BlueTemplar 2y ago
  You're getting off topic here though ? «Had Nylah viewed a Blackout Challenge video through TikTok’s search function, rather than through her FYP, then TikTok may be viewed more like a repository of third-party content than an affirmative promoter of such content.»
  - Sohcahtoa82 2y ago
    You must be responding to the wrong person. It was other people claiming that the algorithm targets children.
- itishappy 2y ago
  I'd say explicitly allowing children on your site (such as TikTok and Google) is targeting children. Any company doing so should be careful with the content on their front pages and feeds.

drpossum 2y ago

You sure are if you knew about it (like tiktok was)

yadaeno 2y ago
Even if they didn’t know it should be their responsibility to know regardless

troyvit 2y ago

Right? Like if I'm a cement company, and I build a sidewalk that's really good and stable, stable enough for a person to plant a milk crate on it, and stand on that milk crate, and hold up a big sign that gives clear instructions on self-asphyxiation, and a child reads that sign, tries it out and dies, am I going to get sued? All I did was build the foundation for a platform.

averageRoyalty 2y ago
That's not a fair analogy though. To be fairer, you'd have to monitor said footpath 24/7 and have a robot and/or a number of people removing milk crate signs that you deemed inappropriate for your foothpath. They'd also move various milk crate signs in front of people as they walked and hide others. If you were indeed monitoring the footpath for milk crate signs and moving them, yes you may be liable for showing or not removing one to someone it wouldn't be appropriate for.
- troyvit 2y ago
  That's a good point, and actually the heart of the issue, and what I missed. In my analogy the stable sidewalk that can hold the milk crate is both the platform and the optimization algorithm. But to your point there's actually a lot more going on with the optimization than just building a place where any rando can market self-asphyxiation. It's about how they willfully targeted people with that content.

awongh 2y ago

I think it depends on some technical specifics, like which meta data was associated with that content, and the degree to which that content was surfaced to users that fit the demographic profile of a ten year old child. If your algorithm decides that things in the 90th percentile of shock value will boost engagement to a user profile that can also include users who are ten years old then you maybe have built a negligent algorithm. Maybe that’s not the case in this particular instance but it could be possible.

thinkingtoilet 2y ago

Of course you should be. Just because an algorithm gave you an output doesn't absolve you from using it. It's some magical mystical thing. It's something you created and you are 100% responsible for what you do with the output of it.

thih9 2y ago

Yes, if a product actively contributes to child fatalities then the manufacturer should be liable. Then again, I guess your platform is about article recommendation and not about recording yourself doing popular trends. And perhaps children are not your target audience, or an audience at all. In many ways the situation was different for TikTok.

plandis 2y ago

Part of the claim is that TikTok knew about this content being promoted and other cases where children had died as a result. > But by the time Nylah viewed these videos, TikTok knew that: 1) "the deadly Blackout Challenge was spreading through its app," 2) "its algorithm was specifically feeding the Blackout Challenge to children," and 3) several children had died while attempting the Blackout Challenge after viewing videos of the Challenge on their For You Pages. App. 31-32. Yet TikTok "took no and/or completely inadequate action to extinguish and prevent the spread of the Blackout Challenge and specifically to prevent the Blackout Challenge from being shown to children on their [For You Pages]." App. 32-33. Instead, TikTok continued to recommend these videos to children like Nylah. Do you think this should be legal? Would you do nothing if you knew children were dying directly because of the content you were feeding them?

ang_cire 2y ago

"I have a catapult that launches loosely demo-targeted things, without me checking what is being loaded into it. I only intend for harmless things to be loaded. Should I be liable if someone loads a boulder and it hurts someone?"