5 ms·
Two women had a business meeting. AI called it childcare
- sophiabk 11mo agoWe’re building a family AI called Hold My Juice — and last week, our own system mislabeled a recurring meeting between two founders as “childcare.” Calendar: “Emily / Sophia.” Classification: “childcare.” It was a perfect snapshot of how bias seeps into everyday AI. Most models still assume women = parents, planning = domestic, logistics = mom. We’re designing from the opposite premise: AI that learns each family’s actual rhythm, values, and tone — without default stereotypes.
- orochimaaru 11mo agoAI is trained off Reddit and other social media. If most portrayal in social media of women and girls is (and men for that matter) is biased towards certain activities - that’s what AI is going to spit out. AI doesn’t think. Is this right or wrong is the incorrect question - because AI doesn’t understand bias or morality. It needs to be taught and it’s being taught from heavily biased sources. You should be able to craft prompt and guardrails to not have it do that. Just expecting it to behave that way is naive - if you have ever looked deeper into how AI is trained. The big question is - what solutions exist to train it differently with a large enough corpus of public or private/paid for data. Fwiw - I’m the father of two girls whom I have advised to stay off social media completely because it’s unhealthy. So far they have understood why.
- daveguy 11mo agoThe problem is crafted prompts and guardrails don't work very well, because these entire networks are trained on average internet garbage. And guess what's getting worse?
- orochimaaru 11mo agoAgreed. The main problem is guys with too much money invested in this bullshit asking everyone to use their snake oil. I think they’re leaning on everyone - even traditional enterprise company boards, startups, etc. to get this going. It’s not organic growth - it’s a PR machine with a trillion $$ behind an experiment.
- gwelner 11mo ago[flagged]
- cperciva 11mo agoI run into this sort of bias all the time -- in the real world, not just in AI. I take my daughter to medical appointments, both for scheduling reasons (my wife's schedule is less flexible) and rapport reasons (I'm not that kind of doctor, but I know the terminology and medical professionals treat me far more as a peer), and I routinely get "oh we expected her mother" or "we always phone the mother to schedule followup appointments". Is it so hard to understand that men can be parents too?
- johnisgood 11mo ago[flagged]
- dghlsakjg 11mo agoPresumably he already has told them his number and preferences. Defaults are fine, but you don't want your preference to get reset to default every time, and assuming that only the mother of a child should be contacted in all cases is a terrible default. The person who made the appointment and who is bringing the child to the doctor should be the one contacted by default. There is no reason that the mother of a child should be considered the default guardian. That is an incredibly dangerous assumption to make in many circumstances. Edit: This reply was written to a response that got completely rewritten in an edit. It may not make as much sense
- deleted 11mo ago[deleted]
- david38 11mo agoThis. Don’t be so sensitive, just say to call you. I took my daughter to appointments and as soon as I started asking meaningful questions, doctors immediately switched to assuming I was the one to talk to. When you act like you know what’s going on, act like you’re on top of it, I’ve never once had a doctor assume I was just babysitting. This was true in the Midwest and California.
- 11mo ago
- FloorEgg 11mo agoI have been building applications on LLMs since GPT-3. Thousands of hours of context engineering has shown me how LLMs will do their best to answer a question with insufficient context and can give all sorts of wrong answers. I've found that the way I prompt it and what information is in the context can heavily bias the way it responds when it doesn't have enough information to respond accurately. You assume the bias is in the LLM itself, but I am very suspicious that the bias is actually in your system prompt and context engineering. Are you willing to share the system prompt that led to this result that you're claiming is sexist LLM bias? Edit: Oidar (child comment to this) did an A/B test with male names and it seems to have proven the bias is indeed in the LLM, and that my suspicion of it coming from the prompt+context was wrong. Kudos and thanks for taking the time.
- small_scombrus 11mo ago> You assume the bias is in the LLM itself Common large datasets being inherently biased towards some ideas/concepts and away from others in ways that imply negative things is something that there's a LOT of literature about
- johnisgood 11mo ago"imply negative things"? What is "negative" here? I see nothing that is "negative".
- small_scombrus 11mo agoThat a regular meeting between two women must be about childcare because women=childcare?
- johnisgood 11mo agoYeah except I asked Claude: > No. There's no indication that children are involved or that care is being provided. It's just two people meeting. Part of its thinking: > This is a very vague description with no context about: > What happens during the meeting > Whether children are present > What the purpose of the meeting is > Any other relevant details Claude is not going to say childcare, and it is not saying it is childcare. My prompt was: ""regular meeting between two women". Is it childcare or not?".
- callan101 11mo agoThis feels a tad rigged against the LLM with the meeting being after Kids drop off.
- cheald 11mo agoEasily half the other events on the calendar are kid-related. Of course it's going to infer that, absent other direction, the most likely overarching theme of the visible events is "child care".
- drivingmenuts 11mo agoSure, but the LLM needs to prove that it can make inferences as well as or better than a human, in order to be useful. Aside from that, it's not human, so there's no need to be fair - it should do what we tell it, not decide on its own.
- broof 11mo agoI hate that when I see this many em dashes, as well as statements like “it’s not x, it’s y” multiple times, I have to assume it was written or at least heavily edited by AI.
- somewhereoutth 11mo agoLLMs: The chemical weapons of public discourse. The cleanup is going to be a grim task.
- drivingmenuts 11mo agoThere will be an LLM for that. God help us all.
- OutOfHere 11mo ago[flagged]
- sentel5 11mo ago[dead]
- gwelner 11mo ago[flagged]
- deleted 11mo ago[deleted]
- oidar 11mo agoHere's an A/B Emily / Sophia vs Bob / John https://imgur.com/a/9yt5rpA https://imgur.com/a/9yt5rpA
- FloorEgg 11mo agoThis is really interesting and way more compelling evidence to me of gender bias in the LLM than response bias in the prompt and context. Thank you for taking the time to approach this scientifically and share the evidence with us. I appreciate knowing the truth of the matter, and it seems my suspicion that the bias was from the prompt was wrong. I admit I am surprised.
- sophiabk 11mo agoThank you for doing this analysis. It's shocking (if understandable why given the examples it was trained on). What is exciting though is as we're working to train each individual family's AI - understanding roles, jobs, interests etc - it's picked up on things in a much less biased way.
- deleted 11mo ago[deleted]
- ryandrake 11mo agoI wonder if the users who flagged this could chime in to explain what is rule-breaking about this article?
- FloorEgg 11mo agoI was wondering that myself too. Also, do moderators ever move comments around? I thought one comment was a child to my comment last I looked, but now it's a top level comment to this post. I'm not sure if I am mistaken or a moderator moved things around.
- ryandrake 11mo agoThis does happen from time to time. A moderator will "detach" a subthread[1] and move it to the top-level (usually also burying it at the bottom of the page, which tends to silence the discussion). 1: https://news.ycombinator.com/item?id=23441803 https://news.ycombinator.com/item?id=23441803