7 ms·

Google pulls AI model after senator says it fabricated assault allegation

84 points by croemer 1y ago

rchaud 1y ago
Terrifying to think that some techbro is out there right now concocting plans for an "AI background check" startup.
- confounder 1y ago
  Like this one? https://checkr.com/our-technology/ai-powered https://checkr.com/our-technology/ai-powered
croemer 1y ago
This is about Gemma, Google's open weights model. And specifically availability through AI studio. I don't think they'll make the weights unavailable.
- aleatorianator 1y ago
  [flagged]
  - jqpabc123 1y ago
    This is about these tools being blatantly flawed and unreliable. In legal terms, marketing such a product is called "negligence" or "libel". Lots of software is flawed and unreliable but this is typically addressed in the terms of service. This may not be possible with AI because the "liability" can extend well beyond just the user.
    - flufluflufluffy 1y ago
      The purpose of the tool is for writing code, it is not for generating factual English sentences.
      - mikkupikku 1y ago
        I do think that might be the only thing they turn out to be any good at, and only then because software is relatively easy to test and iterate on. But does that mean writing code is what the models are "for"? They're marketed as being good for a lot more than coding.
      - jqpabc123 1y ago
        it is not for generating factual English sentences. Then the tool should not be doing it --- but it does. And therein is the legal liability.
        flufluflufluffy 1y ago
        The tool did it because the person asked it to. They used the tool the wrong way. The knives are entering people’s guts. They should not be doing that. The knife companies should be liable for these stabbings.
        jqpabc123 1y ago
        The tool did it because the person asked it to. The tool did it because this is what it was designed and trained to do --- at great expense and effort --- but somewahat less than successfully. You can't have it both ways --- the tool can't be "intelligent" yet too stupid to understand it's own limitations. If people ask the wrong questions, the "intelligent" response would be, "Sorry, I can't do that". Maybe the problem here is that this "intelligent" tool is really as dumb as a rock. And it's only a matter of time until lawyers argue this point to a jury in court.
    - jwitthuhn 1y ago
      It is wrong to release something unreliable even while acknowledging it is unreliable? The product performs as advertised. If people want accurate information an LLM is the wrong tool for the job. From the Gemma 3 readme on huggingface: "Models generate responses based on information they learned from their training datasets, but they are not knowledge bases. They may generate incorrect or outdated factual statements."
      - jqpabc123 1y ago
        If people want accurate information an LLM is the wrong tool for the job. So these vendors spent lots of time and money training LLMs to answer questions that people should not ask --- but are allowed and encouraged to. Nonsensical and unrealistic. I expect the courts will agree and hold the vendors liable.
  - notavalleyman 1y ago
    You wrote this comment under and article about Gemma, an open weights model which anyone can download and run at home. Here is more info and links to the models, so you can interrogate them about Senatorial scandals on your hardware at home. https://huggingface.co/blog/gemma https://huggingface.co/blog/gemma Your claim was so far from the truth of reality, that now, it's incumbent upon you to go back through the chain of faulty reasoning. You took it for granted that a conspiracy theory about suppressing information was true, when actually, the same Gemma model was already open-weighted by the same conspirators who you accuse of keeping Gemma out of regular peoples' reach
jqpabc123 1y ago
AI is going to be a lawyer's wet dream. Imagine the ads on TV: "Has AI lied about you? Your case could be worth millions. Call now!"
- Handy-Man 1y ago
  I mean even lawyers are getting sanctioned for using them without verifying their outputs lol. - https://www.msba.org/site/site/content/News-and-Publications/News/General-News/Massachusetts_Lawyer-Sanctioned_for_AI_Generated-Fictitious_Cases.aspx https://www.msba.org/site/site/content/News-and-Publications... - https://www.reuters.com/legal/government/judge-disqualifies-three-butler-snow-attorneys-case-over-ai-citations-2025-07-24/ https://www.reuters.com/legal/government/judge-disqualifies-... - https://calmatters.org/economy/technology/2025/09/chatgpt-lawyer-fine-ai-regulation/ https://calmatters.org/economy/technology/2025/09/chatgpt-la...
- ghjv 1y ago
  strange to talk about it in the future tense. it's here and yep, it's an object of fascination for the legal system
- rchaud 1y ago
  Unlike medicine, AI isn't regulated, so lawyers won't have anything to go after.
  - advisedwang 1y ago
    I think the parent post imagines defamation cases will be worthwhile. I'm sure there will be some, but an AI simply lying in a query doesn't = damages worth suing over.
    - jqpabc123 1y ago
      It depends on who is the lie was about and what was said. Simple example: A prospective employer refuses to hire you because of some blatant falsehood generated by an LLM.
- 1y ago
hnuser123456 1y ago
There should probably be a little more effort towards making small models that don't just make things up when asked a factual question. All of us who have played with small models know there's just not as much room for factual info, they are middle schoolers who just write anything. Completely fabricated references are clearly an ongoing weakness, and easy to validate.
- rewilder12 1y ago
  LLMs by definition do not make facts. You will never be able to eliminate hallucinations. It's practically impossible. Big tech created a problem for themselves by allowing people to believe the things their products generate using LLMs are facts. We are only reaching the obvious conclusion of where this leads.
  - hnuser123456 1y ago
    To be sure, a lot of this can be blamed on using AI studio to ask a small model a factual question. It's the raw LLM output of a highly compressed model, it's not meant to be everyday user facing like the default Gemini models, and doesn't have the same web search and fact checking behind the scenes. On the other hand, training a small model to hallucinate less would be a significant development. Perhaps with post-training fine-tuning, after getting a sense of what depth of factual knowledge the model has actually absorbed, adding a chunk of training samples with a question that goes beyond the model's fact knowledge limitations, and the model responding "Sorry, I'm a small language model and that question is out of my depth." I know we all hate refusals but surely there's room to improve them.
    - th0ma5 1y ago
      All of these techniques just push the problems around so far. And anything short of 100% accurate is a 100% failure in any single problematic instance.
  - kentm 1y ago
    A talk I went to made the point that LLMs don't sometimes hallucinate. They always hallucinate -- its what they're made to do. Usually those hallucinations align with reality in some way, but sometimes they don't. I always thought that was a correct and useful observation.
MyOutfitIsVague 1y ago
From her letter: > The consistent pattern of bias against conservative figures demonstrated by Google’s AI systems is even more alarming. Conservative leaders, candidates, and commentators are disproportionately targeted by false or disparaging content. That's a little rich given the current administration's relationship to the truth. The present power structure runs almost entirely on falsehoods and conspiracy theories.
- mikkupikku 1y ago
  It might be rich, but is it false? Are Google models more likely to defame conservatives than not? I think plausiblibly they might, through no fault of Google, if only because scandals involving conservatives might be statistically more likely.
  - th0ma5 1y ago
    A lot facts that people deem liberal or leftist or something are simply statistically consistent with the world literature as a whole and problematic for conservative ideals.
    - criddell 1y ago
      Back in 2006 at the White House Correspondent's Dinner, Stephen Colbert said "reality has a well known liberal bias" which I thought was a pretty funny line. https://en.wikipedia.org/wiki/Stephen_Colbert_at_the_2006_White_House_Correspondents%27_Dinner https://en.wikipedia.org/wiki/Stephen_Colbert_at_the_2006_Wh...
      - huhkerrf 1y ago
        Yeah, we're all aware. Midwits have been repeating it ever since.[0] 0: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=%22reality%20has%20a%20well%20known%20liberal%20bias%22&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
      - 1y ago
_jsmh 1y ago
"False or misleading answers from AI chatbots masquerading as facts still plague the industry and despite improvements there is no clear solution to the accuracy problem in sight." One potential solution to the accuracy problem is to turn facts into a marketplace. Make AIs deposit collateral for the facts they emit and have them lose the collateral and pay it to the user when it's found that statements they presented were false. AI would be standing behind its words by having something to lose, like humans. A facts marketplace would make facts easy to challenge and hard to get right. Working POC implementation of facts marketplace in my submissions.
- mikkupikku 1y ago
  I doubt that could ever work. It's trivial to get these models to output fabrications if that's what you want; just keep asking it for more details about a subject than it could reasonably have. This works because the models are absolutely terrible at saying "I don't know", and this might be a fundamental limitation of the tech. Then of course you have the mess of figuring out what the facts even are, there are many contested subjects our society cannot agree on, many of which don't lend themselves to scientific inquiry.
  - _jsmh 1y ago
    Thank you for sharing this opinion. AI != LLMs. A future version of better AI might become capable of saying "I don't know." The mess of figuring out what the facts are, in a way, already has deployed solutions. Journalists are a form of fact checkers. Big news companies with multi-million budgets pay journalists to check things. In effect there already is a market for fact-checking. Managing perceptions by twisting facts is something many have been accused of. When news companies are found to have spread lies they possibly face penalties. So if the big clunky enterprise version of a form of fact-checking is already in place and kind of working, why would something closer to the lean startup-y tail-end of the distribution of fact-checking not work? If a version of a facts marketplace is already working, why doubt it would ever work? X paying users for engagement is a new variation of this. Prediction markets like Polymarket is another. Polymarket teaming up with TruthSocial could be an interesting result. It's okay to have many contested subjects our society cannot agree on and which don't lend themselves to scientific inquiry. It's okay to say I don't know.
chimeracoder 1y ago
LLMs have serious problems with accuracy, so this story is entirety believable - we've all seen LLMs fabricate far more outlandish stuff. Unfortunately, it's also worth pointing out that neither Marsha Blackburn nor Robby Starbuck are reliable narrators historically; nor are they even impartial actors in this particular story. Blackburn has a long history of fighting to regulate Internet speech in order to force them to push ideological content (her words, not mine), so it's not surprising to see that this story originated as part of an unrelated lawsuit over First Amendment rights on the Internet and that Blackburn's response to it is to call for it all to be shut down until it can be regulated according to her partisan agenda (again, her words, not mine) - something which she has already pushed for via legislation that she has coauthored.
swivelmaster 1y ago
At some point we have to be willing to call out, at a societal level, that LLMs have been fundamentally oversold. The response to "It made defamatory facts up" of "You're using it wrong" is only going to fly for so long. Yes, I understand that this was not the intended use. But at some point if a consumer product can be abused so badly and is so easy to use outside of its intended purposes, it's a problem for the business to solve and not for the consumer.
- TZubiri 1y ago
  Maybe someone else actually made up the defamatory fact up, and it was just parroted. But fundamentally the reason ChatGPT became so popular as opposed to its incumbents like Google or Wikipedia, is that it dispensed with the idea of attributing quotes to sources. Even if 90% of the things it says can be attributed, it's by design that it can say novel stuff. The other side of the coin is that for things that are not novel, it attributes the quote to itself rather than sharing the credit with sources, which is what made the thing so popular in the first place, as if it were some kind of magic trick. These are obviously not fixable, but part of the design. I have the theory that the liabilities will be equivalent if not greater to the revenue recouped by OpenAI, but the liabilities will just take a lot longer to realize, considering not only the length of trials, but the length of case law and even new legislation to be created. In 10 years, Sama will be fighting to make the thing an NFP again and have the government bail it out of all the lawsuits that it will accrue. Maybe you can't just do things
- im3w1l 1y ago
  Businesses can't just wave a magic wand and make the models perfect. It's early days with many open questions. As these models are a net positive I think we should focus on mitigating the harms rather than some zero tolerance stance. We shouldn't allow the businesses to be neglectful, but I don't see evidence of that.
  - mindslight 1y ago
    > We shouldn't allow the businesses to be neglectful, but I don't see evidence of that. Calling it "AI", shoving it into many existing workflows as if it's competently answering questions, and generally treating it like an oracle IS being neglectful.
Razengan 1y ago
Just a day ago I asked Gemini to search for Airbnb rooms in an area and give me a summarized list. It told me it can't and I could do it myself. I told it again. Again it told me it can't, but here's how I could do it myself. I told it it sucks and that ChatGPT etc. can do it for me. Then it went and I don't know, scrapped Airbnb or used a previous search it must have had, to pull up rooms with an Airbnb link to each. … After using a bunch of products I now think a common option they all need to have is a toggle between "Monkey's Paw" mode: Do As I Say, vs a "Do What I Mean" mode. Basically where the user takes responsibility and where the AI does. If it can't do or isn't allowed to do something when in Monkey Paw mode then just stop with a single sentence. Don't go on a roundabout gaslighting trip.
- vrighter 1y ago
  "Knowing what they don't know" is one of the big unsolved problems in llm training. There's a finite number of facts. There's an infinite number of non-facts. If it could do what you're suggesting, it wouldn't need to do it in the first place.
renewiltord 1y ago
Placing a model behind a “Use `curl` after generating an API key using `gcloud auth login` and accepting the terms of service” is probably a good idea. Anything but the largest models equipped with search to ground generation is going to hallucinate at a high enough rate that a rando can see it. You need to gate away useful technology from the normies, usually. E.g. kickstarter used to have a problem where normies would think they were pre-ordering a finished product and so they had to pivot to being primarily a pre-order site. Anything that is actually experimental and has less than very high performance needs to be gated away from the normies.
- stormbeard 1y ago
  Google sat on this technology for years and didn't release their early chatbots to the public for this reason. The problem is that OpenAI opened Pandora's box and recklessly leaned into it.
AmbroseBierce 1y ago
[flagged]
- mikeholler 1y ago
  Perhaps we should be trying to make the machines perform correctly, instead of saying that creating fabrications is OK for anyone or anything.
  - AmbroseBierce 1y ago
    Perhaps we should make sure that the human sources are liable for making false allegations and therefore the likelihood of those fabrications existing in the first place is significantly reduced, so any machine -or any other entity- using public information available is more likely to be correct.
- dsr_ 1y ago
  I'm okay with condemning all of them. Bad behavior on one part doesn't excuse the bad behavior of the other.
- swivelmaster 1y ago
  Just because the media has failed doesn’t mean we should accept that kind of failure everywhere.
kmfrk 1y ago
One of the things that really has me worried at the moment are people asking chatbots who to vote for ahead of upcoming elections. Especially in parliamentary democracies where people already take political quizzes to make sense of all the parties and candidates on the ballot.
- black6 1y ago
  If you're asking a machine which human you should vote for, you probably shouldn't be voting.
  - mikkupikku 1y ago
    I think we don't do democracy because we think the masses are informed and make good decisions, but rather because it's the best system for ensuring peaceful transitions of power, thereby creating social stability which is conducive to encouraging investment in the future. So uninformed people participating isn't an unfortunate side effect, but rather the point: making everybody feel included in the decision making processes, to make people more likely to accept political change.
    - IAmBroom 1y ago
      Are you saying?... "I think we do democracy not because we think the masses are informed and make good decisions, but rather because it's the best system for ensuring peaceful transitions of power, thereby creating social stability which is conducive to encouraging investment in the future.
      - mikkupikku 1y ago
        Yes.
    - lesuorac 1y ago
      I think people argue this but I don't think its true. The lack of warlords leads to peaceful transitions. Trump can feel all he wants about the 2020 election but his sphere of influence was too small to take control. This isn't the case for all those power struggles when a monarch dies. Each Lord had their own militia they could mobilize to take control and leads to stuff like War of the Roses. We had this same issue going into the Civil War where the US army was mostly militias so it's pretty easy to grab the southern ones together and go fight the north. This isn't going to work so well post-1812 where a unified federal army exists. Of course, if you start selectively replacing generals with loyalists then you start creating a warlord.

84 points by croemer 1y ago

rchaud 1y ago

Terrifying to think that some techbro is out there right now concocting plans for an "AI background check" startup.

confounder 1y ago
Like this one? https://checkr.com/our-technology/ai-powered https://checkr.com/our-technology/ai-powered

croemer 1y ago

This is about Gemma, Google's open weights model. And specifically availability through AI studio. I don't think they'll make the weights unavailable.

aleatorianator 1y ago
[flagged]
- jqpabc123 1y ago
  This is about these tools being blatantly flawed and unreliable. In legal terms, marketing such a product is called "negligence" or "libel". Lots of software is flawed and unreliable but this is typically addressed in the terms of service. This may not be possible with AI because the "liability" can extend well beyond just the user.
  - flufluflufluffy 1y ago
    The purpose of the tool is for writing code, it is not for generating factual English sentences.
    - mikkupikku 1y ago
      I do think that might be the only thing they turn out to be any good at, and only then because software is relatively easy to test and iterate on. But does that mean writing code is what the models are "for"? They're marketed as being good for a lot more than coding.
    - jqpabc123 1y ago
      it is not for generating factual English sentences. Then the tool should not be doing it --- but it does. And therein is the legal liability.
      - flufluflufluffy 1y ago
        The tool did it because the person asked it to. They used the tool the wrong way. The knives are entering people’s guts. They should not be doing that. The knife companies should be liable for these stabbings.
        jqpabc123 1y ago
        The tool did it because the person asked it to. The tool did it because this is what it was designed and trained to do --- at great expense and effort --- but somewahat less than successfully. You can't have it both ways --- the tool can't be "intelligent" yet too stupid to understand it's own limitations. If people ask the wrong questions, the "intelligent" response would be, "Sorry, I can't do that". Maybe the problem here is that this "intelligent" tool is really as dumb as a rock. And it's only a matter of time until lawyers argue this point to a jury in court.
  - jwitthuhn 1y ago
    It is wrong to release something unreliable even while acknowledging it is unreliable? The product performs as advertised. If people want accurate information an LLM is the wrong tool for the job. From the Gemma 3 readme on huggingface: "Models generate responses based on information they learned from their training datasets, but they are not knowledge bases. They may generate incorrect or outdated factual statements."
    - jqpabc123 1y ago
      If people want accurate information an LLM is the wrong tool for the job. So these vendors spent lots of time and money training LLMs to answer questions that people should not ask --- but are allowed and encouraged to. Nonsensical and unrealistic. I expect the courts will agree and hold the vendors liable.
- notavalleyman 1y ago
  You wrote this comment under and article about Gemma, an open weights model which anyone can download and run at home. Here is more info and links to the models, so you can interrogate them about Senatorial scandals on your hardware at home. https://huggingface.co/blog/gemma https://huggingface.co/blog/gemma Your claim was so far from the truth of reality, that now, it's incumbent upon you to go back through the chain of faulty reasoning. You took it for granted that a conspiracy theory about suppressing information was true, when actually, the same Gemma model was already open-weighted by the same conspirators who you accuse of keeping Gemma out of regular peoples' reach

jqpabc123 1y ago

AI is going to be a lawyer's wet dream. Imagine the ads on TV: "Has AI lied about you? Your case could be worth millions. Call now!"

Handy-Man 1y ago
I mean even lawyers are getting sanctioned for using them without verifying their outputs lol. - https://www.msba.org/site/site/content/News-and-Publications/News/General-News/Massachusetts_Lawyer-Sanctioned_for_AI_Generated-Fictitious_Cases.aspx https://www.msba.org/site/site/content/News-and-Publications... - https://www.reuters.com/legal/government/judge-disqualifies-three-butler-snow-attorneys-case-over-ai-citations-2025-07-24/ https://www.reuters.com/legal/government/judge-disqualifies-... - https://calmatters.org/economy/technology/2025/09/chatgpt-lawyer-fine-ai-regulation/ https://calmatters.org/economy/technology/2025/09/chatgpt-la...
ghjv 1y ago
strange to talk about it in the future tense. it's here and yep, it's an object of fascination for the legal system
rchaud 1y ago
Unlike medicine, AI isn't regulated, so lawyers won't have anything to go after.
- advisedwang 1y ago
  I think the parent post imagines defamation cases will be worthwhile. I'm sure there will be some, but an AI simply lying in a query doesn't = damages worth suing over.
  - jqpabc123 1y ago
    It depends on who is the lie was about and what was said. Simple example: A prospective employer refuses to hire you because of some blatant falsehood generated by an LLM.
1y ago

hnuser123456 1y ago

There should probably be a little more effort towards making small models that don't just make things up when asked a factual question. All of us who have played with small models know there's just not as much room for factual info, they are middle schoolers who just write anything. Completely fabricated references are clearly an ongoing weakness, and easy to validate.

rewilder12 1y ago
LLMs by definition do not make facts. You will never be able to eliminate hallucinations. It's practically impossible. Big tech created a problem for themselves by allowing people to believe the things their products generate using LLMs are facts. We are only reaching the obvious conclusion of where this leads.
- hnuser123456 1y ago
  To be sure, a lot of this can be blamed on using AI studio to ask a small model a factual question. It's the raw LLM output of a highly compressed model, it's not meant to be everyday user facing like the default Gemini models, and doesn't have the same web search and fact checking behind the scenes. On the other hand, training a small model to hallucinate less would be a significant development. Perhaps with post-training fine-tuning, after getting a sense of what depth of factual knowledge the model has actually absorbed, adding a chunk of training samples with a question that goes beyond the model's fact knowledge limitations, and the model responding "Sorry, I'm a small language model and that question is out of my depth." I know we all hate refusals but surely there's room to improve them.
  - th0ma5 1y ago
    All of these techniques just push the problems around so far. And anything short of 100% accurate is a 100% failure in any single problematic instance.
- kentm 1y ago
  A talk I went to made the point that LLMs don't sometimes hallucinate. They always hallucinate -- its what they're made to do. Usually those hallucinations align with reality in some way, but sometimes they don't. I always thought that was a correct and useful observation.

MyOutfitIsVague 1y ago

From her letter: > The consistent pattern of bias against conservative figures demonstrated by Google’s AI systems is even more alarming. Conservative leaders, candidates, and commentators are disproportionately targeted by false or disparaging content. That's a little rich given the current administration's relationship to the truth. The present power structure runs almost entirely on falsehoods and conspiracy theories.

mikkupikku 1y ago
It might be rich, but is it false? Are Google models more likely to defame conservatives than not? I think plausiblibly they might, through no fault of Google, if only because scandals involving conservatives might be statistically more likely.
- th0ma5 1y ago
  A lot facts that people deem liberal or leftist or something are simply statistically consistent with the world literature as a whole and problematic for conservative ideals.
  - criddell 1y ago
    Back in 2006 at the White House Correspondent's Dinner, Stephen Colbert said "reality has a well known liberal bias" which I thought was a pretty funny line. https://en.wikipedia.org/wiki/Stephen_Colbert_at_the_2006_White_House_Correspondents%27_Dinner https://en.wikipedia.org/wiki/Stephen_Colbert_at_the_2006_Wh...
    - huhkerrf 1y ago
      Yeah, we're all aware. Midwits have been repeating it ever since.[0] 0: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=%22reality%20has%20a%20well%20known%20liberal%20bias%22&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
    - 1y ago

_jsmh 1y ago

"False or misleading answers from AI chatbots masquerading as facts still plague the industry and despite improvements there is no clear solution to the accuracy problem in sight." One potential solution to the accuracy problem is to turn facts into a marketplace. Make AIs deposit collateral for the facts they emit and have them lose the collateral and pay it to the user when it's found that statements they presented were false. AI would be standing behind its words by having something to lose, like humans. A facts marketplace would make facts easy to challenge and hard to get right. Working POC implementation of facts marketplace in my submissions.

mikkupikku 1y ago
I doubt that could ever work. It's trivial to get these models to output fabrications if that's what you want; just keep asking it for more details about a subject than it could reasonably have. This works because the models are absolutely terrible at saying "I don't know", and this might be a fundamental limitation of the tech. Then of course you have the mess of figuring out what the facts even are, there are many contested subjects our society cannot agree on, many of which don't lend themselves to scientific inquiry.
- _jsmh 1y ago
  Thank you for sharing this opinion. AI != LLMs. A future version of better AI might become capable of saying "I don't know." The mess of figuring out what the facts are, in a way, already has deployed solutions. Journalists are a form of fact checkers. Big news companies with multi-million budgets pay journalists to check things. In effect there already is a market for fact-checking. Managing perceptions by twisting facts is something many have been accused of. When news companies are found to have spread lies they possibly face penalties. So if the big clunky enterprise version of a form of fact-checking is already in place and kind of working, why would something closer to the lean startup-y tail-end of the distribution of fact-checking not work? If a version of a facts marketplace is already working, why doubt it would ever work? X paying users for engagement is a new variation of this. Prediction markets like Polymarket is another. Polymarket teaming up with TruthSocial could be an interesting result. It's okay to have many contested subjects our society cannot agree on and which don't lend themselves to scientific inquiry. It's okay to say I don't know.

chimeracoder 1y ago

LLMs have serious problems with accuracy, so this story is entirety believable - we've all seen LLMs fabricate far more outlandish stuff. Unfortunately, it's also worth pointing out that neither Marsha Blackburn nor Robby Starbuck are reliable narrators historically; nor are they even impartial actors in this particular story. Blackburn has a long history of fighting to regulate Internet speech in order to force them to push ideological content (her words, not mine), so it's not surprising to see that this story originated as part of an unrelated lawsuit over First Amendment rights on the Internet and that Blackburn's response to it is to call for it all to be shut down until it can be regulated according to her partisan agenda (again, her words, not mine) - something which she has already pushed for via legislation that she has coauthored.

swivelmaster 1y ago

At some point we have to be willing to call out, at a societal level, that LLMs have been fundamentally oversold. The response to "It made defamatory facts up" of "You're using it wrong" is only going to fly for so long. Yes, I understand that this was not the intended use. But at some point if a consumer product can be abused so badly and is so easy to use outside of its intended purposes, it's a problem for the business to solve and not for the consumer.

TZubiri 1y ago
Maybe someone else actually made up the defamatory fact up, and it was just parroted. But fundamentally the reason ChatGPT became so popular as opposed to its incumbents like Google or Wikipedia, is that it dispensed with the idea of attributing quotes to sources. Even if 90% of the things it says can be attributed, it's by design that it can say novel stuff. The other side of the coin is that for things that are not novel, it attributes the quote to itself rather than sharing the credit with sources, which is what made the thing so popular in the first place, as if it were some kind of magic trick. These are obviously not fixable, but part of the design. I have the theory that the liabilities will be equivalent if not greater to the revenue recouped by OpenAI, but the liabilities will just take a lot longer to realize, considering not only the length of trials, but the length of case law and even new legislation to be created. In 10 years, Sama will be fighting to make the thing an NFP again and have the government bail it out of all the lawsuits that it will accrue. Maybe you can't just do things
im3w1l 1y ago
Businesses can't just wave a magic wand and make the models perfect. It's early days with many open questions. As these models are a net positive I think we should focus on mitigating the harms rather than some zero tolerance stance. We shouldn't allow the businesses to be neglectful, but I don't see evidence of that.
- mindslight 1y ago
  > We shouldn't allow the businesses to be neglectful, but I don't see evidence of that. Calling it "AI", shoving it into many existing workflows as if it's competently answering questions, and generally treating it like an oracle IS being neglectful.

Razengan 1y ago

Just a day ago I asked Gemini to search for Airbnb rooms in an area and give me a summarized list. It told me it can't and I could do it myself. I told it again. Again it told me it can't, but here's how I could do it myself. I told it it sucks and that ChatGPT etc. can do it for me. Then it went and I don't know, scrapped Airbnb or used a previous search it must have had, to pull up rooms with an Airbnb link to each. … After using a bunch of products I now think a common option they all need to have is a toggle between "Monkey's Paw" mode: Do As I Say, vs a "Do What I Mean" mode. Basically where the user takes responsibility and where the AI does. If it can't do or isn't allowed to do something when in Monkey Paw mode then just stop with a single sentence. Don't go on a roundabout gaslighting trip.

vrighter 1y ago
"Knowing what they don't know" is one of the big unsolved problems in llm training. There's a finite number of facts. There's an infinite number of non-facts. If it could do what you're suggesting, it wouldn't need to do it in the first place.

renewiltord 1y ago

Placing a model behind a “Use `curl` after generating an API key using `gcloud auth login` and accepting the terms of service” is probably a good idea. Anything but the largest models equipped with search to ground generation is going to hallucinate at a high enough rate that a rando can see it. You need to gate away useful technology from the normies, usually. E.g. kickstarter used to have a problem where normies would think they were pre-ordering a finished product and so they had to pivot to being primarily a pre-order site. Anything that is actually experimental and has less than very high performance needs to be gated away from the normies.

stormbeard 1y ago
Google sat on this technology for years and didn't release their early chatbots to the public for this reason. The problem is that OpenAI opened Pandora's box and recklessly leaned into it.

AmbroseBierce 1y ago

[flagged]

mikeholler 1y ago
Perhaps we should be trying to make the machines perform correctly, instead of saying that creating fabrications is OK for anyone or anything.
- AmbroseBierce 1y ago
  Perhaps we should make sure that the human sources are liable for making false allegations and therefore the likelihood of those fabrications existing in the first place is significantly reduced, so any machine -or any other entity- using public information available is more likely to be correct.
dsr_ 1y ago
I'm okay with condemning all of them. Bad behavior on one part doesn't excuse the bad behavior of the other.
swivelmaster 1y ago
Just because the media has failed doesn’t mean we should accept that kind of failure everywhere.

kmfrk 1y ago

One of the things that really has me worried at the moment are people asking chatbots who to vote for ahead of upcoming elections. Especially in parliamentary democracies where people already take political quizzes to make sense of all the parties and candidates on the ballot.

black6 1y ago
If you're asking a machine which human you should vote for, you probably shouldn't be voting.
- mikkupikku 1y ago
  I think we don't do democracy because we think the masses are informed and make good decisions, but rather because it's the best system for ensuring peaceful transitions of power, thereby creating social stability which is conducive to encouraging investment in the future. So uninformed people participating isn't an unfortunate side effect, but rather the point: making everybody feel included in the decision making processes, to make people more likely to accept political change.
  - IAmBroom 1y ago
    Are you saying?... "I think we do democracy not because we think the masses are informed and make good decisions, but rather because it's the best system for ensuring peaceful transitions of power, thereby creating social stability which is conducive to encouraging investment in the future.
    - mikkupikku 1y ago
      Yes.
  - lesuorac 1y ago
    I think people argue this but I don't think its true. The lack of warlords leads to peaceful transitions. Trump can feel all he wants about the 2020 election but his sphere of influence was too small to take control. This isn't the case for all those power struggles when a monarch dies. Each Lord had their own militia they could mobilize to take control and leads to stuff like War of the Roses. We had this same issue going into the Civil War where the US army was mostly militias so it's pretty easy to grab the southern ones together and go fight the north. This isn't going to work so well post-1812 where a unified federal army exists. Of course, if you start selectively replacing generals with loyalists then you start creating a warlord.