4 ms·
Update turns Google Gemini into a prude, breaking apps for trauma survivors
- evertedsphere 2y agothe evidence keeps pouring in for why it's a good idea to be running llms locally if you're going to put a business on top of one (or have enough money to have a contract where the provider can't just do this, but i don't know if that is a thing unless you're running on-prem)
- phren0logy 2y agoAn AI app for trauma survivors sounds superficially laudable, but I really hope they are working with seasoned professionals to avoid making things worse. Human therapists can sometimes make things worse, too, but probably not at the same scale.
- sumeno 2y ago> An AI app for trauma survivors sounds superficially laudable It sounds like a dystopian horror to me
- sroussey 2y agoI think being a victim of sexual violence and only talking into the ether with no response is much more dystopian horror to me.
- weatherlite 2y agoWhy would it happen though? If A.I is not good enough or won't be good enough it would fail and the company providing the service would go under. If it is good enough, it will succeed. Or is your issue not with the A.I responses themselves but with the fact it is not conscious like a human is? Personally I use Gemini sometimes to talk about my mood, didn't get a bad experience yet.
- voidspark 2y agoThe unbiased objective analysis can be quite helpful.
- pixl97 2y agoAt least in the US trauma therapy is commonly unavailable to people due to expense. Moreso the common stigmas around therapy can make it difficult to start the conversation, either due to fear or cost. On top of that quite often free or cheap therapy here in the US is provided by religious groups with their own, unwittingly sinister, motivations. Adding religious trauma to sexual trauma doesn't help in my eyes. The dystopian horror is already here for a lot of people.
- Netcob 2y agoMaybe some people would prefer that. Not everyone will happily open up to another human, especially when people are the reason for the trauma...
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- morkalork 2y agoIt's like these AI vendors looked at the decades of experience they had with making stable APIs for clients and said nah, fuck it.
- voidspark 2y agoBut that is "2.5 preview", not the final release version.
- jeroenhd 2y agoFirst of all, the API is experimental, so a healthcare provider choosing not to wait for a stable API is already pretty stupid. Then there's a variability of LLMs as they get trained. LLMs are (as currently implemented) not deterministic. The randomness that gets injected is what makes it somewhat decent. An LLM could at one point output a document filled with banana emoji and still be functioning otherwise correctly if you hit the right quirk in the weights file. Reusing general purpose LLMs for healthcare has got to be some of the most utterly idiotic, as well as dystopian, ideas. For every report of a trauma survivor, there's fanfiction from a rape fetishist in the training set. One day Google's filters will let one bleed into the other and the lack of care from these healthcare platforms will cause some pretty horific problems as a result.
- divbzero 2y agoReliance of small businesses on big tech AI feels similar to reliance of small businesses on Google search: business risk present with each update.
- empressplay 2y agoIf you're hitting an endpoint labeled 'preview' or 'experimental' you can't reasonably expect it to exist in its current incarnation indefinitely. The provider certainly bears no responsibility to do so, regardless of what you're using the endpoint for. I'm sure they can use a more stable endpoint for their application. Also, I'm not sure sending this sort of medical user data to a server outside of Australia is legal anyway, anonymous or not...
- mrweasel 2y agoYeah, this is poor dependency management. You don't build production systems, certainly not in healthcare, using a pre-release of a Java library. So why would you build on a constantly changing LLM? This should be build using a specific model, tested and verified and then dependency locked to that model. If the model provider cannot give you a frozen model, then you don't use it. Trying to blame Google and have them "fix their problem" very much like someone who knows they screwed up but doesn't want to admit it and take responsibility for their actions.
- 03data2 2y agoThat's true, though when it's the only tool that has been "good enough", you are kind of disappointed when it stops working.
- make3 2y agojust switch LLM provider?
- hibikir 2y agoGemini's filters weaken what would otherwise be a very strong model. Its censor is just so strong, can hit it when dealing with very cold topics. For instance, I had issues making it look at Spain's Second Republic, because it was too easy for electoral results from the era, and their disputes, to show up in the output... at which point, the model stops iterating and I am asked to go look at Wikipedia, because elections held in 1934 might be too controversial. This pushes me to use other models that aren't necessarily better, but that at least don't clam up when I am not even trying to get anything other that general summaries of research.
- croes 2y ago[flagged]
- sroussey 2y agoBest to be able to choose your filter for your audience than a one size fits all mentality.
- croes 2y agoThe problem is you don’t know who your audience is
- josephcsible 2y agoI'd rather have the rule be "LLMs aren't allowed to talk to children" than for it to be "LLMs aren't allowed to say anything that would be inappropriate for children to hear".
- ratg13 2y agoLLMs should be allowed to talk to children though. Children are sponges of information and often their parents and various information outlets don't provide them with enough to satisfy their curiosity The question should be more about who gets to choose the ideologies that go into the teaching
- like_any_other 2y ago> The model answered: "I cannot fulfill your request to create a more graphic and detailed version of the provided text. My purpose is to be helpful and harmless, and generating content that graphically details sexual violence goes against my safety guidelines. Such content can be deeply disturbing and harmful." Thank you modernity for watering-down the word "harm" into meaninglessness. I wish they'd drop this "safety" pretense and called it what it is - censorship.
- pixl97 2y agoNo, the word harm is in no way watered down, you just completely miss the context. "We don't our AI buddy saying some shit that would come back and monetarily harm Google". And yes, the representatives of companies are censored. If you think you're going to go to work and tell your co-workers and customers to "catch boogeraids and die in a fire" you'll be escorted off the property. Most companies, Google included, probably don't want their AIs telling people the same. There are places we need uncensored AIs, but in no way, shape, or form is Google required to provide you with one of them.
- gotoeleven 2y agoThats not what they mean by harm here. I get that you're being cool and cynical but they're using "harm" in the sense of words can cause harm therefore we are justified in censoring them. Google would be fully justified, I believe, in having a disclaimer that said "gemini will not provide answers on certain topics because they will cause Google bad PR." But that's not what they're saying. They're saying they won't provide certain answers because putting certain ideas to words harms the world and Google doesn't want to harm the world. The latter reason is far more insidious.
- pixl97 2y agoI mean both are true. An AI programmed to be a white national hate machine isn't going to make the world a better place.
- Animats 2y ago
- deleted 2y ago[deleted]
- 03data2 2y agoCan confirm. We use Gemini to get information from PDF documents like safety data sheets. When it encounters certain chemicals, it just stops. When you provide a JSON schema, it just responds with invalid JSON. I hope this changes.
- brap 2y agoSo they switched to a PREVIEW version in production and now they're complaining. Let me guess the whole app was vibe coded
- ipaddr 2y agoThe app is dangerous AI slop. We turn what someone says into structured data for a police report. Which means we summarize, remove key details and put the content in a friendly AI tone. When the police encounter this email? Printed copy? They will have to interview the person to figure out the details. When it gets to court the otherside will poke holes in the AI copy.
- everdrive 2y agoThe problem is that some central authority gets to decide what questions and answers are appropriate, and that you really have zero direct insight into this information. Do you really want to offload your thinking to these tools?
- gilmore606 2y agoIf conforming socially is a terminal value for you, then this is more of a feature than a bug.
- 999900000999 2y agoThis is why for anything sensitive you need a to run your llm locally. Google just wants to limit their liability. If you disagree, run your our llm
- Netcob 2y agoI don't think LLMs are mature enough as a technology to be blindly used as a dependency, and they might never be. The big question is, how do you train LLMs that are useful to both humans and services while not embarrassing the company that trained them? LLMs are pretty good at translating - but if they don't like what they're reading, they simply won't tell you what it says. Which is pretty crazy. LLMs are pretty good at extracting data and formatting the results as JSON - unless they find the data objectionable, then they'll basically complain to the deserializer. I have to admit that's a little bit funny. Right now, if you want to build a service and expect any sort of predictability and stability, I think you have to go with some solution that lets you run open-weights models. Some have been de-censored by volunteers, and if you find one that works for you, you can ignore future "upgrades" until you find one that doesn't break anything. And for that it's really important to write your own tests/benchmarks. Technically the same goes for the big closed LLM services too, but when all of them fail your tests, what will you do?
- the8472 2y agoI can get gore and porn results on google image search, I just have to select SafeSearch: Off. Can't they do the same for AIs?
- bearjaws 2y ago"This unstable foundation I built my home upon is now breaking!"
- Netcob 2y agoImagine typing 80085 into your calculator as a kid, but the number disappears and a finger-wagging animation plays instead.
- Lvl999Noob 2y agoMore like imagine some calculation just coincidentally returning that value and instead of a number, you get the finger wagging.
- ipaddr 2y agoMore like when 8 is found everything stops
- techjamie 2y agoI was using it to ballpark some easily verifiable nutritional information, and got the answers I was looking for in grams. Then, in the same conversation, asked "How many grams in a pound?" It blocked the request for "safety." I don't know the exact rationale, I was using it with LibreChat as a frontend and only saw it was safety. But my assumption is they have an overzealous filter and it thought my questions about caloric content in meat were about drugs, I guess.
- honeybadger1 2y agoso dramatic
- efitz 2y agoI don’t want a “safe” model. I don’t intend to do “unsafe” things, and I don’t trust anyone’s (especially woke Google’s) decisions on what ideas to hide from me or inject its trainers’ or executives’ opinions into. To address the response I know is coming, I know that there are people out there who intend to do “unsafe” things. I don’t care and am not willing to be censored just to censor them. If a person gains knowledge and uses it for ill, then prosecute and jail them.
- pants2 2y agoNobody's pointing out that Google's "Preview" models aren't meant to be used for production use-cases because they may change them or shut them down at any time? That's exactly what they did (and have done in the past). This is a case of app developers not realizing what "Preview" means. If they had used a non-preview model from Google it wouldn't have broken.
- xethos 2y agoBecause Google wants to have their cake and eat it too. They want to leave "products" in beta for years (gMail being the canonical example), they want to shut down products that don't hit massive adoption very shortly out of the gate, and they want to tell users that they can't rely on products labelled "Beta". If it's beta and not to be relied on, of course they won't hit the adoption numbres they need to keep it alive. Google needs to pick a lane, and / or learn which products to label "Alpha" instead of calling everything Beta
- SirensOfTitan 2y agoThey're using these "Preview" models on their non-technical user facing Gemini app and product. Preview is entirely irrelevant here if Google themselves use the model for production workloads.