4 ms·
If you want to skip all the culture wars stuff, the interesting bit here is the alleged description of how Gemini's image generation prompting is preprocessed:
by ThrowawayR2 3y ago
If you want to skip all the culture wars stuff, the interesting bit here is the alleged description of how Gemini's image generation prompting is preprocessed:
"Roughly, the “safety” architecture designed around image generation (slightly different than text) looks like this: a user makes a request for an image in the chat interface, which Gemini — once it realizes it’s being asked for a picture — sends on to a smaller LLM that exists specifically for rewriting prompts in keeping with the company’s thorough “diversity” mandates. This smaller LLM is trained with LoRA on synthetic data generated by another (third) LLM that uses Google’s full, pages-long diversity “preamble.” The second LLM then rephrases the question (say, “show me an auto mechanic” becomes “show me an Asian auto mechanic in overalls laughing, an African American female auto mechanic holding a wrench, a Native American auto mechanic with a hard hat” etc.), and sends it on to the diffusion model. The diffusion model checks to make sure the prompts don’t violate standard safety policy (things like self-harm, anything with children, images of real people), generates the images, checks the images again for violations of safety policy, and returns them to the user.
“Three entire models all kind of designed for adding diversity,” I asked one person close to the safety architecture. “It seems like that — diversity — is a huge, maybe even central part of the product. Like, in a way it is the product?”
“Yes,” he said, “we spend probably half of our engineering hours on this.”"
If true (admittedly a big if), it torpedoes the suggestion made by many that Gemini's misbehavior was simply accidental teething problems that were missed in testing.
- slater 3y agowhat i don't get is why this always gets missed in testing. How did they NOT have at least one turbo-edgelord on payroll just testing it with increasingly offensive 4chan-style prompts, and flagging the results?
- ThrowawayR2 3y agoThe article claims that they did know about it: "First, according to people close to the project, the team responsible for Gemini was not only warned about its “overdiversification” problem before launch (the technical term for erasing white people from human history), but understood the nebulous DEI architecture — separate from causing offense — dramatically eroded the quality of even its most benign search results."
- scarmig 3y agoThe sense at Google is that it's dangerous to rock the boat too much when it comes to certain pieties. It would have been a good idea to have someone doing that, and Google certainly has the money to, and plenty of people thought it was a good idea. But suggesting that or taking on that job would have been at best neutral for your perf packet (if you did nothing) or outright negative (if you actually did the job and raised red flags about Gemini's results). Someone would complain they were offended or felt threatened by your work, and then you're out the door. In fact, Google does have a diversity problem, but it's the opposite of what the corporate apparatus thinks it is.
- dotcoma 3y ago‘Diversity’ at many companies: different skin colors, but same line of thought, or lack thereof.
- nebula8804 3y agoGoogle products haven't been the same since Elon and Brin had a falling out. They lost their best edge case tester. :P
- alemanek 3y agoI am thinking this is almost certainly malicious compliance. I mean diverse Nazi imagery is just too perfect of a miss. They managed to offend literally everyone with that even the actual Nazis.
- dotcoma 3y agoThey offended the victims of the Nazis (who cares if they offended the Nazis).
- philwelch 3y agoIt’s right there in the title: “Google’s Culture of Fear”. Being that turbo-edgelord is a career limiting move in a way that just sitting back and letting the Gemini launch play out isn’t.
- mike_hearn 3y agoNone of the prompts being demoed were offensive, only the outputs, so that wouldn't have helped. As to why they didn't test the outputs of prompts like "images of German soldiers in 1943", that's probably due to the belief in symbolic determinism that underlies their behavior. The hard left is a big fan of the belief that reality comes to reflect its symbolic representation, not that symbolic representations reflect reality. In such belief systems depicting something symbolically can make it come true. An example of this in earlier eras was the Sapir-Whorf hypothesis. This shows up as changing the race/gender/sexuality of people in art forms like job ads or on screen in an attempt to change job demographics in reality, but it can also show up in other ways. For example if you believe in symbolic determinism, then drawing Nazis is dangerous because doing that actually creates Nazis in reality (or in weaker forms of the belief, increase the probability of people's beliefs becoming more Nazi-like). Remember that 50% of their effort went into stopping people drawing pictures of white men even when that would be highly realistic, because they believe that depicting us directly creates problems in the real world. For such people testing prompts expected to produce only white men would not only be risky in a careerist "what would HR say" sense, but it's perceived as risky in its own right! The very act of testing could make the world worse. That's the whole reason they're trying to stop users doing it in the first place! It's also why they constantly bring up physical safety even when talking about symbols. So the Gemini team have ignored the actual feedback they got and decided that the error was making groups of bad people diverse. They're going to fix Nazis and Vikings to either refuse to depict them at all, or to show them as all white men (because those groups are bad), but ignore all the cases where groups of good people get incorrectly diversified. And of course they won't do anything about the text. This is all predictable given their beliefs about the nature of reality.
- munksbeer 3y agoBecause I imagine that ahering to the corporate diversity policies is a huge deal in promotion and reward, so no-one is going to want to put their head above the parapet.
- bongoman37 3y ago[dead]
- llm_trw 3y agoBecause everyone like that's been fired from google in 2008.
- patrick451 3y agoNobody serious actually believed it was an accident to begin with.
- tim333 3y agoI think the multi ethnic German nazis were not really what they were intending.