5 ms·

It's been two weeks since Google turned off the generation of images with people. Still no word on when it's coming back. This tells me it wasn't just system

It's been two weeks since Google turned off the generation of images with people. Still no word on when it's coming back. This tells me it wasn't just system prompts and prompt injection, it was done via RLHF and it's going to take awhile to un-gimp the model. I just tried "generate an image of a man camping in the woods" and it came back with "Sure, here is an image of a man camping in the woods" followed by a 10-15 second pause followed by "We are working to improve Gemini’s ability to generate images of people. We expect this feature to return soon and will notify you in release updates when it does." This would seem to indicate they're letting it generate an image and then running some sort of human detector on the output instead of filtering the prompt on the front end.

mousetree 3y ago
Given the degree of the controversy, I think even if it was just simple fixes in system prompts, they would want to take a few weeks of testing and red teaming before re-releasing.
- FirmwareBurner 3y ago
  >they would want to take a few weeks of testing and red teaming before re-releasing. One wonders why they haven't done this kind of testing before going public with such an important feature of such an important new flagship product. It's not like the prompts were some super niche stuff that required a lot of imagination to conjure. And they could have blocked the people image generation feature from the start if they saw it malfunctioning like that in tests and didn't have enough time to fix it on such a short notice before launch. I find it hard to believe nobody internal at Google bothered to play with Gemini to generate people for shits and giggles and saw the hilarious wrong outputs and raise the alarm. Or they did ran tests, saw the issues but they decided internally that there's nothing wrong with those generated outputs, which is even worse. Either way, whichever reason it was, why would you use or even want to pay for a Google product if this is the level of QC and broken functionality you can expect from Google products? To further my point. I use ChatGPT to translate stuff from English to German a lot and now I used it to write a legal complaint in German and I tried getting Gemini to translate it as well to compare it to ChatGPT, and instead of doing that, it just said it won't do it because it can't offer legal advice. FFS, I didn't ask you for legal advice, I already did my research on that topic, I just asked you to translate verbatim from English to German. Gemini's "safety" guardrails make it beyond useless at this point that I wouldn't even use it even for free. How does Google plant to make money from it? Or is it gonna be another side-project they'll cancel in 2-5 years.
  - londons_explore 3y ago
    > I find it hard to believe nobody internal at Google bothered to play with Gemini to generate people for shits and giggles and saw the hilarious wrong outputs and raise the alarm. The alarm was drowned out by the "Shit OpenAI is miles ahead in this game, we need to launch ASAP to try to close the gap" alarm.
    - FirmwareBurner 3y ago
      Sure, but the moment you as a big company go live with something you have to automatically assume people will fuck with it. Surely someone out of Google's 130k staff who wasn't overworked with getting Gemini ready for launch had the chance to fuck with it and see where it fall short.
notaustinpowers 3y ago
Per The Verge, it looks like the image generation model isn't even Gemini, but some older image generation model they found on a shelf in their basement and just made Gemini make a call to it for any image requests. https://www.theverge.com/2024/3/4/24090879/some-details-on-googles-gemini-scandal https://www.theverge.com/2024/3/4/24090879/some-details-on-g...
afp14 3y ago
So here's my hypothesis as to how it works now - and yes I believe this "problematic bias" is baked into the core model - I think they have a "dumb" simple "is this a person" detector stitched onto the end of the pipeline. If it detects a person, drop the generated images. This suggests a jailbreak, if you can fool the "dumb" person detector (likely shallow vs deep learning) with your deep learning generator, the images pass through. I had some success last week with "upside down" or "standing on their head" though it may have been patched. You could consider occlusion "wearing a mask" etc. Basically find phrasing that would fool a naive person/face detector in the corresponding image. As for the model, I truly believe it to be paradoxically pro-white racist. Imagine any stereotype for any race. You say "generate <stereotype> of a <historically white character/role>" and the model swaps out the white race for the stereotyped race, seemingly embracing racist stereotypes. On the other hand, for white stereotypes, they are much harder to produce since the model is hesitant to render white folks.

mousetree 3y ago

Given the degree of the controversy, I think even if it was just simple fixes in system prompts, they would want to take a few weeks of testing and red teaming before re-releasing.

FirmwareBurner 3y ago
>they would want to take a few weeks of testing and red teaming before re-releasing. One wonders why they haven't done this kind of testing before going public with such an important feature of such an important new flagship product. It's not like the prompts were some super niche stuff that required a lot of imagination to conjure. And they could have blocked the people image generation feature from the start if they saw it malfunctioning like that in tests and didn't have enough time to fix it on such a short notice before launch. I find it hard to believe nobody internal at Google bothered to play with Gemini to generate people for shits and giggles and saw the hilarious wrong outputs and raise the alarm. Or they did ran tests, saw the issues but they decided internally that there's nothing wrong with those generated outputs, which is even worse. Either way, whichever reason it was, why would you use or even want to pay for a Google product if this is the level of QC and broken functionality you can expect from Google products? To further my point. I use ChatGPT to translate stuff from English to German a lot and now I used it to write a legal complaint in German and I tried getting Gemini to translate it as well to compare it to ChatGPT, and instead of doing that, it just said it won't do it because it can't offer legal advice. FFS, I didn't ask you for legal advice, I already did my research on that topic, I just asked you to translate verbatim from English to German. Gemini's "safety" guardrails make it beyond useless at this point that I wouldn't even use it even for free. How does Google plant to make money from it? Or is it gonna be another side-project they'll cancel in 2-5 years.
- londons_explore 3y ago
  > I find it hard to believe nobody internal at Google bothered to play with Gemini to generate people for shits and giggles and saw the hilarious wrong outputs and raise the alarm. The alarm was drowned out by the "Shit OpenAI is miles ahead in this game, we need to launch ASAP to try to close the gap" alarm.
  - FirmwareBurner 3y ago
    Sure, but the moment you as a big company go live with something you have to automatically assume people will fuck with it. Surely someone out of Google's 130k staff who wasn't overworked with getting Gemini ready for launch had the chance to fuck with it and see where it fall short.

notaustinpowers 3y ago

Per The Verge, it looks like the image generation model isn't even Gemini, but some older image generation model they found on a shelf in their basement and just made Gemini make a call to it for any image requests. https://www.theverge.com/2024/3/4/24090879/some-details-on-googles-gemini-scandal https://www.theverge.com/2024/3/4/24090879/some-details-on-g...

afp14 3y ago

So here's my hypothesis as to how it works now - and yes I believe this "problematic bias" is baked into the core model - I think they have a "dumb" simple "is this a person" detector stitched onto the end of the pipeline. If it detects a person, drop the generated images. This suggests a jailbreak, if you can fool the "dumb" person detector (likely shallow vs deep learning) with your deep learning generator, the images pass through. I had some success last week with "upside down" or "standing on their head" though it may have been patched. You could consider occlusion "wearing a mask" etc. Basically find phrasing that would fool a naive person/face detector in the corresponding image. As for the model, I truly believe it to be paradoxically pro-white racist. Imagine any stereotype for any race. You say "generate <stereotype> of a <historically white character/role>" and the model swaps out the white race for the stereotyped race, seemingly embracing racist stereotypes. On the other hand, for white stereotypes, they are much harder to produce since the model is hesitant to render white folks.