5 ms·

“AI products must be tested rigorously before they go to market to ensure they do not have the capability to generate this material,” Not possible.

by chrisjj 10mo ago

SpicyLemonZest 10mo ago
It's extremely possible! As the source article notes, the Grok developers specifically chose to make their AI more permissive of sexual content than their competitors, which won't produce such images. This isn't a scenario where someone developed a complex jailbreak to circumvent Grok's built-in protections.
ben_w 10mo ago
> Not possible. To which governments, courts, and populations likely respond "We don't care if you can't go to market. We don't want models that do this. Solve it or don't offer your services here." Also… I think they probably could solve this. AI image analysis is a thing. AI that estimate age from an image has been a thing for ages. It's not like the idea of throwing the entire internet worth of images at a training sessions just to make a single "allowed/forbidden" filter is even ridiculous compared to the scale of all the other things going on right now.
- ls612 10mo ago
  These models generate probably a billion images a day. If getting it wrong for even one of those images is enough to get the entire model banned then it probably isn't possible and this de facto outlaws all image models. That may precisely be the point of this tbh.
  - lokar 10mo ago
    If they can't prevent child porn, then it should be banned.
    - ls612 10mo ago
      Should photoshop be outlawed? What about MS Paint? Both of them I’m pretty sure are capable of creating this stuff. Also, lets test your commitment to consistency on this matter. In most jurisdictions possession and creation of CSAM is a strict liability crime, so do you support prosecuting whatever journalist demonstrated this capability to the maximum extent of the law? Or are you only in favor of protecting children when it happens to advance other priorities of yours?
      - lokar 10mo ago
        Photoshop is fine, running a business where you produce CSAM for people with photoshop is not. And this has been very clear for a while now. I did not see the details of what happened, but if someone did in fact take a photo of a real child they had no connection to and caused the images to be created, then yes, they should be investigated, and if the prosecutor thinks they can get a conviction they should be charged. That is just what the law says today (AIUI), and is consistent with how it has been applied.
        ls612 10mo ago
        Somehow I doubt the prosecutor will apply the same standard to the other image generation models, which I bet (obviously without evidence given the nature of this discussion) can be convinced by a motivated adversary to do the same thing at least once. But alas, selective prosecution is the foundation of political power in the west and pointing that out gets you nothing but downvotes. patio11 once put it that pointing out how power is exercised is the first thing that those who wield power prohibit when they gain it.
        lokar 10mo ago
        You often see (appropriately, IMO) a certain amount of discretion wrt prosecution when things are changing quickly. I doubt anyone will go to jail over this. What (I think) should happen is state or federal law enforcement need to make it very clear to Xai (and the others) that this is unacceptable, and that if it keep happening, and you are not showing that you are fixing it (even if that means some degradation in the capability of the system/service), then you will be charged. One of the strengths of the western legal system that I think is under appreciated by people here is that it is subject to interpretation. Law is not Code. This makes it flexible to deal with new situations, and this is (IME) always accompanied by at least a small amount of discretion in enforcement. And in the end, the laws and how they are interpreted and enforced are subject to democratic forces.
  - nl 10mo ago
    Even the OP's quote made it clear this isn't the case. Companies need to show they rigorously tested that the model doesn't do this. It's like cyber insurance requirements - for better or worse, you need to show that you have been audited, not prove you are actually safe.
  - ben_w 10mo ago
    > These models generate probably a billion images a day. Collectively, probably more. Grok? Not unless you count each frame of a video, I think. > If getting it wrong for even one of those images is enough to get the entire model banned then it probably isn't possible and this de facto outlaws all image models. If the threshold is one in a billion… well, the risk is for adversarial outcomes, so you can't just toss a billion attempts at it and see what pops out, but a billion images, if it's anything like Stable Diffusion you can stop early, and my experiments with SD suggested the energy cost even for a full generation is only $0.0001/image*, so a billion is merely $100k. Given the current limits of GenAI tools, simply not including unclothed or scantily clad people in the training set would prevent this. I mean, I guess you could leave topless bodybuilders in there, then all these pics would look like Arnold Schwarzenegger, almost everyone would laugh and not care. > That may precisely be the point of this tbh. Perhaps. But I don't think we need that excuse if this was the goal, and I am not convinced this is the goal in the EU for other reasons besides. * https://benwheatley.github.io/blog/2022/10/09-19.33.04.html https://benwheatley.github.io/blog/2022/10/09-19.33.04.html
- krapp 10mo ago
  >To which governments, courts, and populations likely respond "We don't care if you can't go to market. We don't want models that do this. Solve it or don't offer your services here." No, they likely won't. AI has become far too big to fail at this point. So much money has been invested in it that speculation on AI alone is holding back a global economic collapse. Governments and companies have invested in AI so deeply that all failure modes have become existential. If models can't be contained, controlled or properly regulated then they simply won't be contained, controlled or properly regulated. We'll attempt it, of course, but the limits of what the law deems acceptable will be entirely defined by what is necessary for AI to succeed, because at this point it must. There's no turning back.
  - ben_w 10mo ago
    > No, they likely won't. AI has become far too big to fail at this point. So much money has been invested in it that speculation on AI alone is holding back a global economic collapse. Governments and companies have invested in AI so deeply that all failure modes have become existential. Not in Europe it hasn't, and definitely not for specifically image generation, where it seems to be filling the same role as clipart, stock photos, and style transfer that can be done in other ways. Image editing is the latest hotness in GenAI image models, but knowledge of this doesn't seem to have percolated very far around the economy, only with weird toys like this one currently causing drama. > If models can't be contained, controlled or properly regulated then they simply won't be contained, controlled or properly regulated. I wish I could've shown this kind of message to people 3.5 years ago, or even 2 years ago, saying that AI will never take over because we can always just switch it off. Mind you, for 2 years ago I did, and they still didn't like it.
    - pureagave 10mo ago
      I'm sorry to tell you this, but the EU has already been lost.
      - wolvoleo 10mo ago
        Because we're not on the forefront of AI development? It also means we have less to lose when the bubble blows. I'm quite happy with the policies here. And we will become more independent from US tech. It'll just take time.
BigTTYGothGF 10mo ago
Then maybe they shouldn't go to market.
- deleted 10mo ago
  [deleted]
- pureagave 10mo ago
  AI is a nation defense issue. No nation has the luxury to stop their AI companies without the risk of losing national sovereignty.
  - belter 10mo ago
    So child porn is now a national security issue?
  - squigz 10mo ago
    Lumping image gen models, LLMs, and other forms of recent machine learning altogether and dressing it up in the "National Defence" ribbon doesn't seem like a great idea. I don't think the ability for citizens to make deep fake porn of whoever they want is the same as a country not investing in practical defensive applications of AI.
  - dragonwriter 10mo ago
    > AI is a nation defense issue. AI image editors attached to social media networks with a design that allows producing AI edits (including, but not limited to, nonconsensual intimate images and child pornography) of other user’s media without consent are not a national defense issue, and, even to the extent that AI arguably is a national defense issue, those particular applications can be curtailed entirely by a nation without any adverse impact on national defense. You can distort any issue by zooming out to orbital level and ignoring the salient details.
  - UncleMeat 10mo ago
    "We have to make the revenge porn machine for national defense" is the sort of thing that makes people light bay area tech busses on fire.
lokar 10mo ago
Then your business can fairly be ruled illegal. You don't have the right to act in violation of the law merely because it's the only way to make a buck.
- kelseyfrog 10mo ago
  In practice, once a business reaches a size threshold, the law is creatively decided to preserve its existence rather than terminate it. Legality is a function of economics.
  - lokar 10mo ago
    Until people have had enough and push back And if you want to change the law to allow the business, go for it. But until then, we must follow the law.
  - ben_w 10mo ago
    > Legality is a function of economics. Sometimes it is. Sometimes "democracy" isn't just a buzzword. X.com has been blocked by poorer nations than France (specifically, Brazil) for not following local law.
belter 10mo ago
Possible or not, what about starting by criminal investigation, to force disclosure, and find out if Musk company had child porn in the training data?
- fragmede 10mo ago
  It probably doesn't have pictures of fishes driving cybertrucks, but it's able to generate those, so I doubt there'd need to be CSAM in the database, but maybe I don't know how these things really work.
- belter 10mo ago
  AI generates child porn, HN downvotes a proposal for an investigation...
dragonwriter 10mo ago
> “AI products must be tested rigorously before they go to market to ensure they do not have the capability to generate this material,” > Not possible. Note that the description of the accusation earlier in the article is: > The French government accused Grok on Friday of generating “clearly illegal” sexual content on X without people’s consent, flagging the matter as potentially violating the European Union’s Digital Services Act. It may be impossible to perfectly regulate what content the model can create, it is quite practical for the Grok product to enforce consent of the user whose content is being operated on before content can be generated based on it and, after the context is generated, before it can be viewed by or distributed to anyone else.
- deleted 10mo ago
  [deleted]
- chrisjj 10mo ago
  > it is quite practical for the Grok product to enforce consent of the user whose content is being operated on No, because it cannot even ID that user.
xenospn 10mo ago
If it's possible to create a model that generates photorealistic images based on a single line of text, it is 100% possible to restrict the output.
wolvoleo 10mo ago
I'm sure it's possible. If anything they can just run an AI check after generation. Similar to the way Google makes sure it doesn't return CSAM in their results. If they can filter that, the AI providers can check their own output too.
- chrisjj 10mo ago
  I think you've mistaken CSAM for child pornography.
maplethorpe 10mo ago
Sure it is. Forbid training models on images of humans, humanoids, or living creatures, and they won't be able to generate images of those things. It's not like AI is some uncontrollable magic force that hatched out of an egg. It can only output what you put in.

“AI products must be tested rigorously before they go to market to ensure they do not have the capability to generate this material,” Not possible.

by chrisjj 10mo ago

“AI products must be tested rigorously before they go to market to ensure they do not have the capability to generate this material,” Not possible.

SpicyLemonZest 10mo ago

It's extremely possible! As the source article notes, the Grok developers specifically chose to make their AI more permissive of sexual content than their competitors, which won't produce such images. This isn't a scenario where someone developed a complex jailbreak to circumvent Grok's built-in protections.

ben_w 10mo ago

> Not possible. To which governments, courts, and populations likely respond "We don't care if you can't go to market. We don't want models that do this. Solve it or don't offer your services here." Also… I think they probably could solve this. AI image analysis is a thing. AI that estimate age from an image has been a thing for ages. It's not like the idea of throwing the entire internet worth of images at a training sessions just to make a single "allowed/forbidden" filter is even ridiculous compared to the scale of all the other things going on right now.

ls612 10mo ago
These models generate probably a billion images a day. If getting it wrong for even one of those images is enough to get the entire model banned then it probably isn't possible and this de facto outlaws all image models. That may precisely be the point of this tbh.
- lokar 10mo ago
  If they can't prevent child porn, then it should be banned.
  - ls612 10mo ago
    Should photoshop be outlawed? What about MS Paint? Both of them I’m pretty sure are capable of creating this stuff. Also, lets test your commitment to consistency on this matter. In most jurisdictions possession and creation of CSAM is a strict liability crime, so do you support prosecuting whatever journalist demonstrated this capability to the maximum extent of the law? Or are you only in favor of protecting children when it happens to advance other priorities of yours?
    - lokar 10mo ago
      Photoshop is fine, running a business where you produce CSAM for people with photoshop is not. And this has been very clear for a while now. I did not see the details of what happened, but if someone did in fact take a photo of a real child they had no connection to and caused the images to be created, then yes, they should be investigated, and if the prosecutor thinks they can get a conviction they should be charged. That is just what the law says today (AIUI), and is consistent with how it has been applied.
      - ls612 10mo ago
        Somehow I doubt the prosecutor will apply the same standard to the other image generation models, which I bet (obviously without evidence given the nature of this discussion) can be convinced by a motivated adversary to do the same thing at least once. But alas, selective prosecution is the foundation of political power in the west and pointing that out gets you nothing but downvotes. patio11 once put it that pointing out how power is exercised is the first thing that those who wield power prohibit when they gain it.
        lokar 10mo ago
        You often see (appropriately, IMO) a certain amount of discretion wrt prosecution when things are changing quickly. I doubt anyone will go to jail over this. What (I think) should happen is state or federal law enforcement need to make it very clear to Xai (and the others) that this is unacceptable, and that if it keep happening, and you are not showing that you are fixing it (even if that means some degradation in the capability of the system/service), then you will be charged. One of the strengths of the western legal system that I think is under appreciated by people here is that it is subject to interpretation. Law is not Code. This makes it flexible to deal with new situations, and this is (IME) always accompanied by at least a small amount of discretion in enforcement. And in the end, the laws and how they are interpreted and enforced are subject to democratic forces.
- nl 10mo ago
  Even the OP's quote made it clear this isn't the case. Companies need to show they rigorously tested that the model doesn't do this. It's like cyber insurance requirements - for better or worse, you need to show that you have been audited, not prove you are actually safe.
- ben_w 10mo ago
  > These models generate probably a billion images a day. Collectively, probably more. Grok? Not unless you count each frame of a video, I think. > If getting it wrong for even one of those images is enough to get the entire model banned then it probably isn't possible and this de facto outlaws all image models. If the threshold is one in a billion… well, the risk is for adversarial outcomes, so you can't just toss a billion attempts at it and see what pops out, but a billion images, if it's anything like Stable Diffusion you can stop early, and my experiments with SD suggested the energy cost even for a full generation is only $0.0001/image*, so a billion is merely $100k. Given the current limits of GenAI tools, simply not including unclothed or scantily clad people in the training set would prevent this. I mean, I guess you could leave topless bodybuilders in there, then all these pics would look like Arnold Schwarzenegger, almost everyone would laugh and not care. > That may precisely be the point of this tbh. Perhaps. But I don't think we need that excuse if this was the goal, and I am not convinced this is the goal in the EU for other reasons besides. * https://benwheatley.github.io/blog/2022/10/09-19.33.04.html https://benwheatley.github.io/blog/2022/10/09-19.33.04.html
krapp 10mo ago
>To which governments, courts, and populations likely respond "We don't care if you can't go to market. We don't want models that do this. Solve it or don't offer your services here." No, they likely won't. AI has become far too big to fail at this point. So much money has been invested in it that speculation on AI alone is holding back a global economic collapse. Governments and companies have invested in AI so deeply that all failure modes have become existential. If models can't be contained, controlled or properly regulated then they simply won't be contained, controlled or properly regulated. We'll attempt it, of course, but the limits of what the law deems acceptable will be entirely defined by what is necessary for AI to succeed, because at this point it must. There's no turning back.
- ben_w 10mo ago
  > No, they likely won't. AI has become far too big to fail at this point. So much money has been invested in it that speculation on AI alone is holding back a global economic collapse. Governments and companies have invested in AI so deeply that all failure modes have become existential. Not in Europe it hasn't, and definitely not for specifically image generation, where it seems to be filling the same role as clipart, stock photos, and style transfer that can be done in other ways. Image editing is the latest hotness in GenAI image models, but knowledge of this doesn't seem to have percolated very far around the economy, only with weird toys like this one currently causing drama. > If models can't be contained, controlled or properly regulated then they simply won't be contained, controlled or properly regulated. I wish I could've shown this kind of message to people 3.5 years ago, or even 2 years ago, saying that AI will never take over because we can always just switch it off. Mind you, for 2 years ago I did, and they still didn't like it.
  - pureagave 10mo ago
    I'm sorry to tell you this, but the EU has already been lost.
    - wolvoleo 10mo ago
      Because we're not on the forefront of AI development? It also means we have less to lose when the bubble blows. I'm quite happy with the policies here. And we will become more independent from US tech. It'll just take time.

BigTTYGothGF 10mo ago

Then maybe they shouldn't go to market.

deleted 10mo ago
[deleted]
pureagave 10mo ago
AI is a nation defense issue. No nation has the luxury to stop their AI companies without the risk of losing national sovereignty.
- belter 10mo ago
  So child porn is now a national security issue?
- squigz 10mo ago
  Lumping image gen models, LLMs, and other forms of recent machine learning altogether and dressing it up in the "National Defence" ribbon doesn't seem like a great idea. I don't think the ability for citizens to make deep fake porn of whoever they want is the same as a country not investing in practical defensive applications of AI.
- dragonwriter 10mo ago
  > AI is a nation defense issue. AI image editors attached to social media networks with a design that allows producing AI edits (including, but not limited to, nonconsensual intimate images and child pornography) of other user’s media without consent are not a national defense issue, and, even to the extent that AI arguably is a national defense issue, those particular applications can be curtailed entirely by a nation without any adverse impact on national defense. You can distort any issue by zooming out to orbital level and ignoring the salient details.
- UncleMeat 10mo ago
  "We have to make the revenge porn machine for national defense" is the sort of thing that makes people light bay area tech busses on fire.

lokar 10mo ago

Then your business can fairly be ruled illegal. You don't have the right to act in violation of the law merely because it's the only way to make a buck.

kelseyfrog 10mo ago
In practice, once a business reaches a size threshold, the law is creatively decided to preserve its existence rather than terminate it. Legality is a function of economics.
- lokar 10mo ago
  Until people have had enough and push back And if you want to change the law to allow the business, go for it. But until then, we must follow the law.
- ben_w 10mo ago
  > Legality is a function of economics. Sometimes it is. Sometimes "democracy" isn't just a buzzword. X.com has been blocked by poorer nations than France (specifically, Brazil) for not following local law.

belter 10mo ago

Possible or not, what about starting by criminal investigation, to force disclosure, and find out if Musk company had child porn in the training data?

fragmede 10mo ago
It probably doesn't have pictures of fishes driving cybertrucks, but it's able to generate those, so I doubt there'd need to be CSAM in the database, but maybe I don't know how these things really work.
belter 10mo ago
AI generates child porn, HN downvotes a proposal for an investigation...

dragonwriter 10mo ago

> “AI products must be tested rigorously before they go to market to ensure they do not have the capability to generate this material,” > Not possible. Note that the description of the accusation earlier in the article is: > The French government accused Grok on Friday of generating “clearly illegal” sexual content on X without people’s consent, flagging the matter as potentially violating the European Union’s Digital Services Act. It may be impossible to perfectly regulate what content the model can create, it is quite practical for the Grok product to enforce consent of the user whose content is being operated on before content can be generated based on it and, after the context is generated, before it can be viewed by or distributed to anyone else.

deleted 10mo ago
[deleted]
chrisjj 10mo ago
> it is quite practical for the Grok product to enforce consent of the user whose content is being operated on No, because it cannot even ID that user.

xenospn 10mo ago

If it's possible to create a model that generates photorealistic images based on a single line of text, it is 100% possible to restrict the output.

wolvoleo 10mo ago

I'm sure it's possible. If anything they can just run an AI check after generation. Similar to the way Google makes sure it doesn't return CSAM in their results. If they can filter that, the AI providers can check their own output too.

chrisjj 10mo ago
I think you've mistaken CSAM for child pornography.

maplethorpe 10mo ago

Sure it is. Forbid training models on images of humans, humanoids, or living creatures, and they won't be able to generate images of those things. It's not like AI is some uncontrollable magic force that hatched out of an egg. It can only output what you put in.