15 ms·

Everyone has their own fears about AI, but my fears are especially chilling; what if AI was used to imitate a person saying something blasphemeous? My country

Everyone has their own fears about AI, but my fears are especially chilling; what if AI was used to imitate a person saying something blasphemeous? My country is already has blasphemy lynching mobs based on the slightest perceived insult, real or imagined. They will mob you, lynch you, burn your corpse, then distribute sweets while you family hide and issue video messages denouncing you and forgiving the mob. And this was before AI was easy to access. You can say a lot of things about 'oh backward countries' but this will not stay there, this will spread. You can't just give a toddler a knife and then blame them for stabbing someone. Has nothing to do with fame, with security, with copyright. This will get people killed. And we have no tools to control this. https://x.com/search?q=blasphemy https://x.com/search?q=blasphemy I fear the future.

pnut 2y ago
I guess then, you should use AI to generate videos of all of the lynch mob leadership committing blasphemy and let them sort it out internally?
- ryzvonusef 2y ago
  You joke but, given the religious/sectarian nature of the issue, all it does is empower one sect to act against the leaders of the other sect. Check the twitter link, you won't have to scroll much to find a mullah being blasted for blasphemy. No one is safe.
  - javcasas 2y ago
    Well, then target dead people. Find someone who died 6 months ago, have AI make them a blasphemer, have the sect look for dead people. After the third or fourth time, they'll stop. Not because they want to stop, but because they don't trust this being another prank by Mullah Achmed the Fool.
    - ryzvonusef 2y ago
      Just like how you are the traffic, you are also the mob Mobs are not formal organisations with leadership and membership dues. Mobs are general everyday people who are sick and tird of this miserable life and need an excuse to rabble and rouse. So in this case, fake dead person's blasphemy will only be channeled towards their family members, mobs just need an excuse to get crazy. AI is not what causes mobs, it just helps lower the incitement barrier by providing quality bait.
      - javcasas 2y ago
        In every group there are leaders (even informal ones, like that woman that shares nonstop all the bullshit to her 347 facebook friends). Make them into fools. - Stone the blasphemer! - Who? The one that died las year? The one that died in the war in '94? The one that left 4 years ago? Don't you have anything better to do other than fooling yourself and spreading more rumors? Doesn't make it any less dangerous while the mass is being vaccinated against bullshitters.
      - javcasas 2y ago
        > fake dead person's blasphemy will only be channeled towards their family members More alternatives: fake full persons. The mobs will be confused when they are directed to hunt down someone with a family name in a neighbor where no one has that family name. Furthermore, send them to the car repair shop between the coffee shop and the market in a street with no car shops, markers and coffee shops.
vasco 2y ago
The best we can hope for is that one personally avoids this for the first 5 years or so, and then it gets so widespread and easy that everyone will start doubting any videos they watch. The same way it took social media like reddit a few years of "finding the culprit" / "name and shame" till mods figured out that many times the online mob gets it wrong and so now that is usually not allowed. But many people will suffer this until laws get passed or it enters into common consciousness that a video is more likely to be fake than it is to be real. Might be more than 5 years though. And unfortunately laws usually only get passed after there's proven damage to some people from it.
- pjc50 2y ago
  > everyone will start doubting any videos they watch. This kills the medium. Just as ubiquitous scam calls have moved people away from phones, this moves people away from using media which cannot be trusted. Done enough this destroys reporting and therefore democracy. I wonder when the first nonexistent candidate will be elected.
  - Eisenstein 2y ago
    It is inevitable at this point that video as 'proof' will be killed. That we cannot do reporting and it destroying democracy is a little too much 'sky is falling'. Democracy existed before video.
  - BurningFrog 2y ago
    We've had Photoshop for decades, and I still see pictures everywhere.
    - latexr 2y ago
      Again this tired argument. Photoshop requires skill and time. AI generation takes a few seconds of typing some words. The scale is not comparable.
      - wpm 2y ago
godelski 2y ago
> what if AI was used to imitate a person saying something blasphemeous? I've been contemplating writing an open letter to Dang to nuke my account. Because at this time you can likely deanonymize any user with a fair amount of comments. As long as you can correlate. You can certainly steal their language, even if not 100% accurate. It may be caution, but it isn't certain that we won't enter a dark forest and there's reason to believe we could be headed that way. But at the same time, is not retreating to the shadows giving up?
- microtonal 2y ago
  There should be a way to cryptographically sign posts (everywhere). I know, building a web of trust sucks, etc. But if there was someone with your username signing with a particular key for 10 years and then suddenly there is something controversial somewhere with a different key, something fishy is going on. Of course, this could be misused to post something with plausible deniability, but if you want to say something controversial, why wouldn't you make another account for that anyway? I know that one could theoretically sign posts with GPG, but it would be much nicer and less noisy if sites would have UI to show something like: Signed by <fingerprint>, key used for N years. One issues is that most social media want your identity to be the account on their service and not some identity (i.e. key) that you control.
  - godelski 2y ago
    > There should be a way to cryptographically sign posts (everywhere). This just confirms it is me. Which yes, reduces the problem of me being replicated, but does not do anything for my anonymity. That part may not seem important to you, considering you use your real name, but it is to me. It allows me to be more open. Key signing will not be the solution because it isn't addressing the problem. It exacerbates it.
    - microtonal 2y ago
      This just confirms it is me. Which yes, reduces the problem of me being replicated, but does not do anything for my anonymity. That part may not seem important to you, considering you use your real name, but it is to me. It allows me to be more open. Ah, think I see your point. Your worry is that language use, etc. could be use to deanonymize you, by correlating with text that was not written anonymously. But that's a separate issue from voice or writing style cloning to pretend it's you that said it. In the latter case you could use a pseudonymous signing key? I agree that deanonymization is an issue that is hard to tackle. I wonder if someone studied how unique writing style is. E.g. browser fingerprints are fairly unique, but I wonder to what extend you can filter a person from, say a pool of 100 million, using writing style alone (grammar, vocabulary use, etc.). I guess it becomes quite easy if you engage in a lot of domain-specific discussions and use their vocabularies. E.g. if I'd talk about Marlin-kernels here, you could probably narrow me down to a few hundred people. Throw in another comment about the Glove80. Maybe ten people at most?
Kenji 2y ago
[dead]
firtoz 2y ago
> You can say a lot of things about 'oh backward countries' but this will not stay there, this will spread I'm sorry, but this is a cope out. The "lynching from apparent cultural deviation" is something that needs to be moved on from. Developed countries do the same too to some extent, with "cancel culture" and such. There are ways to have progress in this, and, well, to feed someone's entrepreneurial spirit, it's one of those really hard problems that a lot of people, let's say, "a growing niche market", needs it to be solved.
- ryzvonusef 2y ago
  Indeed, if one were to post a AI video of someone saying some racial slur or otherwise verboten language, sure it won't get them killed, but given how unemployeable and pariah they would be, that would be a death by a thousand cuts. But Blasphemy by whatever means, is one of the tools by which society sets certain boundries, and it's really hard to move away from a model that worked so 'well' for us since the first civiliations.
kmlx 2y ago
> what if AI was used to imitate a person saying something blasphemeous? > My country is already has blasphemy lynching mobs in your case the problem is not AI, it’s your country.
- latexr 2y ago
  The comment didn’t say the problem was AI, it said they feared its consequences, which is a perfectly valid concern. It’s like if someone said “I’m scared of someone bringing a semi-automatic weapon to my school and doing a mass shooting. My country has lax laws about guns and their proper use”. And then you said “in your case the problem is not guns, it’s your country”. I mean, it’s technically true, but also unhelpful. Such ingrained laws are hard to change and you can be placed in danger for even trying. Before someone decries the gun example as not being comparable, it is possible to live in a country with a monumental number of guns and not have mass murdering every day. It’s called Switzerland. But let’s please stick to the subject of AI, which is what the thread is about. The gun example is the first analogy which came to mind, and analogies are never perfect, so it’s unproductive to nitpick the example. I don’t mean to shift the conversation from one contentious topic to another.
  - underdeserver 2y ago
    Switzerland allows you to own a gun, but (generally) not to bear it. Huge difference.
    - bryanrasmussen 2y ago
      America also has laws against taking guns into theaters and indiscriminately shooting people, once you have easy access to guns the gun becomes a potential solution to problems. Maybe Switzerland just has less problems where that potential solution appeals to people.
      - input_sh 2y ago
        I'd say that the key difference is that Switzerland has compulsory military service in which you're taught how to operate a weapon properly. Therefore, everyone that has one has gone through months of training. Vs the US, where there are loopholes that you can use to avoid even a basic background check, and then use it for the very first time to shoot someone.
blueflow 2y ago
> And we have no tools to control this. Do you know "The boy who cried wolf"? Fabricate some allegations yourself and this will train people to disbelieve them.
- latexr 2y ago
  That sounds like a dangerous proposition. Either they fabricate allegations about a “nobody” and put them in trouble or they fabricate allegations about those in power and will be investigated and put themselves in danger. Neither strategy is good.
  - blueflow 2y ago
    It worked pretty fine with #MeToo. Not saying there was no collateral damage, but the end result was that people now default to not believing such allegations.
    - latexr 2y ago
      I don’t understand what you’re arguing, could you be more specific? I know about #MeToo, but what exactly are you saying worked, and what are you claiming people default to not believing?
  - GreenWatermelon 2y ago
    It doesn't have to be dangerous allegations. Fabricate a video of your country's dictator carrying a giant Boulder, with a flattering subtitle. This will send a message that videos could be deepfaked to such a degree.
- ryzvonusef 2y ago
  Doesn't work. You are assuming that people who are part of lynch mobs have the critical thinking skills to differentiate between real vs fake, and use logic. Reminds me of the post I read on twitter, of some Thai/Chinese New Yorker whose mother told him not to speak Mandarin in public when COVID related Anti-Asian hate was rampant.... And he had to explain to her that she can't expect the sort of person who hits a random Asian to differentiate between Thai and Mandarin.
sureglymop 2y ago
My specific fear is that if a picture of you next to your name is available online, that becomes part of the training set of a future model. Paranoically, I do not have any picture of myself available online. I could then trivially generate pictures or even videos of you e.g. by knowing your name. Of course that's just an example but I do think that's where we are headed and so the concept of "trust" will change a lot.
- marginalia_nu 2y ago
  Seems like the end game for that technological development is kind of self-defeating. Once it's 2 clicks away to generate a believable video of someone at the kkk kitten barbecue getting along with ted bundy and jeff epstein, surely the evidence value of that would dwindle, and the brief period in history when video evidence was both accessible and somewhat believable will come to an end.
  - cess11 2y ago
    Lynchings aren't commonly waged based on a solid process of evaluating evidence.
    - marginalia_nu 2y ago
      In that regard, the presence of this type of evidence isn't worth much either, a malicious rumor or an accusation is plenty sufficient to stoke the flames of an angry and terrified mob, nobody's going to pause the proceedings and wait for a fair and cool-headed judgement of evidence. This is after all a phenomenon that is far older than the smartphone.
  - thfuran 2y ago
    It turns out that humans aren't perfectly rational actors. Actors get hatemail and death threats because of roles they've played, "evidence" that isn't just potentially suspect but actually known to be entirely fictional.
  - disqard 2y ago
benterix 2y ago
I'm very sorry to say this but if you live in a country that is killing others for what they say, AI is probably not your biggest problem. And I don't believe an easy solution exists.
- ryzvonusef 2y ago
  AI doesn't create problems, but AI certainly lowers the barriers and improves the 'quality' of the bait. To explain for a more developed country context, the fakes that previously required skill in Photoshop and Audacity etc now is much simpler to implement with AI, allowing far more dipshits to create and share fake image/audio/video of someone they are pissed at during their lunch break on their phone. That's way too quick, allowing people to shoot far too many arrows in a huff, before their reasonable brain has time to make them realise the consequences of their actions.
- HeatrayEnjoyer 2y ago
  "You can't refuse this brand new technology but you must change your society's culture that's been around for centuries so you are compatible with it." is a repulsively Silicon Valley answer.
  - progbits 2y ago
    Not every culture deserves to live on, certainly not just because "it has been around" and absolutely not one that lynches people. Tolerance of intolerance is a bullshit strategy that won't work.
    - HeatrayEnjoyer 2y ago
      No, but it's not Silicon Valley CEOs' place to be making that decision unilaterally, nor is their culture some shining paragon of virtue either. They are not entitled to treat others paternally. It's not anyone's place to be executing it in such a hamfisted and harmful manner either.
  - lurking_swe 2y ago
mrkramer 2y ago
The only logical legal solution is that any content of you shared by you is legitimate one and all other content of you shared by somebody else is presumed non-authenthic and possibly fake.
- F-Lexx 2y ago
  What if a third party gains access to your social media account(s) and starts posting fake content from there?
  - cynicalsecurity 2y ago
    It happens all the time. They usually post spam and scam.
  - mrkramer 2y ago
    My view on deepfakes: I was thinking about deepfakes and how to protect from them and my conclusion is; there is no way you can protect from them from the practical point of view but from the legal point of view, governments can make laws where everything shared by somebody else that involves you is presumed fake unless there is substantial circumstantial evidence that says otherwise e.g. witnesses, legitimate metadata etc. etc. Even before LLMs and GenAI, Photoshop became a synonym for messing around and faking photos so there is nothing new here but now there is more powerful "faking" software available to the masses. Before computers and digital mass media sharing some compromising photo or tape could've been assumed authentic but now in the era of computers and software there is no way you can tell that something is authentic for sure. >What if a third party gains access to your social media account(s) and starts posting fake content from there? You can cause chaos and bad press in the short-term but when the original owner of the account restores ownership of the account everything falls apart. Like the commenter below said it happens all the time and it doesn't have any real impact on anything whatsoever.
movedx 2y ago
One way that we technical folk can help prevent this is by purchasing a domain that we can call our own and then host a website that's very clear: "If my image or voice is used in a piece of digital media that is not linked here from this domain, it was not produced by me." That, and cryptographic materials being used to sign stuff too. I think that's possibly the best we can hope for from a technical perspective as well as waiting for the legal system to catch up.
- ryzvonusef 2y ago
  But, but, it sounds so realistic! Listen kiddo, I dunno what 'cryptographic signatures' are, all I know is it sounds exactly like movedx saying they likes pineapple on pizza, and I know what I heard, sounds just like them, heard them dozen of times on TV, must have been an undercover journalist who exposed them, I say a person who likes pineapple on pizza is not welcome in my house whatever you say, now be gone!
  - yapyap 2y ago
    used the wrong example there, pineapple on pizza is a nice, delicious, refreshing topping
    - ryzvonusef 2y ago
      Dude I am not going in give real examples, I live in a country with lynch mobs, remember? But the point was, doesn't have to be lynch mobs, for developed countries, even 'lesser' consequences are stull terrifying. Imagine everyone in your office giving you the stink eye after lunch and you being fired by HR 'at-will', and not knowing what exactly caused it, after all the day started so normally...
    - latexr 2y ago
      Fine, then use dead rats on pizza. Don't focus on the letter of the example when you clearly understand the spirit. Conversation on HN aims to be better than that. Don’t reply with shallow dismissals, steel man the other person’s argument. Your account is a week old and half your comments are dead. I recommend referring to the guidelines. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://en.wikipedia.org/wiki/Straw_man#Steelmanning https://en.wikipedia.org/wiki/Straw_man#Steelmanning
gwervc 2y ago
This is nothing to do with AI but with intolerance of a certain religion. That religion is killing a lot in my country and many others too, but both the governments (national and supranational) and corporations censor any criticism of it. Even here on HN I got posts and accounts removed by the moderation for the slightest hint of criticism against it, and fully expected a downvoting mob by writing this comment. Sadly, it'll will continue for a long time giving how taboo the subject is.
- ryzvonusef 2y ago
  You are focusing too much on 'that' religion and not realising that parallel analogies exist for other countries, religions and culture too. Sure, not lynch mobs, but AI-generate fake media can certain ruin people's lives, and unlike photshop etc, the barriers of skill and time required are very low, and the quality is very high. I share my country's experience because I wanted to share my personal perspective and fears, but please don't under estimate how AI can affect you. Just because you won't be death doesn't mean they can't turn you into a social pariah with a few clicks.
- sensanaty 2y ago
  If you were in the US and someone were to make a deepfake of you saying a racial slur, do you think you'd fair better than if you were a blasphemer in a Sharia country? The religion isn't the (whole) issue here, this situation can apply in the secular West just as easily. The punishment won't be death, but it can still ruin people's lives. A fake pedophilia accusation comes to mind, where even if proven innocent you'll still be royally fucked for the rest of your life unless you spend considerable expense and effort.
  - thfuran 2y ago
    >If you were in the US and someone were to make a deepfake of you saying a racial slur, do you think you'd fair better than if you were a blasphemer in a Sharia country? Of course.
losvedir 2y ago
I think the answer, counterintuitively, is to make these AI tools more open and accessible. As long as they're restricted or regulated or inaccessible people will continue to think of videos and recordings as not fakeable. But make voice cloning something easy and fun to do with a $1 app, let the teens have their prank call fun and pretty soon it should work its way into the public consciousness. I had my 70 year mother ask me last week if she should remove her voicemail message because can't people steal her voice with it? I was surprised but I guess she heard it on a Fox segment or something. I think it might be a rough couple years but hopefully we'll be through it soon.
- CoastalCoder 2y ago
  > I had my 70 year mother ask me last week if she should remove her voicemail message because can't people steal her voice with it? I was surprised but I guess she heard it on a Fox segment or something. Out of curiosity, how much training data is needed currently to mimic a voice at various levels of convincingness?
  - alternatex 2y ago
    Skype does it in real time (for live translation) with a few seconds of audio. For the reasons discussed in this thread, it continuously forgets its previous training from the call to not make the voice too similar, but just similar enough to distinguish the speakers.
  - jerpint 2y ago
    Just a few seconds of speech gets you pretty far with a lot of models
    - caeril 2y ago
      Yes, this is generally true, in order to clone the timbre/pitch/tonality, but if you're going to run a voice cloning scam to fool friends and family members, you need much more to get the cadence, pauses, vocal tics, etc right.
  - 2y ago
pmarreck 2y ago
> My country is already has blasphemy lynching mobs based on the slightest perceived insult, real or imagined. They will mob you, lynch you, burn your corpse, then distribute sweets while you family hide and issue video messages denouncing you and forgiving the mob. Blasphemy laws—and the violence that sometimes accompanies them—are a cultural issue, not a technological one. When the risk of mob violence is in play, it's hard to have rational discussions about any kind of perceived offense, especially when it can be manipulated, even technologically, as you pointed out. The hypothetical of voice theft amplifies this: If a stolen voice were used to blaspheme, who would truly be responsible? This is why we must resist the urge to give into culturally sanctioned violence or fear, regardless of religious justification. The truth doesn’t need to be violently defended; it stands by itself. If a system cannot tolerate dissent without devolving into chaos, then the problem lies within the system, not the dissent. “An appeaser is one who feeds the crocodile, hoping it will eat him last.” - Winston Churchill
- ryzvonusef 2y ago
  You are focusing too much on my specific problem instead of using it as a guide to understand your own situation. Sure we have mobs and you don't, but we are talking about AI here. Infact let's imagine a totally different culture to illustrate my point. Imagine you are an Israeli, and people in your office have a habit of sending Whatsapp voice notes to confirm various things instead of calls, because that way you can have a record but don't have to type every damn thing out. Totally innocent and routine behaviour, you are just doing what many other people do. A colleague pissed at you for whatever damn stupid reason creates a fake of your voice saying you support Hamas by using said voice notes, using an online tool that doesn't cost much or require much... are you saying just because you won't be lynched, that there isn't a problem? You are confused why everyone is pissed at you and why suddenly your boss fired you, and by the time you find out the truth... the lie has spread to enough people in your social circle that there is no clearing your name. Think of how little data in voice samples is required to generate an audio clip thats sounds very realistic, and how better it will get in an year. You don't need fancy PC or tech knowledge for that, already websites exist that do for cheap. Just because you weren't lynched is no solace. People are the problem, AI is just providing quality tools with minimal skill and cost required, thus broadening the user base.
cynicalsecurity 2y ago
Is your country US? Somehow I think it is.
- ryzvonusef 2y ago
  https://news.ycombinator.com/user?id=ryzvonusef https://news.ycombinator.com/user?id=ryzvonusef It's in my profile :)
- shagymoe 2y ago
  Oh yes, the United States, founded on religious freedom, is the place where you get stoned in the street for blasphemy.
  - cynicalsecurity 2y ago
    Try to say anything related to sex or sexual freedom in the US and they will immediately lynch you. Welcome to USAbad, a Puritanic republic.
    - shagymoe 2y ago
      That's hilarious.
loceng 2y ago
We have ourselves. We have to create a culture of learning to quell reactive emotions - so we're less ideological and more critical thinker.
charlieyu1 2y ago
From Hong Kong. We already had fake audio messages that sounded like a protest leader during 2014 protests… It was always there, even a long time ago
valval 2y ago
You’d simply make such things highly illegal. No matter how I spin it in my head, there’s nothing particularly scary about this, like there isn’t about identity theft or any other crime, in reality. Even if blasphemy is illegal in your country, people would probably agree that falsely accusing someone of blasphemy is also wrong.
- zwirbl 2y ago
  Lynching someone is highly illegal, whatever the cause. And yet...
fennecbutt 2y ago
The people are the problem not the tool.
- disqard 2y ago
  I'm reminded of Chris Rock's "Guns, don't kill people, bullets do!" In a more serious vein, this is definitely about unleashing an extremely powerful technology, at scale, for profit, and with insufficient safeguards (imagine if you could homebrew nuclear weapons -- that's inconceivable!) There will be collateral damage. How much, and at what point will it trigger some legislation? Time will tell.
veunes 2y ago
The analogy of handing a toddler a knife is spot on. AI is an incredibly powerful tool, but without proper safeguards, regulations or education, it can cause irreparable harm
Jeff_Brown 2y ago
Given that this tech is unstoppable, the best defense might be a good offense: Flood the internet with clips of prominent religious and political leaders, especially those largely responsible for mob violence historically, saying preposterously blasphemous things they would obviously never say.
bufferoverflow 2y ago
It's sounds like a problem with your crazy population, not with AI.
smusamashah 2y ago
I can absolutely relate with your fear, but I think this will eventually be helpful to dismiss those mobs. Might even desensitize people boiling over 'blasphemy'. Yes, for the first few instances it will hurt. Then, eventually it will become common enough to be known by common folk. Enough that those people themselves will be sceptic enough to not act. I recall photoshop blackmailing stories where usually woman were the target. Now literally "everyone" knows pictures can be manipulated/photoshopped. It will take a while yes, but eventually common folk will learn that these audios/videos can't be trusted.
cloudguruab 2y ago
It’s not just a problem that’ll stay in one place either. This tech is getting easier, and the consequences could be deadly. Scary times, for sure.

pnut 2y ago

I guess then, you should use AI to generate videos of all of the lynch mob leadership committing blasphemy and let them sort it out internally?

ryzvonusef 2y ago
You joke but, given the religious/sectarian nature of the issue, all it does is empower one sect to act against the leaders of the other sect. Check the twitter link, you won't have to scroll much to find a mullah being blasted for blasphemy. No one is safe.
- javcasas 2y ago
  Well, then target dead people. Find someone who died 6 months ago, have AI make them a blasphemer, have the sect look for dead people. After the third or fourth time, they'll stop. Not because they want to stop, but because they don't trust this being another prank by Mullah Achmed the Fool.
  - ryzvonusef 2y ago
    Just like how you are the traffic, you are also the mob Mobs are not formal organisations with leadership and membership dues. Mobs are general everyday people who are sick and tird of this miserable life and need an excuse to rabble and rouse. So in this case, fake dead person's blasphemy will only be channeled towards their family members, mobs just need an excuse to get crazy. AI is not what causes mobs, it just helps lower the incitement barrier by providing quality bait.
    - javcasas 2y ago
      In every group there are leaders (even informal ones, like that woman that shares nonstop all the bullshit to her 347 facebook friends). Make them into fools. - Stone the blasphemer! - Who? The one that died las year? The one that died in the war in '94? The one that left 4 years ago? Don't you have anything better to do other than fooling yourself and spreading more rumors? Doesn't make it any less dangerous while the mass is being vaccinated against bullshitters.
    - javcasas 2y ago
      > fake dead person's blasphemy will only be channeled towards their family members More alternatives: fake full persons. The mobs will be confused when they are directed to hunt down someone with a family name in a neighbor where no one has that family name. Furthermore, send them to the car repair shop between the coffee shop and the market in a street with no car shops, markers and coffee shops.

vasco 2y ago

The best we can hope for is that one personally avoids this for the first 5 years or so, and then it gets so widespread and easy that everyone will start doubting any videos they watch. The same way it took social media like reddit a few years of "finding the culprit" / "name and shame" till mods figured out that many times the online mob gets it wrong and so now that is usually not allowed. But many people will suffer this until laws get passed or it enters into common consciousness that a video is more likely to be fake than it is to be real. Might be more than 5 years though. And unfortunately laws usually only get passed after there's proven damage to some people from it.

pjc50 2y ago
> everyone will start doubting any videos they watch. This kills the medium. Just as ubiquitous scam calls have moved people away from phones, this moves people away from using media which cannot be trusted. Done enough this destroys reporting and therefore democracy. I wonder when the first nonexistent candidate will be elected.
- Eisenstein 2y ago
  It is inevitable at this point that video as 'proof' will be killed. That we cannot do reporting and it destroying democracy is a little too much 'sky is falling'. Democracy existed before video.
- BurningFrog 2y ago
  We've had Photoshop for decades, and I still see pictures everywhere.
  - latexr 2y ago
    Again this tired argument. Photoshop requires skill and time. AI generation takes a few seconds of typing some words. The scale is not comparable.
    - wpm 2y ago

godelski 2y ago

> what if AI was used to imitate a person saying something blasphemeous? I've been contemplating writing an open letter to Dang to nuke my account. Because at this time you can likely deanonymize any user with a fair amount of comments. As long as you can correlate. You can certainly steal their language, even if not 100% accurate. It may be caution, but it isn't certain that we won't enter a dark forest and there's reason to believe we could be headed that way. But at the same time, is not retreating to the shadows giving up?

microtonal 2y ago
There should be a way to cryptographically sign posts (everywhere). I know, building a web of trust sucks, etc. But if there was someone with your username signing with a particular key for 10 years and then suddenly there is something controversial somewhere with a different key, something fishy is going on. Of course, this could be misused to post something with plausible deniability, but if you want to say something controversial, why wouldn't you make another account for that anyway? I know that one could theoretically sign posts with GPG, but it would be much nicer and less noisy if sites would have UI to show something like: Signed by <fingerprint>, key used for N years. One issues is that most social media want your identity to be the account on their service and not some identity (i.e. key) that you control.
- godelski 2y ago
  > There should be a way to cryptographically sign posts (everywhere). This just confirms it is me. Which yes, reduces the problem of me being replicated, but does not do anything for my anonymity. That part may not seem important to you, considering you use your real name, but it is to me. It allows me to be more open. Key signing will not be the solution because it isn't addressing the problem. It exacerbates it.
  - microtonal 2y ago
    This just confirms it is me. Which yes, reduces the problem of me being replicated, but does not do anything for my anonymity. That part may not seem important to you, considering you use your real name, but it is to me. It allows me to be more open. Ah, think I see your point. Your worry is that language use, etc. could be use to deanonymize you, by correlating with text that was not written anonymously. But that's a separate issue from voice or writing style cloning to pretend it's you that said it. In the latter case you could use a pseudonymous signing key? I agree that deanonymization is an issue that is hard to tackle. I wonder if someone studied how unique writing style is. E.g. browser fingerprints are fairly unique, but I wonder to what extend you can filter a person from, say a pool of 100 million, using writing style alone (grammar, vocabulary use, etc.). I guess it becomes quite easy if you engage in a lot of domain-specific discussions and use their vocabularies. E.g. if I'd talk about Marlin-kernels here, you could probably narrow me down to a few hundred people. Throw in another comment about the Glove80. Maybe ten people at most?

Kenji 2y ago

[dead]

firtoz 2y ago

> You can say a lot of things about 'oh backward countries' but this will not stay there, this will spread I'm sorry, but this is a cope out. The "lynching from apparent cultural deviation" is something that needs to be moved on from. Developed countries do the same too to some extent, with "cancel culture" and such. There are ways to have progress in this, and, well, to feed someone's entrepreneurial spirit, it's one of those really hard problems that a lot of people, let's say, "a growing niche market", needs it to be solved.

ryzvonusef 2y ago
Indeed, if one were to post a AI video of someone saying some racial slur or otherwise verboten language, sure it won't get them killed, but given how unemployeable and pariah they would be, that would be a death by a thousand cuts. But Blasphemy by whatever means, is one of the tools by which society sets certain boundries, and it's really hard to move away from a model that worked so 'well' for us since the first civiliations.

kmlx 2y ago

> what if AI was used to imitate a person saying something blasphemeous? > My country is already has blasphemy lynching mobs in your case the problem is not AI, it’s your country.

latexr 2y ago
The comment didn’t say the problem was AI, it said they feared its consequences, which is a perfectly valid concern. It’s like if someone said “I’m scared of someone bringing a semi-automatic weapon to my school and doing a mass shooting. My country has lax laws about guns and their proper use”. And then you said “in your case the problem is not guns, it’s your country”. I mean, it’s technically true, but also unhelpful. Such ingrained laws are hard to change and you can be placed in danger for even trying. Before someone decries the gun example as not being comparable, it is possible to live in a country with a monumental number of guns and not have mass murdering every day. It’s called Switzerland. But let’s please stick to the subject of AI, which is what the thread is about. The gun example is the first analogy which came to mind, and analogies are never perfect, so it’s unproductive to nitpick the example. I don’t mean to shift the conversation from one contentious topic to another.
- underdeserver 2y ago
  Switzerland allows you to own a gun, but (generally) not to bear it. Huge difference.
  - bryanrasmussen 2y ago
    America also has laws against taking guns into theaters and indiscriminately shooting people, once you have easy access to guns the gun becomes a potential solution to problems. Maybe Switzerland just has less problems where that potential solution appeals to people.
    - input_sh 2y ago
      I'd say that the key difference is that Switzerland has compulsory military service in which you're taught how to operate a weapon properly. Therefore, everyone that has one has gone through months of training. Vs the US, where there are loopholes that you can use to avoid even a basic background check, and then use it for the very first time to shoot someone.

blueflow 2y ago

> And we have no tools to control this. Do you know "The boy who cried wolf"? Fabricate some allegations yourself and this will train people to disbelieve them.

latexr 2y ago
That sounds like a dangerous proposition. Either they fabricate allegations about a “nobody” and put them in trouble or they fabricate allegations about those in power and will be investigated and put themselves in danger. Neither strategy is good.
- blueflow 2y ago
  It worked pretty fine with #MeToo. Not saying there was no collateral damage, but the end result was that people now default to not believing such allegations.
  - latexr 2y ago
    I don’t understand what you’re arguing, could you be more specific? I know about #MeToo, but what exactly are you saying worked, and what are you claiming people default to not believing?
- GreenWatermelon 2y ago
  It doesn't have to be dangerous allegations. Fabricate a video of your country's dictator carrying a giant Boulder, with a flattering subtitle. This will send a message that videos could be deepfaked to such a degree.
ryzvonusef 2y ago
Doesn't work. You are assuming that people who are part of lynch mobs have the critical thinking skills to differentiate between real vs fake, and use logic. Reminds me of the post I read on twitter, of some Thai/Chinese New Yorker whose mother told him not to speak Mandarin in public when COVID related Anti-Asian hate was rampant.... And he had to explain to her that she can't expect the sort of person who hits a random Asian to differentiate between Thai and Mandarin.

sureglymop 2y ago

My specific fear is that if a picture of you next to your name is available online, that becomes part of the training set of a future model. Paranoically, I do not have any picture of myself available online. I could then trivially generate pictures or even videos of you e.g. by knowing your name. Of course that's just an example but I do think that's where we are headed and so the concept of "trust" will change a lot.

marginalia_nu 2y ago
Seems like the end game for that technological development is kind of self-defeating. Once it's 2 clicks away to generate a believable video of someone at the kkk kitten barbecue getting along with ted bundy and jeff epstein, surely the evidence value of that would dwindle, and the brief period in history when video evidence was both accessible and somewhat believable will come to an end.
- cess11 2y ago
  Lynchings aren't commonly waged based on a solid process of evaluating evidence.
  - marginalia_nu 2y ago
    In that regard, the presence of this type of evidence isn't worth much either, a malicious rumor or an accusation is plenty sufficient to stoke the flames of an angry and terrified mob, nobody's going to pause the proceedings and wait for a fair and cool-headed judgement of evidence. This is after all a phenomenon that is far older than the smartphone.
- thfuran 2y ago
  It turns out that humans aren't perfectly rational actors. Actors get hatemail and death threats because of roles they've played, "evidence" that isn't just potentially suspect but actually known to be entirely fictional.
- disqard 2y ago

benterix 2y ago

I'm very sorry to say this but if you live in a country that is killing others for what they say, AI is probably not your biggest problem. And I don't believe an easy solution exists.

ryzvonusef 2y ago
AI doesn't create problems, but AI certainly lowers the barriers and improves the 'quality' of the bait. To explain for a more developed country context, the fakes that previously required skill in Photoshop and Audacity etc now is much simpler to implement with AI, allowing far more dipshits to create and share fake image/audio/video of someone they are pissed at during their lunch break on their phone. That's way too quick, allowing people to shoot far too many arrows in a huff, before their reasonable brain has time to make them realise the consequences of their actions.
HeatrayEnjoyer 2y ago
"You can't refuse this brand new technology but you must change your society's culture that's been around for centuries so you are compatible with it." is a repulsively Silicon Valley answer.
- progbits 2y ago
  Not every culture deserves to live on, certainly not just because "it has been around" and absolutely not one that lynches people. Tolerance of intolerance is a bullshit strategy that won't work.
  - HeatrayEnjoyer 2y ago
    No, but it's not Silicon Valley CEOs' place to be making that decision unilaterally, nor is their culture some shining paragon of virtue either. They are not entitled to treat others paternally. It's not anyone's place to be executing it in such a hamfisted and harmful manner either.
- lurking_swe 2y ago

mrkramer 2y ago

The only logical legal solution is that any content of you shared by you is legitimate one and all other content of you shared by somebody else is presumed non-authenthic and possibly fake.

F-Lexx 2y ago
What if a third party gains access to your social media account(s) and starts posting fake content from there?
- cynicalsecurity 2y ago
  It happens all the time. They usually post spam and scam.
- mrkramer 2y ago
  My view on deepfakes: I was thinking about deepfakes and how to protect from them and my conclusion is; there is no way you can protect from them from the practical point of view but from the legal point of view, governments can make laws where everything shared by somebody else that involves you is presumed fake unless there is substantial circumstantial evidence that says otherwise e.g. witnesses, legitimate metadata etc. etc. Even before LLMs and GenAI, Photoshop became a synonym for messing around and faking photos so there is nothing new here but now there is more powerful "faking" software available to the masses. Before computers and digital mass media sharing some compromising photo or tape could've been assumed authentic but now in the era of computers and software there is no way you can tell that something is authentic for sure. >What if a third party gains access to your social media account(s) and starts posting fake content from there? You can cause chaos and bad press in the short-term but when the original owner of the account restores ownership of the account everything falls apart. Like the commenter below said it happens all the time and it doesn't have any real impact on anything whatsoever.

movedx 2y ago

One way that we technical folk can help prevent this is by purchasing a domain that we can call our own and then host a website that's very clear: "If my image or voice is used in a piece of digital media that is not linked here from this domain, it was not produced by me." That, and cryptographic materials being used to sign stuff too. I think that's possibly the best we can hope for from a technical perspective as well as waiting for the legal system to catch up.

ryzvonusef 2y ago
But, but, it sounds so realistic! Listen kiddo, I dunno what 'cryptographic signatures' are, all I know is it sounds exactly like movedx saying they likes pineapple on pizza, and I know what I heard, sounds just like them, heard them dozen of times on TV, must have been an undercover journalist who exposed them, I say a person who likes pineapple on pizza is not welcome in my house whatever you say, now be gone!
- yapyap 2y ago
  used the wrong example there, pineapple on pizza is a nice, delicious, refreshing topping
  - ryzvonusef 2y ago
    Dude I am not going in give real examples, I live in a country with lynch mobs, remember? But the point was, doesn't have to be lynch mobs, for developed countries, even 'lesser' consequences are stull terrifying. Imagine everyone in your office giving you the stink eye after lunch and you being fired by HR 'at-will', and not knowing what exactly caused it, after all the day started so normally...
  - latexr 2y ago
    Fine, then use dead rats on pizza. Don't focus on the letter of the example when you clearly understand the spirit. Conversation on HN aims to be better than that. Don’t reply with shallow dismissals, steel man the other person’s argument. Your account is a week old and half your comments are dead. I recommend referring to the guidelines. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://en.wikipedia.org/wiki/Straw_man#Steelmanning https://en.wikipedia.org/wiki/Straw_man#Steelmanning

gwervc 2y ago

This is nothing to do with AI but with intolerance of a certain religion. That religion is killing a lot in my country and many others too, but both the governments (national and supranational) and corporations censor any criticism of it. Even here on HN I got posts and accounts removed by the moderation for the slightest hint of criticism against it, and fully expected a downvoting mob by writing this comment. Sadly, it'll will continue for a long time giving how taboo the subject is.

ryzvonusef 2y ago
You are focusing too much on 'that' religion and not realising that parallel analogies exist for other countries, religions and culture too. Sure, not lynch mobs, but AI-generate fake media can certain ruin people's lives, and unlike photshop etc, the barriers of skill and time required are very low, and the quality is very high. I share my country's experience because I wanted to share my personal perspective and fears, but please don't under estimate how AI can affect you. Just because you won't be death doesn't mean they can't turn you into a social pariah with a few clicks.
sensanaty 2y ago
If you were in the US and someone were to make a deepfake of you saying a racial slur, do you think you'd fair better than if you were a blasphemer in a Sharia country? The religion isn't the (whole) issue here, this situation can apply in the secular West just as easily. The punishment won't be death, but it can still ruin people's lives. A fake pedophilia accusation comes to mind, where even if proven innocent you'll still be royally fucked for the rest of your life unless you spend considerable expense and effort.
- thfuran 2y ago
  >If you were in the US and someone were to make a deepfake of you saying a racial slur, do you think you'd fair better than if you were a blasphemer in a Sharia country? Of course.

losvedir 2y ago

I think the answer, counterintuitively, is to make these AI tools more open and accessible. As long as they're restricted or regulated or inaccessible people will continue to think of videos and recordings as not fakeable. But make voice cloning something easy and fun to do with a $1 app, let the teens have their prank call fun and pretty soon it should work its way into the public consciousness. I had my 70 year mother ask me last week if she should remove her voicemail message because can't people steal her voice with it? I was surprised but I guess she heard it on a Fox segment or something. I think it might be a rough couple years but hopefully we'll be through it soon.

CoastalCoder 2y ago
> I had my 70 year mother ask me last week if she should remove her voicemail message because can't people steal her voice with it? I was surprised but I guess she heard it on a Fox segment or something. Out of curiosity, how much training data is needed currently to mimic a voice at various levels of convincingness?
- alternatex 2y ago
  Skype does it in real time (for live translation) with a few seconds of audio. For the reasons discussed in this thread, it continuously forgets its previous training from the call to not make the voice too similar, but just similar enough to distinguish the speakers.
- jerpint 2y ago
  Just a few seconds of speech gets you pretty far with a lot of models
  - caeril 2y ago
    Yes, this is generally true, in order to clone the timbre/pitch/tonality, but if you're going to run a voice cloning scam to fool friends and family members, you need much more to get the cadence, pauses, vocal tics, etc right.
- 2y ago

pmarreck 2y ago

> My country is already has blasphemy lynching mobs based on the slightest perceived insult, real or imagined. They will mob you, lynch you, burn your corpse, then distribute sweets while you family hide and issue video messages denouncing you and forgiving the mob. Blasphemy laws—and the violence that sometimes accompanies them—are a cultural issue, not a technological one. When the risk of mob violence is in play, it's hard to have rational discussions about any kind of perceived offense, especially when it can be manipulated, even technologically, as you pointed out. The hypothetical of voice theft amplifies this: If a stolen voice were used to blaspheme, who would truly be responsible? This is why we must resist the urge to give into culturally sanctioned violence or fear, regardless of religious justification. The truth doesn’t need to be violently defended; it stands by itself. If a system cannot tolerate dissent without devolving into chaos, then the problem lies within the system, not the dissent. “An appeaser is one who feeds the crocodile, hoping it will eat him last.” - Winston Churchill

ryzvonusef 2y ago
You are focusing too much on my specific problem instead of using it as a guide to understand your own situation. Sure we have mobs and you don't, but we are talking about AI here. Infact let's imagine a totally different culture to illustrate my point. Imagine you are an Israeli, and people in your office have a habit of sending Whatsapp voice notes to confirm various things instead of calls, because that way you can have a record but don't have to type every damn thing out. Totally innocent and routine behaviour, you are just doing what many other people do. A colleague pissed at you for whatever damn stupid reason creates a fake of your voice saying you support Hamas by using said voice notes, using an online tool that doesn't cost much or require much... are you saying just because you won't be lynched, that there isn't a problem? You are confused why everyone is pissed at you and why suddenly your boss fired you, and by the time you find out the truth... the lie has spread to enough people in your social circle that there is no clearing your name. Think of how little data in voice samples is required to generate an audio clip thats sounds very realistic, and how better it will get in an year. You don't need fancy PC or tech knowledge for that, already websites exist that do for cheap. Just because you weren't lynched is no solace. People are the problem, AI is just providing quality tools with minimal skill and cost required, thus broadening the user base.

cynicalsecurity 2y ago

Is your country US? Somehow I think it is.

ryzvonusef 2y ago
https://news.ycombinator.com/user?id=ryzvonusef https://news.ycombinator.com/user?id=ryzvonusef It's in my profile :)
shagymoe 2y ago
Oh yes, the United States, founded on religious freedom, is the place where you get stoned in the street for blasphemy.
- cynicalsecurity 2y ago
  Try to say anything related to sex or sexual freedom in the US and they will immediately lynch you. Welcome to USAbad, a Puritanic republic.
  - shagymoe 2y ago
    That's hilarious.

loceng 2y ago

We have ourselves. We have to create a culture of learning to quell reactive emotions - so we're less ideological and more critical thinker.

charlieyu1 2y ago

From Hong Kong. We already had fake audio messages that sounded like a protest leader during 2014 protests… It was always there, even a long time ago

valval 2y ago

You’d simply make such things highly illegal. No matter how I spin it in my head, there’s nothing particularly scary about this, like there isn’t about identity theft or any other crime, in reality. Even if blasphemy is illegal in your country, people would probably agree that falsely accusing someone of blasphemy is also wrong.

zwirbl 2y ago
Lynching someone is highly illegal, whatever the cause. And yet...

fennecbutt 2y ago

The people are the problem not the tool.

disqard 2y ago
I'm reminded of Chris Rock's "Guns, don't kill people, bullets do!" In a more serious vein, this is definitely about unleashing an extremely powerful technology, at scale, for profit, and with insufficient safeguards (imagine if you could homebrew nuclear weapons -- that's inconceivable!) There will be collateral damage. How much, and at what point will it trigger some legislation? Time will tell.

veunes 2y ago

The analogy of handing a toddler a knife is spot on. AI is an incredibly powerful tool, but without proper safeguards, regulations or education, it can cause irreparable harm

Jeff_Brown 2y ago

Given that this tech is unstoppable, the best defense might be a good offense: Flood the internet with clips of prominent religious and political leaders, especially those largely responsible for mob violence historically, saying preposterously blasphemous things they would obviously never say.

bufferoverflow 2y ago

It's sounds like a problem with your crazy population, not with AI.

smusamashah 2y ago

I can absolutely relate with your fear, but I think this will eventually be helpful to dismiss those mobs. Might even desensitize people boiling over 'blasphemy'. Yes, for the first few instances it will hurt. Then, eventually it will become common enough to be known by common folk. Enough that those people themselves will be sceptic enough to not act. I recall photoshop blackmailing stories where usually woman were the target. Now literally "everyone" knows pictures can be manipulated/photoshopped. It will take a while yes, but eventually common folk will learn that these audios/videos can't be trusted.

cloudguruab 2y ago

It’s not just a problem that’ll stay in one place either. This tech is getting easier, and the consequences could be deadly. Scary times, for sure.