9 ms·
Scammers deepfake CEO’s voice to talk underling into $243k transfer?
- Miner49er 7y agoHow do they know it was AI and not a voice actor?
- kobbe 7y agoIt seems like they don't know. "The target of the scam was convinced that he was speaking with his boss due to a “subtle German accent” and specific “melody” to the man’s voice and wired the money as requested. According to a representative of Euler Hermes Group SA, the firm’s insurance company, the CEO was targeted by a new kind of scam that used AI-enhanced technology to create an audio deepfake of his employer’s voice. " - https://cyberscout.com/en/blog/voice-deepfake-scams-ceo-out-of-243000 https://cyberscout.com/en/blog/voice-deepfake-scams-ceo-out-...
- stillworks 7y agoAlan Turing would like to know as well...
- vectorEQ 7y agoits a random assumption / hypothesis. maybe even it was the real ceo... :D that'd be too funny
- cyborgx7 7y agoBe an effective way to get some money.
- ta999999171 7y agoDoes this trend bring us into the promised "post-truth" world? ...or will it force humanity to finally acknowledge the very basic security concept of public/private keys for signing important shit?
- yonixw 7y agoSimple "Only act on official mail" policy can defeat this post-apocalypse deep-fake terminator.
- jjoonathan 7y agoSure, so long as the official mail is set up to check public/private keys!
- yonixw 7y agoMy point is, even with quantum computing™, all you need is simple email server with 3 attempt before user lock. If you can't manage this, public key is not the answer. I Have seen only <10 scenarios when private key in the user hands solves the problem. None were scam related.
- jjoonathan 7y ago> simple email server These days, DKIM is necessary for even the simplest of email servers that wants to interact with any of the major email platforms.
- michannne 7y agoI hope it does. I have for years thought about the two possible post-fake worlds we can end up with, either everyone learning gradually on their own how to tell truth from fiction and learning how to efficiently do their own research, or a mass-surveillance system designed to 'protect' those that 'cannot determine truth from fiction themselves'. I guess we'll see faster than I thought.
- segfaultbuserr 7y agoI recently chatted about the potential use of DeepFake in future propaganda with other HN readers, and apparently it's going to be here soon. You may want to follow this thread, https://news.ycombinator.com/item?id=21269150 https://news.ycombinator.com/item?id=21269150 But in short, Several observations... * A major threat is end-to-end VoIP/telephony encryption. Currently, the most common way to verify one's public key and ensure that no MITM wiretapping is going on, is reading a hash (encoded to words) aloud in a phone conversation. It's used to many protocols, such as Signal, or ZRTP by Phil Zimmermann et al. There is no real security, but it's considered a shortcut with reasonable security for most people, the assumption is that voice synthesis cannot be done in real-time yet. But with DeepFake it's disastrous for cryptography. Now, this shortcut is going to be blocked soon. Full verification, like signing your VoIP key with your long-term public key, or asking security questions (e.g. OTR's SMP algorithm) is needed (perhaps not to everyone, but it's now required for a lot of people to a greater extent). * KirinDave suggested that, in additional to signing, timestamping will also be important. If automatic synthesis of video and audio becomes widespread, one way to prove the authenticity of the material is to timestamp it as soon as it's recorded, or even timestamping it in real-time if real-time forgery is a serious threat (I hope not). This is one of few use cases that a blockchain actually makes sense if you want minimum trust over 3rd-parties. The final thought is that it's time to rewatch Ghost in The Shell (the TV animation series, not the movie). Released in 2000s, it portrayed and predicted our world remarkably well, and it'll give you a lot of inspirations of what would the future society look like. In one episode, the protagonists realized a government conspiracy aimed to intensify a military conflict that involves nuclear weapon, and they had the following dialogue. > "How do we stop it? Can we post the video footage online?" > "No. Video footage is seen as completely untrustworthy today."
- jacquesm 7y agoEvery company should pre-emptively attempt all known forms of CEO fraud every couple of weeks as a kind of vaccination against the real thing.
- ttul 7y agoThis is a thing already. Large banks spend up to multiple billions per year on security (for example) and even a 3,000 person enterprise can easily drop millions a year. Red-teaming is an absolutely necessary aspect of security these days.
- inetknght 7y agoRelated discussion (separate article) from yesterday: https://news.ycombinator.com/item?id=21525878 https://news.ycombinator.com/item?id=21525878
- ipoopatwork 7y agoMy bet is on a voice actor -- or an underling invoking deep fake to cover his arse
- vkou 7y agoTransferring $230,000 to your cousin's account, and hoping to stay out of jail, by telling everyone that a deep-fake CEO told you to do that is an incredibly dumb way to steal. I mean, people do incredibly dumb things, so it is certainly possible. I doubt it, though.
- newnewpdro 7y agoSounds like a good story for some embezzlment.
- Verdex 7y agoBoss, executive, manager, supervisor. Person in charge. Of these people there are many who want their subordinates to enact their will in the way that requires their own personal least amount of effort. They want to give ambiguous direction and receive exactly what they imagined but still be delightfully surprised. They don't want to hear about the unintended consequences of getting exactly what they asked for; they want it to "just work and don't trouble me with the details". Pedantry is unacceptable. Everything must be interpreted exactly how it was meant to be. The rules are meant for others. If a subordinate fails while breaking a rule, then they are fired. If a subordinate fails because they didn't break a rule, then they are fired. Ultimately, as we come closer to perfect impersonation at the press of a button, these 'people in charge' will have a tough time adapting. They want absolute obedience and unquestioning loyalty. Unless their subordinate was talking to a social engineer using a deep fake. On the other hand. The people in charge who trust their subordinates, who let them fail and encourage them to do better, who let them question their approach and decisions. These people in charge will find themselves and their organizations more resilient to these attacks. "Yeah, Jeff might be a real jerk sometimes and he's always second guessing decisions, but he doesn't transfer $12 million just because he gets a phone call where someone who sounds like me is upset that the money hasn't been transferred."
- honoredb 7y agoThis. Technical verification solutions help, but none of them defend very well against your CEO apparently ordering you to bypass the verification step.
- core-questions 7y agoSo the idealized authoritarian CEO must first have ordered you to never bypass it, and then must regularly test you to ensure that it cannot be bypassed. There's no reason this sort of authorization process couldn't be trivially handled with any one of a number of simple technologies. If there isn't already some sort of Personal Verification Service to do this, well, there's a niche to be had. Come up with a better name.
- wpietri 7y ago
- deleted 7y ago[deleted]
- scarmig 7y agoAt some point, we are all just going to blame deepfakes for all of our foibles. "Honey, I sent $20k to that camgirl because I thought she was you! Deepfakes!"
- scarmig 7y agoReminds of when some scammer bilked Google and Facebook out of $100MM through fake invoices. I'm not sure deepfakes is really likely or a differentiator here.
- crankylinuxuser 7y agoOne way to defeat this (kindof) is to institute a callback on the internally shared number. So it would have went like this: 1. scammer deepfakes CEO voice 2. demands X dollars transferred elsewhere 3. Employee hangs up and calls back using pre-shared phone# 4. Gets confirmation Of course this can be somewhat thwarted by sim-stealing attacks. But this is just another layer of defense, that stops another 99.xxx % of attacks. Enough of those and these attacks would be infeasible.
- parliament32 7y agoOr you just don't authorize multi-million dollar unreversable transfers with a phone call? Processes for this kind of thing are in place for a reason.
- crankylinuxuser 7y agoWell, the obvious counterexample would be Fintech. Calls and messages over approved SEC communications methods can be and are normal operating procedure. But in this article's case, we're talking about $.243 million , which flies counter to "don't authorize multi-million dollar unreversable transfers". Not sure what your point was here.
- yellow_lead 7y agoI have a slightly hard time believing all possible responses could be regenerated or done on the fly with a true deep fake. Maybe possible with great planning.
- LoSboccacc 7y agooccam razor: dude transferred the fund to himself after reading about the "deepfake for audio" research as it was circulating recently and faked the scam assuming worst case scenario is him being fired and rich.
- jbattle 7y agoI assumed a human was listening in and typing responses for a text-to-voice program to speak out
- Liquix 7y agoIt'd be pretty tough for the operator would have to wait for the other party to finish their thought, comprehend said thought, formulate an appropriate response, and then type it out on a keyboard.. Whilst maintaining the expected rhythm of a real-time phone conversation
- jbattle 7y agoFaster than writing an AI that could pass the turing test ;) But yeah a voice actor seems like an even easier (if less press-worthy) way to run this grift
- slowenough 7y agoAre there ML models that can detect a faked / synthesized voice, even if we can't?
- LoSboccacc 7y agothe models to detect if it's fake are used to reinforce it's training https://en.wikipedia.org/wiki/Generative_adversarial_network https://en.wikipedia.org/wiki/Generative_adversarial_network
- Scoundreller 7y agoThe best question is: is there something computationally easy to check but computationally difficult to generate?
- LoSboccacc 7y agoprobably but whatever will be used it will just be the next generation input, it's going to continue escalate indefinitely.
- dazhengca 7y agoHow could there be enough samples of his voice? Seems like quite a jump to “it was a deep fake” unless there’s more info they haven’t revealed
- Dockson 7y agoFive seconds is enough https://news.ycombinator.com/item?id=21525878 https://news.ycombinator.com/item?id=21525878
- ry4nolson 7y agoFrom yesterday: https://news.ycombinator.com/item?id=21525878 https://news.ycombinator.com/item?id=21525878 Only requires 5 seconds of voice audio to synthesize believable speech.
- danso 7y agoThe cadence of the synthesized voices are still noticeably artificial, even for the short demo phrases. This is not to say this isn't impressive. But how much does this method improve when it isn't constrained by a 5-second sample? If we feed it several hours of public speeches from Martin Luther King Jr, or hell, tens of hours of audio from President Obama or Trump – will it have the same artificial cadence, even if the tone and pitch of the imitated voice is accurate?
- epa 7y agoPoor internal controls on the CEO's team part.
- ghaff 7y agoYes, in this case. On the other hand, it's not hard to see that we're probably increasingly headed towards a future with more rigorous identity and authorization verification for at least some actions and purposes. And those more rigorous processes will inevitably lead to more edge cases that force people to go to a lot of trouble to verify who they are. (Yes, of course we can get your access to your Google account back. Just bring a notarized birth certificate and two other forms of ID to our Mountain View office between 10-11am on a Friday.)
- epa 7y agoI think we will do better, but i'm an optimist.
- acspock 7y agoIf you're suspicious, ask them to laugh.
- paul7986 7y agoAt present day to fight these tricks like your bank calling you or a contact you know calling you both saying fraud...just hang up and dial their phone number directly. Call your bank yourself and ask to be transferred to the fraud department...dial your bosses number directly, etc. Until the scammers are able to take control of your phone number which I hope never happens I think the above is a good solution to fight this junk.
- loopdoend 7y agoThat latter part happens all the time, it’s called SIM swapping.
- deleted 7y ago[deleted]
- zamubafoo 7y agoThis and other discussions on HN make me think on the fact that so much of your voice has been recorded by companies when you call their support lines. Are these troves of voice recordings stored safely with appropriate levels of access control? I mean most people can usually pick up on the difference between a voice recording and a generated voice sample based on the recording. Have there been studies where generated voices are then subjected to telephone compression and layered with background noise to appear to be a phone call from a car? If DeepFake voices only need 5 second clips to produce good enough versions of anyone's voice and any discrepancy in quality could be attributed as a bad telephone connection or masked with background noise, is anyone really safe?
- octbash 7y ago> This and other discussions on HN make me think on the fact that so much of your voice has been recorded by companies when you call their support lines. Are these troves of voice recordings stored safely with appropriate levels of access control?" "When we said 'quality and training purposes', we were referring to training a neural network."
- allovernow 7y agoThose "Grandma send money to your grandson to bail him out" scams are about to get a lot more interesting. It's one thing to convince your grandparents never to send money to random people calling on your behalf, but if the conversation happens in your voice? How do harden against such an attack when dealing with people who are technologically illiterate? Perhaps this opens up a market for secure identity verification that is accessable to the layman...
- cpitman 7y agoTo be fair, these scammers already claim to be the grandson. My grandma got a call from me, from jail, in Mexico. I guess I had a little too much to drink at a wedding, and I had a cold, so my voice was a little off. And I desperately needed to post bail in a foreign country. It wasn't the voice that tipped my grandma off. Just that I do not drink. So that was close.
- yborg 7y agoThis isn't really that hard to fix, and it's basically horse-battery-staple a la Hogwarts dorms. Families need to establish a verification phrase or word and ask "what's the password?" if there is any doubt about identity. Given deepfakery of images, even a video call won't be safe in the future and personal info about people can be datamined, so the "3 security questions" approach is also already useless.
- ttraub 7y agoArtificial voices must have improved a lot in the past 2-3 years. None of the ones I've ever heard would pass muster. Something weird in their intonation, a kind of sameness or monotony in their speech i.e. lack of excitability, a bit too perfect and precise. I just went to cereproc, one of these companies advertising a very realistic synthetic voice, and none of their voices convinced me at all, though I have to admit it was pretty good. When they get so good that they are indistinguishable except through Turing tests, and maybe not even then, we'll all be in trouble. I somehow expect that we haven't yet reached that point, though it can't be long in coming.
- malux85 7y agoYou read the article right? It literally happened. It’s already good enough to fool people who aren’t paying attention.
- unfunco 7y ago> "Analysts told the WSJ that they believe that artificial intelligence- (AI)-based software was used to create a convincing imitation of the German CEO’s voice…" > "If this turns out to indeed be a deepfake audio scam…" It didn't literally happen, they _think_ that's what happened. I'm sceptical personally, I didn't realise deep fake audio could be done in realtime now? And who is this CEO that must have hundreds of hours of publicly available audio that the voice could be trained on?
- armagon 7y agoTake a look at this article that hit HN yesterday: https://news.ycombinator.com/item?id=21525878 https://news.ycombinator.com/item?id=21525878
- deleted 7y ago[deleted]
- JulianMorrison 7y agoYou can probably avoid the uncanny valley by a "bad phone connection" filter.
- TwoBit 7y agoAnybody who will give away 243K based on a phone call alone is a moron.
- cpitman 7y agoWorking with much larger sums of money quickly distorts your perceptions of what is "a lot". 243k is under the signing authority of a random Director level manager, and to a CEO would likely feel like a rounding error.
- tastroder 7y agoPrevious discussion from 2 months ago: https://news.ycombinator.com/item?id=20864659 https://news.ycombinator.com/item?id=20864659
- mtw 7y agoIf this worked, this scheme is going to be widespread next year, making phones even more useless. It's ok to get spam and scam for emails if you're not paying anything but there's no reason to get the same for phones if you're paying top $ for it.
- ryanlol 7y agoOdds are there was no deepfaking involved here, nonsense clickbait title.
- rurp 7y ago> Bafflingly, though, Dessa said that its team created the Rogan replica voice with a text-to-speech deep learning system they developed called RealTalk, which generates life-like speech using only text inputs. I'm missing something here. How could they replicate the sound of his voice using only text inputs[emphasis theirs]?
- CapmCrackaWaka 7y agoThey probably trained the model with audio, then told it what to say using text inputs. That made me do a double-take at first too, it's a pretty misleading wording.
- tantalor 7y agoThe voice model is trained on voice inputs. The text-to-speech synthesis is based on text only, rather than audio of a different voice. So, together you have voice and text as inputs.
- p1esk 7y agoDo you know if anyone done this without using text? I mean training a model only using voice samples from different people, and using voice input only during inference.
- peteradio 7y agoSounds pretty stupid honestly. There is so much more to voice then the words that are said, and punctuation is just not possible to convey everything. Can their replicate sing? I think not..
- blt 7y agoI think it's pure ignorance on the part of the author, thinking that text-to-speech is somehow harder than speech-to-speech with a voice change.
- scott_s 7y agoI had the same question, and came the conclusion: that's not what they do. In the first post they claim that they used only text inputs [1], but in the post which actually explains what they do [2], they clearly use audio clips to train their model. [1] https://medium.com/dessa-news/real-talk-speech-synthesis-5dd0897eef7f https://medium.com/dessa-news/real-talk-speech-synthesis-5dd... [2] https://medium.com/dessa-news/realtalk-how-it-works-94c1afda62f0 https://medium.com/dessa-news/realtalk-how-it-works-94c1afda...
- parliament32 7y ago"Deepfakes voices!" makes a cool headline but there's a 90% chance either the employee or the CEO is in on it. Right now the only real fact is "the employee claims someone who sounded like the CEO called"... it's far more probable that either the employee is lying or the CEO actually made the call.
- p1esk 7y agoNot sure what makes you skeptical. The technology to do this sort of scam was ready at least a year ago. If anything I’m surprised it hasn’t happened sooner (it probably did, we just haven’t heard about it). If making money transfers is part of your job and your boss calls you to make an urgent transfer for a reasonable amount, why wouldn’t you do it?
- parliament32 7y agoBecause the simplest explanation is often the right one. Do you think it's more likely that some scamming group somehow got thousands of hours of the CEO's voice, trained a world-class voice model, got internal info about how the CEO usually calls and requests transfers, found the correct employee to target, spoofed the incoming number... or was the employee's drinking buddy like "hey, just transfer X to this offshore account and say the CEO called you, we can pin it on that deepfakes stuff we saw in the news"? Or even more likely, CEO wants to embezzle some money so he makes a call for the transfer then feigns ignorance? If there was any sort of proof that this happened, like a recording, then sure. But there's no facts in the article other than "analysts suspect..." and an attention-grabbing headline. Maybe I'm just jaded but after everything that's happened over the last few years, but I take everything the media publishes with a grain of salt these days.
- p1esk 7y agoMy understanding is you don’t need 1000s of hours of CEO’s voice. You train the model on many voices, then fine tune it on a few hours of the target voice. Regardless of what happened in this particular case, this type of scam is now possible, feasible, and will get easier to perform with time. We should be concerned.
- deleted 7y ago[deleted]
- tjpaudio 7y agoWhat I don't get about these scams is what is preventing them from working with the bank to reverse the fraudulent transfers? Sure I get that in these instances money is usually moving between the banking systems of two different countries, but just like we have extradition treaties, how is this not a thing?
- mrlala 7y agoWell, the issue is I think specifically wire transfers are designed to NOT be reversible? I'm not saying they can't reverse in case of fraud (like this). But it's probably way more difficult because the money would have been IMMEDIATELY available in the fraudsters account, in which case they probably knew to move the money somewhere else immediately / withdraw it (in some regards, I'm not sure how you withdraw $200k cash). Anyway- that's my guess why this is difficult to reverse.. if it was easy to reverse, both sides would be very suspicious of each other during a large transfer.
- rajekas 7y agoMy dystopian mind thinks it's only a matter of time when a deepfake launches fire and fury across the globe. Imagine the Cuban Missile Crisis with deepfakes. Yes, I know there are layers and layers of safeguards, but....
- drjesusphd 7y agoInstitutions can adapt, and I'm fairly confident they can handle this okay. I'm more worried about the damage that will be done to democracy from a political perspective and the criminal justice system. If you think large chunks of the US lack a shared epistemology now, just imagine when people have legitimate reasons to mistrust voice or video recordings.
- _bxg1 7y agoAnd so it begins.
- BryantD 7y agoAlthough the general class of problem is real, this story didn't hold up under examination. Here's a follow-up article in Spiegel: https://www.spiegel.de/netzwelt/web/deepfakes-werden-erkennungsmethoden-die-naechsten-uploadfilter-a-1286373.html https://www.spiegel.de/netzwelt/web/deepfakes-werden-erkennu... It's in German, but Google Translate to the rescue. Key paragraphs: 'But when asked, how do you know that such a software and no voice imitator has been used, a spokeswoman replies to SPIEGEL by email: "We do not know with 100% certainty. Theoretically, it could have been a human voice imitator But we do not assume that there are some clues (but no evidence). " 'In turn, the clues she mentions have no technical relevance; they are by no means interpreted as evidence of a deepfake. The supposedly "first case" can therefore at best be termed a "possible case". Which is quite symptomatic of the debate about deepfakes and the resulting risks.'
- slovenlyrobot 7y agoPossible case, furthered tempered by "infosec industry source". :) It is their job to sell fear
- sho 7y agoThe entire "infosec industry" is fraudulent, in my estimation. A bunch of assholes selling powerpoints and not a whole lot else. I've seen it a lot. None of them know what they're talking about. Source: my direct experience
- slovenlyrobot 7y agoI wouldn't say that, but they do suffer from something the IT industry as a whole suffers from: there is/was a need, and by fulfilling that need, they find themselves in a quandary where they're committed to something that must continue to exist in order to avoid inflicting self-harm. You can't go into business selling software on the assumption you'll sell a single copy and your work is finished. Infosec is no different. There is always this paradoxical need to ensure you never become entirely redundant, regardless of the function you specialize in I think a lot of the great "evolutionary difficulties" in our industry can be phrased this way. It's why things like Microsoft Excel are laughed at even as hundreds of thousands are trained in the art of constructing effectively bespoke spreadsheets apps in Django by the million every year. I hope for a correction some day, just as much as I hope I'm on the right side of it when it comes..
- danso 7y agoSure, this is a troubling new variation, but it's still the kind of social engineering that can be deflected with reasonable protocols, such as a second channel of authentication (e.g. "Sounds good boss, can you email me the transfer/order id to confirm things and I'll finish it from here". The underling CEO was as vulnerable to someone imitating his boss's voice the old-fashioned way. Regardless of the advances in AI, I think text-based social engineering will still be prevalent and efficient. IIRC, the Anonymous hackers who targeted HBGary got SSH access by fooling the company's chief security officer. Sure, they did by emailing from the company owner's account (thanks to a weak admin password surfaced through other vulnerabilities). But the hackers didn't even know the account name when asking for a password reset. The CSO could've stopped things by asking to do a call, e.g. "Hey let me call you, I need to walk you through this part" or even just texting the phone for confirmation. https://arstechnica.com/tech-policy/2011/02/anonymous-speaks-the-inside-story-of-the-hbgary-hack/3/ https://arstechnica.com/tech-policy/2011/02/anonymous-speaks...
- heyheyhey2000 7y agoAbsolutely spectacular - wow.
- flimflamm 7y agoMaybe it was genuine screw up and the deep fake story was created to cover ones ass.
- mitchtbaum 7y agoThe Motel 6 Saga! (Sequel) https://youtu.be/6ePFuTLYTaE https://youtu.be/6ePFuTLYTaE
- jonplackett 7y agoWouldn’t have taken much effort to verify this. Doesn’t even sound like he was that sure it was his CEO, just says he had a German accent and a similar melody. You’d think for a transfer this large he would want to be a bit more certain than that!
- throwaway35784 7y agoTrust is under full on assault by the nefarious among us. I ask you, how will we be able to trust moving forward? Will all transacting be in person? We can fake id's too. How will we be able to verify reality in this age of deception?
- donohoe 7y agoThere is zero information to backup the claim this was a deepfake and not just impersonation.
- the-dude 7y agoI haven't even seen information backing up the claim of the calls. It could be all made up.
- the-dude 7y agoSo the CEO was caught embezzling 243k and he blamed it on a deepfake? Genius.
- rhacker 7y agoI tell banks and other people that I will call them back to verify the authenticity of the incoming call.
- exabrial 7y agoCombine this with callerId spoofing and it'd probably fool me.
- dmix 7y agoA simple predefined verbal password that updates once in a while would be sufficient. Especially if you're transferring hundreds of thousands based on a single phone call.
- fitzroy 7y agoAlmost everything digital in audio seems to have been a decade or two ahead of video (non-linear editing, lossless/uncompressed quality, digital synthesis of elements, distribution, etc), primarily due to the bandwidth and processing power requirements being much greater for video. So I'm curious how, with regard to deepfake technology, video seems to be well ahead of audio. Is audio deep fake technology simply less interesting to people? Are listeners far more sensitive to voice not being perfect? Is the human input just a lot less helpful in modeling the output with voice vs video (where the initial human input is only slightly more helpful than just synthesizing from text-to-speech)?