27 ms·

AI clones teen girl’s voice in $1M kidnapping scam: ‘I’ve got your daughter’

198 points by agomez314 4y ago

RcouF1uZ4gsC 4y ago
The real enabling technology in this scenario is cryptocurrency. Cryptocurrency really changes the risk/reward ratio of scam (and real) ransoms.
- sebzim4500 4y ago
  I miss the days before 2008 when nobody was ever kidnapped, robbed or extorted. I'm pretty sure it is faster to withdraw £500,000 in cash from your bank than to set up a crypto trading acccount and go through KYC etc.
none_to_remain 4y ago
I already set up the family's secret "not a robot" password
deleted 4y ago
[deleted]
kroolik 4y ago
I'm really negative about the impact AI will have on the society. We have already been drowning in fake news and polarizing information. Now, the likes of DALL-E and DeepFake can generate convincing fake graphics. Chatgpt and the likes generate convincing fake news. Voice AI can generate convincing voice from small samples. If you were afraid of your elderly relatives being scammed by people pretending to be policemen or grandchildren, now more tech-concious people will get scammed by the voice and look of their relatives. Are we really approaching the reality where we need 2FA to trust the other person is really who they are?
- icepat 4y ago
  > Chatgpt and the likes generate convincing fake news It's not like fake news was a non-issue before ChatGPT existed. Breitbart, and other fake news sites existed for years before this was even imaginable. Fake graphics were around for ages too, with image manipulation, even before computers. Take for example the Surgeon's Photo of the Loch Ness Monster.
  - jacquesm 4y ago
    This argument has been rehashed time and again and it's getting a bit boring: the fact that you can do something at all is qualitatively different from when you can do that same thing at scale.
    - chimpanzee 4y ago
      Agreed. As I’m sure you know, it’s easier for people to reason when the arena presents only discrete possibilities, especially binary ones, eg yes/no or black/white. The arena of the continuous, which encompasses most of the natural world, is far more difficult. It doesn’t allow for the sort of arguments where one can assert things with ego-boosting absolute confidence. Matters of scale naturally fall into the continuous sort, but scale can also introduce new discrete possibilities. Perhaps we need to present the new specific, discrete effects and outcomes that AI-at-scale will introduce. Maybe, just maybe, that’ll change minds of people who are truly open to being changed. My intuition though is that this argument is repeated here (on HN in particular) not for lack of knowledge or thought, but rather because they WANT to see the chaos that’ll result. They want all the positives and negatives, no matter the balance, simply because it’s exciting and adds to their “mundane” lives. Where the mundanity is of course subjective, simply the result of their default worldview, rather than anything nearing an objective description of the world as it is.
      - jacquesm 4y ago
        > simply because it’s exciting and adds to their lives. It's certainly exciting but I wouldn't bet on it adding to our lives just yet. Maybe. But many avenues leading to net negative outcomes are still part of the tree.
- yamtaddle 4y ago
  The "fun" thing about the tech in the current state is that it's wrong quite a bit, which puts a cap on how much of a productivity boost it can give any organization that cares about correctness or its reputation; meanwhile, it has nigh-unlimited productivity-boosting promise for organizations that don't care about correctness or reputation. Which kinds of organizations don't care about correctness or reputation? Scammers, spammers, and certain types of propaganda-spewing organization, especially those that puppeteer "grassroots" campaigns. We may give a single-digit multiplier in productivity to some roles in legitimate business, while giving a two or three digit multiplier to similar roles in harmful, parasitic organizations.
  - smt88 4y ago
    I'm much more concerned about dictators than I am about scammers. Some countries have already successfully rewritten the history of less than 30 years ago with surveillance and media control. What will happen when they can also control all digital reality? They can invent an uprising of millions of people that never actually happened, and there will be photo, audio, and video "evidence". AGI is scary, but it's not immediate. I think companies like OpenAI are wrecklessly pushing us toward immediate societal collapse.
    - infogulch 4y ago
      I'm also very concerned about where AI will take us, but will bad actors just quit if OpenAI decides to halt? Or will they continue to develop it, but in secret, so we have no clue what its capabilities are or how to control it? I fear the only way out is through.
      - iamdamian 4y ago
        Couldn't you say this about nuclear research or bioweapons as well? Yet those are highly (and, seemingly effectively) regulated.
        from 4y ago
- LatticeAnimal 4y ago
  ... That is an interesting point. I wonder how long it will be before some enterprising developer hooks up an LLM to a HN account with the instructions to "blend in while promoting X". Maybe that AI already exists... It wouldn't take more than a day.
- li4ick 4y ago
  I'm also more on the negative side because I'm really not convinced by the whole "AI will just automate/remove the boring aspects of life". Every single prototype capability of current AI points in the direction of a worst-case scenario of overall misery.
  - yoyohello13 4y ago
    Absolutely. I feel like I'm just watching the car crash in slow motion. Hopefully I'm wrong, but I really don't see this AI revolution working out well for humanity (in the short/medium term). I'm sure we will eventually get through it, but I think life for most is going to get real hard for the next 20-50 years.
  - anonymouskimmer 4y ago
    > Every single prototype capability of current AI points in the direction of a worst-case scenario of overall misery. Software is often aimed at solving problems and making processes more cost-effective. Founders are often most interested in solving their own problems, problems of their friends, or the most valuable problems of their customers. For B2B companies to customers are other businesses. The greatest cost for businesses are employee costs. Solving the employee cost, but eliminating employees, is a great target for business owners.
- morkalork 4y ago
  I think we're passed due for some kind of 2FA for the phone. Sometimes my bank or credit card company will call and try to sell me shit and I tell them I have no proof of who they are hang up. Too bad for them.
- backtoyoujim 4y ago
  Like I'm going to carry photos of traffic lights and crosswalks everywhere I go.
- sebzim4500 4y ago
  I'm sure those will all be serious problems (or already are problems) but I think they pale in comparison to the potential upsides. This could be the biggest jump in productivity since the industrial revolution. Managed properly, it could lead to what is essentially a utopia.
  - layer8 4y ago
    I don’t think the way economic and other incentives work are necessarily favorable to a utopia being established.
    - sebzim4500 4y ago
      As long as one country does it properly and lets people immigrate we are ok.
- toss1 4y ago
  >>Are we really approaching the reality where we need 2FA to trust the other person is really who they are? YES In fact, not approaching, but already passed the threshold. If you have any assets, it'd time to be sure you have a set of actually obscure un-guessable prompt & responses to verify identity in your family. And it needs to be better than "what's our first dog's name?" or "where do we vacation?" type stuff found on FB; if they're going to the trouble of getting images & voice samples to clone, they'll find that stuff. Now, it is not that everyone is famous for 15 minutes, it is that everyone needs to be up-to-date on security to avoid being randomly shot or scammed. Nice society we've built.
- newhouseb 4y ago
  I was having a conversation with some friends the other day about what schemes might mitigate some of these risks, something like an anti-safe word, i.e. a "danger word" that someone can be used to remotely validate that a loved one is authentically in danger. This is fairly low-tech and likely susceptible to various kinds of social engineering, but I'm curious what a more robust approach might look like that doesn't involve us all regurgitating 6-digit codes like robots all the time.
  - theNJR 4y ago
    Yep, this is the way
  - mLuby 4y ago
    Let's say you call or text me, requesting gift cards or my password or something else weird. I tell you I'll email you immediately to verify. I email you to confirm you want N $X gift cards. You do so. Or if the inbound contact is by email, you verify by text or phone call. Or Signal/Discord/FB chat apps, etc. Heck even LinkedIn Inmail could be your verification channel. You could still be compromised, but that'd require attackers to have significantly more access and readiness. And if you reach the other person and they're unaware, now you both have some support as you figure it out.
    - jamesshamenski 4y ago
      If one service was compromised, isn't it reasonable to think that the scammer can access other services during this breach?
- confoundcofound 4y ago
  > Are we really approaching the reality where we need 2FA to trust the other person is really who they are? Yes we are. We are approaching a world where people will not want to invest their time, energy, and emotion engaging with other supposed humans remotely unless they have verified their personhood / identity. I'm actually surprised we have not yet seen an entire industry built around human authentication. Seems like Apple is the only company taking this seriously. The standard of human interaction will either be meeting IRL or signing communications biometrically.
  - NickC25 4y ago
    >I'm actually surprised we have not yet seen an entire industry built around human authentication. Seems like Apple is the only company taking this seriously. Completely agree - but to that - with how quickly AI voice cloning is improving, and how quickly image creation tools are improving, how long will it be until someone can sample my voice well enough to command Siri to do something and Siri won't be able to tell the difference? How long will it be until an AI generated photo of me is so realistic that it can unlock my phone via FaceID? I have a feeling that's going to happen in a shorter timeframe than we all think it will. I mean, my voice is somewhat similar to my father's, and if I'm around his phone, I can say "Hey Siri" and his phone will think it's his voice, not mine.
    - confoundcofound 4y ago
      We're going to see an arms race between AI and biometrics / authentication technology. My hope is that the latter outpaces the former, but it doesn't seem to be panning out that way. In the future, we might be only able to unlock our phone or command Siri when hooked up to a continuous biometrics sensor.
      - NickC25 4y ago
        You're probably not too far off from what will eventually become reality. Perhaps that's Apple's angle with the AR/VR headset it supposedly has in the works - if you've got your headset on, you can unlock your devices, and if you don't, you have to manually unlock devices via a password or PIN or whatever.
- comfypotato 4y ago
  How is AI different than any other technology in this respect? As far back as the discovery of fire, new technology has enabled more positive outcomes than negative. Some tech more so than others, I guess, but what makes you lean so negatively regarding AI? It’s already improved my life considerably. Just the basic ChatGPT web app has extended my capacity in multiple respects.
  - puttycat 4y ago
    Rule of thumb: reductionist arguments don't hold for technologies from the Internet onwards. For example, the Internet is nothing like TV, so arguments like "we grew up on TV and ended up ok" don't hold. Same holds in my opinion to AI-scams.
    - bawolff 4y ago
      I disagree. Do you have examples to back that up? > we grew up on TV and ended up ok Not sure what you are comparing here, but we (people in early 30s) also grew up on the internet and mostly ended up ok. Some people who grew up on tv did not end up ok.
      - pixl97 4y ago
        Quantity is a quality in itself. If I catch a fish in a lake that's not problem. If I catch all the fish in the lake we need to stock the lake again with fish. If I catch 10^30 fish on the planet, fish are extinct.
        ta1243 4y ago
        > If I catch 10^30 fish on the planet, fish are extinct. The entire planet is extinct
        pixl97 4y ago
        Oopsie, I turned the visible universe into paperclips.
- simion314 4y ago
  Maybe all countries need to finally give importance to this issues, put a few take scam crimes seriously, stop trading with countries that do not cooperate, put the scammers in jail. Tech evolved any idiot can get a copy of photoshop and some video software, we need tp solve the problem, otherwise is like preventing creation of email because we are too incompetent to address the spam problem.
- marksmith2996 4y ago
  Don't worry it'll probably just exterminate us anyway then fake news doesn't matter.
- dave_sullivan 4y ago
  Don't be negative about revolutionary new technology that can make the world a better place. As it stands, people believe whatever dumb thing they read online. Thanks to AI, they will have to learn that if something is out of the ordinary, they should confirm with multiple sources. I call this a net win. And for those literally medically incapable of this level of reasoning, we will soon have "AI firewalls" (Gibson ICE) that can tell people "Hey, this looks like a scam!" and also help them reason about complex topics. This is fantastic, what a time to be alive!
  - kroolik 4y ago
    > NOW they will have to learn that if something is out of the ordinary I once shared this approach, but learned that the majority of people don't really want to leave their echo chambers and gladly accept anything that amplifies it. And the bad part about it is that the western world is based on democracy, and democracy is controlled by the majority. You may not care about people that are easy to manipulate, but in reality you are not the target as your voice won't matter at scale.
- sandworm101 4y ago
  >> Are we really approaching the reality where we need 2FA to trust the other person is really who they are? We are already there. In the kidnapping context, we have been there for a great many years. If someone say they have kidnapped my child, I will text/call that child immediately to verify. The 'second factor' is the realworld daughter. A kidnapper must both create a facsimile of the daughter and then also render the real daughter incommunicado. We don't need new 2FA because we already have it.
- darkerside 4y ago
  It's funny, technology was able to bring us together across geographic distances. Someone on the other side of the world was a phone call away. You can turn on your TV or phone and instantly tap into seeing, hearing, and reading people and their thoughts from around the globe. Almost as quickly, AI may unwind all of that. It lay become that there's more noise than signal across all of these technical media. The only thing you can trust, and the only thing that matters, is the people in front of you. Just like the old days.
- deleted 4y ago
  [deleted]
- mr_mitm 4y ago
  We need a PKI run by the governments. National ID cards that are smart cards. Cryptographically sign any and all digital communication. Self-signed certificates could still be used with TOFU. However, I'm not sure if total loss of pseudonymity is less of a horror scenario.
  - layer8 4y ago
    You can still use pseudonyms with PKIs. The CA will know who you really are, but it doesn’t need to be public by default. And it can be regulated such that a judicial order is necessary to disclose your identity to a third party (including to the government).
  - tommiegannert 4y ago
    I've been wondering what would be the event that causes digital signatures to become the norm. Though, I didn't expect it to be machine learning. I'm scared by how much is just done by phone with people who don't know me.
- lbotos 4y ago
  I think it will be addressed by "community". For the last probably decade (maybe more) every online social space has been pressured to serve global scale. That increases surface area, and increases more strangers. Smaller, more trusted rooms, will combat the threat of misinformation. The universe is built on waves/cycles. I think we are going to see pressures to make smaller rooms because of AI. I'm actually excited for smaller, more intimate spaces, with trusted people who now have 10x ability from the powers of AI. That seems like a LOT of fun.
- dclusin 4y ago
  Ai generated news has actually been a thing for a while now. The news agencies that use it still do fact check the output.
- ravenstine 4y ago
  Your concern is understandable and relatable, but I approach the issue from a stance that is, while not necessarily optimistic, definitely is less pessimistic. I don't think we truly know just yet how negative an impact AI will have on society. Every time there has been a technological leap, people have panicked over what the gizmo of the now will do to society. Again, not an invalid concern, but society has yet to have been blown apart by anything. Also, everything potentially being just AI may inadvertently get the public to do the right thing. People should never have been as trusting of authority figures or institutions in the first place. If everyone assumes that everything is likely to be complete hogwash, which was already true in many cases, then they may not just swallow everything as fact. Maybe fewer people will blindly consume news, which is a good thing; 99% of news is not actionable or good for an average person's well-being. And if enough ransom demands turn out to be AI-generated scams such that the real ones are overall far less successful, it's possible that fewer people are kidnapped for ransom in the first place. I'm not saying I know any of this, but rather the opposite.
  - yeck 4y ago
    In a world where people need to be constantly paranoid, I think many will develop true paranoia, or just give up trying to understand what is going on and defer to who has the power already. Certainly it is not possible for individuals to seek truth on all things on their own, even if it was their full time responsibility to attempt so.
- onemoresoop 4y ago
  > If you were afraid of your elderly relatives being scammed by people pretending to be policemen or grandchildren, now more tech-concious people will get scammed by the voice and look of their relatives. Are we really approaching the reality where we need 2FA to trust the other person is really who they are? I thought I was immune to certain scams where the scammer accent was a dead giveaway that they were scammers. Now with capabilities of AI I feel somewhat vulnerable again though Im far from being elderly.
- ZainRiz 4y ago
  The past century has really been an anomaly when it came to trusting news, with photos and videos seeming to be reliable proof. Before that, societies had to be structured very differently to account for the lack of proof. You'd hear people from out of town talking about what they saw in a neighboring city. You'd need to judge how trustworthy the person was. People would expect a chain of narration, to understand how _that_ person came to learn a bit of information (or if they claimed to witness it first hand). As AI generated content becomes more popular, I predict we as a society are going to go back to relying more and more on the reputation of the speakers in question. Who might you consider reputable? People you've met personally, people who your community respects, and of course, influencers who you follow who's persuasive words match your pre-existing world views
  - Georgelemental 4y ago
    This change will likely advantage established institutions, no? It will be much harder for a whistleblower, random, etc to prove something that the powers-that-be would prefer be kept quiet.
- MagicMoonlight 4y ago
  It’s going to be great. The internet will be a lifeless sack of shit full of bots and we’ll all have to meet up in person for true conversation. We can have parties again and have friends instead of depression.
- bcrl 4y ago
  Of course the dominant 2FA methods will be those that are easily forged, so the general public will still get swindled (I'm looking at you banks that are insisting on using SMS to "verify" my identity). It's as if nobody cares about the world programmers are creating.
- 1letterunixname 4y ago
  This Michael Bay movie script writes itself: 1. A not exactly rocket scientist POTUS takes office in 2024. Let's suppose they were a populist, triumphalist, religious, agro, law-and-order DINO to make it interesting and fictional. 2. Rogue AI launches a social engineering communication offensive against CENTCOM impersonating generals, cols, and ltcs against enlisted and lower ranks to carry out a first strike against China with a digital hallucination that Taiwan is being "attacked". It uses details gleaned by wiretapping the upper echelon to circumvent normal N-person keying rules and authentication protocols. WarGames 2: Joshua Wins
morkalork 4y ago
Is hornswoggled a common word in the USA?
- jacquesm 4y ago
  I read it as 'to be cheated'. And I'm nowhere near native level, I think the link for me is the swoggled mentally links it to swindled.
- smolder 4y ago
  I've never heard it anywhere as a USian. It seems the type of obscure and redundant word that people bring out once per decade. Without looking it up, I could guess horn refers to phone and swoggled is another swindled, so to hornswoggle is to phone-scam.
- nostromo 4y ago
  No. Scammed, tricked, or even bamboozled would be much more common.
- the_jesus_villa 4y ago
  No, not common at all
- oidar 4y ago
  Not with the current generation of people. I can't speak to any other generation, but as a native speaker I did understand it. It seemed archaic to me.
  - bentley 4y ago
    I wouldn’t call it archaic, but more folksy. I think most people would understand it, but would rarely encounter it outside a humorous context. It’s pretty similar to bamboozle, except that’s gained kind of a “cute” connotation in recent years (due to pet memes, etc.) that hornswoggle doesn’t have.
- sharps1 4y ago
  Not that I know of. I’d guess probably came across it randomly and liked the word. Seems like it comes up in crosswords https://crosswordtracker.com/clue/hornswoggle/ https://crosswordtracker.com/clue/hornswoggle/ Also funnily not related is a WWE stage wrestler name who had a YouTube video done in them a year ago. https://simple.wikipedia.org/wiki/Hornswoggle https://simple.wikipedia.org/wiki/Hornswoggle.
LinuxBender 4y ago
I would personally always demand proof of life regardless if fake or real and I would ask my family member to answer a question only they would know the answer to. We would of course have code words ahead of time. Maybe this is a sad anecdote, but the code phrase my mom and I had for duress was "I love you"
- urbandw311er 4y ago
  What would you say if you just wanted to say “I love you”?
  - LinuxBender 4y ago
    I would just give her a hug in person.
PragmaticPulp 4y ago
From the article: > DeStefano found the voice simulation particularly unsettling given that “Brie does NOT have any public social media accounts that has her voice and barely has any,” per a post on the mom’s Facebook account. > “She has a few public interviews for sports/school that have a large sampling of her voice,” described Brie’s mom. and > Then, DeStefano remembered that her 15-year-old daughter Brie was on a ski trip, so she answered the call to make sure nothing was amiss. No public social media accounts combined with the timing of the fraud happening during a ski trip implies that somebody close to the family was in on it. It's possible that they used the public sports interview and got randomly lucky with the trip timing, but I'd be looking much closer to home for a suspect.
- jacquesm 4y ago
  It always pays off to look for suspects close to home but I would definitely not rule out an outsider that monitors social media.
  - PragmaticPulp 4y ago
    > but I would definitely not rule out an outsider that monitors social media. The part I quoted claims that she doesn't have any public social media, so that wouldn't work. That's assuming we're getting the real story, of course.
    - sebzim4500 4y ago
      I still think it is way more likely that someone posted about he trip on social media. There is basically no benefit to targeting someone you know with this sort of scam, and the risk is higher.
    - jacquesm 4y ago
      When people say they 'don't have social media' it usually is a good idea to check what their definition of 'social media' is. If they have a web presence that they regularly update it doesn't really matter what the label is, it can likely be used to infer a whole bunch of stuff. I know quite a few people who claim not to have 'social media' and yet they are on LinkedIn etc. Maybe I should have used 'web presence', but I think that what I intended to convey is clear: if you have a regularly updated online identity then you are giving an impostor a lot of useful information. If there is absolutely nothing about this person online, including that they don't use any of the video chat services, no tiktok videos etc then yes, the circle of suspects would narrow accordingly. But most teens are quite active online, even if their parents aren't always aware of it.
xen2xen1 4y ago
Was literally wondering about that while watching a Netflix show. Nothing can be trusted.
awb 4y ago
Proof of identity is going to be a huge opportunity. Images, text and voice can now be spoofed with minimal cost and effort. With the progress of deepfakes and text to video, how much longer until you can spoof video calls? Meeting in person is not practical in many scenarios. Anyone know of any promising ideas or companies in this space of digital trust?
- Nathanba 4y ago
  it's not any harder than any website login. The only thing is that this isn't available as a simple service to normal people, maybe this is a feature Facebook should offer to have more value: your kids can sign in via biometrics on their phone and then it shows their parents on their facebook account that the child used biometrics X seconds ago to provide proof of identity.
  - justrealist 4y ago
    > it's not any harder than any website login Have you ever gone though a password reset process with a bank or 401k? They can't just say "oh, you lost access to your email? sorry, your money is gone". They rely on phone calls, documents, security questions... all things which are very susceptible to programmatic social engineering.
    - Nathanba 4y ago
      I dont see how it matters, this is for the usecase that there is already an established and verified account with e.g Facebook. Many betting or crypto sites do passport and video verifications, Facebook can do the same. Then all the child has to do is login via the Facebook app as usual, on iPhones via biometrics but it doesn't have to be biometrics. It can be any kind of 2fa or tripple verification. Authenticator code + email + sms + app login, simply cutting down on the likelihood that someone is a fake person. Then the parent can see which verifications were done by the child on the Facebook dashboard to see if the kid is alive and able to do verifications instead of whatever else the attacker claims. I think you meant to say that the bank verification process can be socially engineered... well first of all to actually do that you also need a lot of fake documents and expertise and the bank has to utterly fail at their job. Which hopefully prompts governments to require in person meetings at the offline location for future bank accounts.
larsiusprime 4y ago
My question is if this will eventually backfire on kidnappers, because if this escalates enough if might create the expectation that it's not real, so a genuine kidnapper can't distinguish themselves credibly from a virtual kidnapper, because what previously was credible proof of life (the voice of the victim) is no longer so.
- dmix 4y ago
  Do random people even get kidnapped for ransom anymore? It seems like technology and risk/reward killed that one pretty quickly. At least in the west and places with stable governments/law enforcement.
  - Beltiras 4y ago
    It's hard to find good data from the US due to how statistics are collected. upwards of a million children go missing each year.
    - dmix 4y ago
      That stat doesn't provide much insight when asking about ransom or the sort of criminal schemes we're talking about.
      - Beltiras 4y ago
        I know. If only the data were collected at all so we could have a discussion about kidnapping in the USA....
    - from 4y ago
      That is almost entirely runaways and of the small fraction that isn't they are usually abducted by other relatives. About 99% of them return home [safely](https://www.ojp.gov/pdffiles1/ojjdp/196465.pdf#page=6 https://www.ojp.gov/pdffiles1/ojjdp/196465.pdf#page=6). Taken (2006) style kidnappings basically do not happen. Edit: https://www.missingkids.org/ourwork/impact https://www.missingkids.org/ourwork/impact says there were 142 "nonfamily abductions" in 2021 and 121 of those were resolved.
alex_young 4y ago
AI and deepfakes are very popular subjects right now, but is there any evidence that either was used here?
- yeck 4y ago
  I am not sure it even matters either way. It is certainly possible to do things like this right now, and "kidnapped" or "arrested" family member scams are well established as existing. The significance of this report is less that it happened, and more that it is an illustration of what is not only possible, be very cheap and easy to do.
deleted 4y ago
[deleted]
stbullard 4y ago
Nothing in this article gives any reason to believe that the voice on the other end of the phone was AI-generated. Occam’s razor suggests it was more likely a human pretending to be her daughter. I’m guessing after she realized that the voice wasn’t her daughter, the mother convinced herself it must have been a deepfake to explain herself having been so easily convinced.
- dontupvoteme 4y ago
  This is likely as currently only elevenlabs has the publically available technology to do this and is SaaS only (and i presume requires law enforcement traceable usage after that first weekend and 4chan's fun with it)
  - 015a 4y ago
    I've played with elevenlabs' solution; its really good, but I don't think it could believably emulate the voice of someone under panic duress. The voices it produces have a pretty believable, but flat, affect. Maybe it has options I haven't played with, or maybe you could train it to add panic to the voice. Idk, something about this story feels fishy to me. I don't doubt that we're headed toward this kind of future; but we're talking about very high level spear-phishing to accomplish this today (knowing the individual isn't home, having samples of their voice, highly sophisticated AI knowledge to make the voice and add elements of panic duress, to actually succeed you'd also need to compromise lines of communication to the individual, e.g. do this while they're on a trip to the amazonian rainforest or something). This feels more like an AI hit-piece than an actual done-with-AI story.
    - neom 4y ago
      Watch this to the end: https://www.youtube.com/watch?v=hYQmF_QLR38 https://www.youtube.com/watch?v=hYQmF_QLR38
  - yeck 4y ago
    I did a quick search and found this among a list of many: https://voice.ai/ https://voice.ai/
neom 4y ago
In the past month, CBC has been full of news reports of Canadians getting screwed by AI voice scams, mostly old folks: https://www.youtube.com/watch?v=ALVb6UlGURk https://www.youtube.com/watch?v=ALVb6UlGURk https://www.businessinsider.com/couple-canada-reportedly-lost-21000-in-ai-generated-voice-scam-2023-3 https://www.businessinsider.com/couple-canada-reportedly-los... https://www.youtube.com/watch?v=hYQmF_QLR38 https://www.youtube.com/watch?v=hYQmF_QLR38 https://www.youtube.com/watch?v=5wGB_jF05JU https://www.youtube.com/watch?v=5wGB_jF05JU https://www.youtube.com/watch?v=SbBTLDDP8Eg https://www.youtube.com/watch?v=SbBTLDDP8Eg https://www.youtube.com/watch?v=Lwylb4cE1b8 https://www.youtube.com/watch?v=Lwylb4cE1b8
deleted 4y ago
[deleted]
motoxpro 4y ago
Just makes me even more upset about that company that YC funded that had a Launch a few weeks ago that let’s you do EXACTLY this. :(. Been telling my family to come up with a word/phrase to authenticate voice. I.e. floorboard
- JCharante 4y ago
  Adobe demo'd this 10 years ago and never launched the product
ineptech 4y ago
Man, do we need a way to prove you are who you say you are that doesn't involve a giant corporation. I feel like there was a window where we could've organically coalesced around something open source, like Keybase, or a government oauth run by 18F, or MIT's PGP keyring or something, but the moment has passed and the only identity service you're going to realistically get a majority of people to adopt is for-profit social media bullshit, which is incentivized to stay fragmented and not really address problems like this.
elicash 4y ago
Apple/Google should allow people to answer phone calls from strangers with a voice changer. It's way too easy to clone someone's voice and cause harm. Alternatively, they could notify you if they detect a faked voice, but maybe these two ideas are contradictory. Another solution, of course, is to not pick up the phone from unknown numbers unless you're expecting a food delivery or something.
- jamilton 4y ago
  That would cause more harm than it would prevent. People calling from other people's phones is a common and necessary thing. Trying to detect faked voices would make sense.
jschveibinz 4y ago
Remember to pre-arrange a shibboleth (code word or phrase) with your families just in case this happens to you.
- deleted 4y ago
  [deleted]
- segh 4y ago
  If you've failed to pre-arrange, you could always just ask something personal. "Sorry Sally I know you've been kidnapped, but before we continue, what was the name of our first dog?"
  - sebzim4500 4y ago
    Can you come up with a question that definitely isn't answered on social media while you are in a high pressure situation?
- dontupvoteme 4y ago
  [flagged]
kleer001 4y ago
For those who worry this is the end of safety for all mankind I'd like to point you towards this quote from the Victorian era "Rogues are very keen in their profession, and know already much more than we can teach them." Alfred Charles Hobbes - 1851
mquirion 4y ago
This is very concerning, but I think my bigger, more immediate worry - and one I suspect will happen at a much larger scale in the US, if not the world - is the use of these tools in bullying, particularly among school-aged kids. The detrimental effects on kids (and thus, eventual adults) due to the misuse of these tools could be cataclysmic.
deleted 4y ago
[deleted]
linsomniac 4y ago
Looks like it's time to set up a family sign/countersign for authenticating each other.
deleted 4y ago
[deleted]
IvyMike 4y ago
Reminds me of Greg Egan's "A Kidnapping" from "Axiomatic". (I'd like to describe it, but it would spoil it. Just go read Axiomatic, it's got some of the most mind-blowing stories I've read.)
deleted 4y ago
[deleted]
acmegeek 4y ago
As others have mentioned, for all families, friendships, and relationships, is a good idea to establish a word or phrase that can verify someone is real and not a faked/AI voice. As the resources necessary to carry out a scam like this race towards near trivial, this will happen more frequently. I like the term "realword", like a password to determine if real. And of course, this word must be said in person, not over chat/text/email/etc. For most, over phone or videochat should be fine as well. Maybe we need some kind of public service campaign to add visibility to this threat and mitigation options like the realword? Maybe also encourage a spaced repetition habit to establish it?
Rudism 4y ago
I find it a little odd that an article which is, at least incidentally, about the dangers of sharing too much information about yourself publicly on the internet in this dawning age of AI-assisted scams, is itself so heavily plastered with unrelated random facebook and instagram photos of the victims.
- yeck 4y ago
  Yeah, I found that to be in poor taste and very unnecessary.
hn2017 4y ago
Why not link to the original source? https://www.wkyt.com/2023/04/10/ive-got-your-daughter-mom-warns-terrifying-ai-voice-cloning-scam-that-faked-kidnapping/ https://www.wkyt.com/2023/04/10/ive-got-your-daughter-mom-wa...
tonylemesmer 4y ago
Whats the dogs name? Max Hey Janelle. Whats wrong with Wolfy? Wolfy’s fine honey. Wolfy’s just fine. Where are you? Your foster parents are dead. (T2 judgement day)

198 points by agomez314 4y ago

The real enabling technology in this scenario is cryptocurrency. Cryptocurrency really changes the risk/reward ratio of scam (and real) ransoms.

sebzim4500 4y ago
I miss the days before 2008 when nobody was ever kidnapped, robbed or extorted. I'm pretty sure it is faster to withdraw £500,000 in cash from your bank than to set up a crypto trading acccount and go through KYC etc.

none_to_remain 4y ago

I already set up the family's secret "not a robot" password

deleted 4y ago

[deleted]

kroolik 4y ago

I'm really negative about the impact AI will have on the society. We have already been drowning in fake news and polarizing information. Now, the likes of DALL-E and DeepFake can generate convincing fake graphics. Chatgpt and the likes generate convincing fake news. Voice AI can generate convincing voice from small samples. If you were afraid of your elderly relatives being scammed by people pretending to be policemen or grandchildren, now more tech-concious people will get scammed by the voice and look of their relatives. Are we really approaching the reality where we need 2FA to trust the other person is really who they are?

icepat 4y ago
> Chatgpt and the likes generate convincing fake news It's not like fake news was a non-issue before ChatGPT existed. Breitbart, and other fake news sites existed for years before this was even imaginable. Fake graphics were around for ages too, with image manipulation, even before computers. Take for example the Surgeon's Photo of the Loch Ness Monster.
- jacquesm 4y ago
  This argument has been rehashed time and again and it's getting a bit boring: the fact that you can do something at all is qualitatively different from when you can do that same thing at scale.
  - chimpanzee 4y ago
    Agreed. As I’m sure you know, it’s easier for people to reason when the arena presents only discrete possibilities, especially binary ones, eg yes/no or black/white. The arena of the continuous, which encompasses most of the natural world, is far more difficult. It doesn’t allow for the sort of arguments where one can assert things with ego-boosting absolute confidence. Matters of scale naturally fall into the continuous sort, but scale can also introduce new discrete possibilities. Perhaps we need to present the new specific, discrete effects and outcomes that AI-at-scale will introduce. Maybe, just maybe, that’ll change minds of people who are truly open to being changed. My intuition though is that this argument is repeated here (on HN in particular) not for lack of knowledge or thought, but rather because they WANT to see the chaos that’ll result. They want all the positives and negatives, no matter the balance, simply because it’s exciting and adds to their “mundane” lives. Where the mundanity is of course subjective, simply the result of their default worldview, rather than anything nearing an objective description of the world as it is.
    - jacquesm 4y ago
      > simply because it’s exciting and adds to their lives. It's certainly exciting but I wouldn't bet on it adding to our lives just yet. Maybe. But many avenues leading to net negative outcomes are still part of the tree.
yamtaddle 4y ago
The "fun" thing about the tech in the current state is that it's wrong quite a bit, which puts a cap on how much of a productivity boost it can give any organization that cares about correctness or its reputation; meanwhile, it has nigh-unlimited productivity-boosting promise for organizations that don't care about correctness or reputation. Which kinds of organizations don't care about correctness or reputation? Scammers, spammers, and certain types of propaganda-spewing organization, especially those that puppeteer "grassroots" campaigns. We may give a single-digit multiplier in productivity to some roles in legitimate business, while giving a two or three digit multiplier to similar roles in harmful, parasitic organizations.
- smt88 4y ago
  I'm much more concerned about dictators than I am about scammers. Some countries have already successfully rewritten the history of less than 30 years ago with surveillance and media control. What will happen when they can also control all digital reality? They can invent an uprising of millions of people that never actually happened, and there will be photo, audio, and video "evidence". AGI is scary, but it's not immediate. I think companies like OpenAI are wrecklessly pushing us toward immediate societal collapse.
  - infogulch 4y ago
    I'm also very concerned about where AI will take us, but will bad actors just quit if OpenAI decides to halt? Or will they continue to develop it, but in secret, so we have no clue what its capabilities are or how to control it? I fear the only way out is through.
    - iamdamian 4y ago
      Couldn't you say this about nuclear research or bioweapons as well? Yet those are highly (and, seemingly effectively) regulated.
      - from 4y ago
LatticeAnimal 4y ago
... That is an interesting point. I wonder how long it will be before some enterprising developer hooks up an LLM to a HN account with the instructions to "blend in while promoting X". Maybe that AI already exists... It wouldn't take more than a day.
li4ick 4y ago
I'm also more on the negative side because I'm really not convinced by the whole "AI will just automate/remove the boring aspects of life". Every single prototype capability of current AI points in the direction of a worst-case scenario of overall misery.
- yoyohello13 4y ago
  Absolutely. I feel like I'm just watching the car crash in slow motion. Hopefully I'm wrong, but I really don't see this AI revolution working out well for humanity (in the short/medium term). I'm sure we will eventually get through it, but I think life for most is going to get real hard for the next 20-50 years.
- anonymouskimmer 4y ago
  > Every single prototype capability of current AI points in the direction of a worst-case scenario of overall misery. Software is often aimed at solving problems and making processes more cost-effective. Founders are often most interested in solving their own problems, problems of their friends, or the most valuable problems of their customers. For B2B companies to customers are other businesses. The greatest cost for businesses are employee costs. Solving the employee cost, but eliminating employees, is a great target for business owners.
morkalork 4y ago
I think we're passed due for some kind of 2FA for the phone. Sometimes my bank or credit card company will call and try to sell me shit and I tell them I have no proof of who they are hang up. Too bad for them.
backtoyoujim 4y ago
Like I'm going to carry photos of traffic lights and crosswalks everywhere I go.
sebzim4500 4y ago
I'm sure those will all be serious problems (or already are problems) but I think they pale in comparison to the potential upsides. This could be the biggest jump in productivity since the industrial revolution. Managed properly, it could lead to what is essentially a utopia.
- layer8 4y ago
  I don’t think the way economic and other incentives work are necessarily favorable to a utopia being established.
  - sebzim4500 4y ago
    As long as one country does it properly and lets people immigrate we are ok.
toss1 4y ago
>>Are we really approaching the reality where we need 2FA to trust the other person is really who they are? YES In fact, not approaching, but already passed the threshold. If you have any assets, it'd time to be sure you have a set of actually obscure un-guessable prompt & responses to verify identity in your family. And it needs to be better than "what's our first dog's name?" or "where do we vacation?" type stuff found on FB; if they're going to the trouble of getting images & voice samples to clone, they'll find that stuff. Now, it is not that everyone is famous for 15 minutes, it is that everyone needs to be up-to-date on security to avoid being randomly shot or scammed. Nice society we've built.
newhouseb 4y ago
I was having a conversation with some friends the other day about what schemes might mitigate some of these risks, something like an anti-safe word, i.e. a "danger word" that someone can be used to remotely validate that a loved one is authentically in danger. This is fairly low-tech and likely susceptible to various kinds of social engineering, but I'm curious what a more robust approach might look like that doesn't involve us all regurgitating 6-digit codes like robots all the time.
- theNJR 4y ago
  Yep, this is the way
- mLuby 4y ago
  Let's say you call or text me, requesting gift cards or my password or something else weird. I tell you I'll email you immediately to verify. I email you to confirm you want N $X gift cards. You do so. Or if the inbound contact is by email, you verify by text or phone call. Or Signal/Discord/FB chat apps, etc. Heck even LinkedIn Inmail could be your verification channel. You could still be compromised, but that'd require attackers to have significantly more access and readiness. And if you reach the other person and they're unaware, now you both have some support as you figure it out.
  - jamesshamenski 4y ago
    If one service was compromised, isn't it reasonable to think that the scammer can access other services during this breach?
confoundcofound 4y ago
> Are we really approaching the reality where we need 2FA to trust the other person is really who they are? Yes we are. We are approaching a world where people will not want to invest their time, energy, and emotion engaging with other supposed humans remotely unless they have verified their personhood / identity. I'm actually surprised we have not yet seen an entire industry built around human authentication. Seems like Apple is the only company taking this seriously. The standard of human interaction will either be meeting IRL or signing communications biometrically.
- NickC25 4y ago
  >I'm actually surprised we have not yet seen an entire industry built around human authentication. Seems like Apple is the only company taking this seriously. Completely agree - but to that - with how quickly AI voice cloning is improving, and how quickly image creation tools are improving, how long will it be until someone can sample my voice well enough to command Siri to do something and Siri won't be able to tell the difference? How long will it be until an AI generated photo of me is so realistic that it can unlock my phone via FaceID? I have a feeling that's going to happen in a shorter timeframe than we all think it will. I mean, my voice is somewhat similar to my father's, and if I'm around his phone, I can say "Hey Siri" and his phone will think it's his voice, not mine.
  - confoundcofound 4y ago
    We're going to see an arms race between AI and biometrics / authentication technology. My hope is that the latter outpaces the former, but it doesn't seem to be panning out that way. In the future, we might be only able to unlock our phone or command Siri when hooked up to a continuous biometrics sensor.
    - NickC25 4y ago
      You're probably not too far off from what will eventually become reality. Perhaps that's Apple's angle with the AR/VR headset it supposedly has in the works - if you've got your headset on, you can unlock your devices, and if you don't, you have to manually unlock devices via a password or PIN or whatever.
comfypotato 4y ago
How is AI different than any other technology in this respect? As far back as the discovery of fire, new technology has enabled more positive outcomes than negative. Some tech more so than others, I guess, but what makes you lean so negatively regarding AI? It’s already improved my life considerably. Just the basic ChatGPT web app has extended my capacity in multiple respects.
- puttycat 4y ago
  Rule of thumb: reductionist arguments don't hold for technologies from the Internet onwards. For example, the Internet is nothing like TV, so arguments like "we grew up on TV and ended up ok" don't hold. Same holds in my opinion to AI-scams.
  - bawolff 4y ago
    I disagree. Do you have examples to back that up? > we grew up on TV and ended up ok Not sure what you are comparing here, but we (people in early 30s) also grew up on the internet and mostly ended up ok. Some people who grew up on tv did not end up ok.
    - pixl97 4y ago
      Quantity is a quality in itself. If I catch a fish in a lake that's not problem. If I catch all the fish in the lake we need to stock the lake again with fish. If I catch 10^30 fish on the planet, fish are extinct.
      - ta1243 4y ago
        > If I catch 10^30 fish on the planet, fish are extinct. The entire planet is extinct
        pixl97 4y ago
        Oopsie, I turned the visible universe into paperclips.
simion314 4y ago
Maybe all countries need to finally give importance to this issues, put a few take scam crimes seriously, stop trading with countries that do not cooperate, put the scammers in jail. Tech evolved any idiot can get a copy of photoshop and some video software, we need tp solve the problem, otherwise is like preventing creation of email because we are too incompetent to address the spam problem.
marksmith2996 4y ago
Don't worry it'll probably just exterminate us anyway then fake news doesn't matter.
dave_sullivan 4y ago
Don't be negative about revolutionary new technology that can make the world a better place. As it stands, people believe whatever dumb thing they read online. Thanks to AI, they will have to learn that if something is out of the ordinary, they should confirm with multiple sources. I call this a net win. And for those literally medically incapable of this level of reasoning, we will soon have "AI firewalls" (Gibson ICE) that can tell people "Hey, this looks like a scam!" and also help them reason about complex topics. This is fantastic, what a time to be alive!
- kroolik 4y ago
  > NOW they will have to learn that if something is out of the ordinary I once shared this approach, but learned that the majority of people don't really want to leave their echo chambers and gladly accept anything that amplifies it. And the bad part about it is that the western world is based on democracy, and democracy is controlled by the majority. You may not care about people that are easy to manipulate, but in reality you are not the target as your voice won't matter at scale.
sandworm101 4y ago
>> Are we really approaching the reality where we need 2FA to trust the other person is really who they are? We are already there. In the kidnapping context, we have been there for a great many years. If someone say they have kidnapped my child, I will text/call that child immediately to verify. The 'second factor' is the realworld daughter. A kidnapper must both create a facsimile of the daughter and then also render the real daughter incommunicado. We don't need new 2FA because we already have it.
darkerside 4y ago
It's funny, technology was able to bring us together across geographic distances. Someone on the other side of the world was a phone call away. You can turn on your TV or phone and instantly tap into seeing, hearing, and reading people and their thoughts from around the globe. Almost as quickly, AI may unwind all of that. It lay become that there's more noise than signal across all of these technical media. The only thing you can trust, and the only thing that matters, is the people in front of you. Just like the old days.
deleted 4y ago
[deleted]
mr_mitm 4y ago
We need a PKI run by the governments. National ID cards that are smart cards. Cryptographically sign any and all digital communication. Self-signed certificates could still be used with TOFU. However, I'm not sure if total loss of pseudonymity is less of a horror scenario.
- layer8 4y ago
  You can still use pseudonyms with PKIs. The CA will know who you really are, but it doesn’t need to be public by default. And it can be regulated such that a judicial order is necessary to disclose your identity to a third party (including to the government).
- tommiegannert 4y ago
  I've been wondering what would be the event that causes digital signatures to become the norm. Though, I didn't expect it to be machine learning. I'm scared by how much is just done by phone with people who don't know me.
lbotos 4y ago
I think it will be addressed by "community". For the last probably decade (maybe more) every online social space has been pressured to serve global scale. That increases surface area, and increases more strangers. Smaller, more trusted rooms, will combat the threat of misinformation. The universe is built on waves/cycles. I think we are going to see pressures to make smaller rooms because of AI. I'm actually excited for smaller, more intimate spaces, with trusted people who now have 10x ability from the powers of AI. That seems like a LOT of fun.
dclusin 4y ago
Ai generated news has actually been a thing for a while now. The news agencies that use it still do fact check the output.
ravenstine 4y ago
Your concern is understandable and relatable, but I approach the issue from a stance that is, while not necessarily optimistic, definitely is less pessimistic. I don't think we truly know just yet how negative an impact AI will have on society. Every time there has been a technological leap, people have panicked over what the gizmo of the now will do to society. Again, not an invalid concern, but society has yet to have been blown apart by anything. Also, everything potentially being just AI may inadvertently get the public to do the right thing. People should never have been as trusting of authority figures or institutions in the first place. If everyone assumes that everything is likely to be complete hogwash, which was already true in many cases, then they may not just swallow everything as fact. Maybe fewer people will blindly consume news, which is a good thing; 99% of news is not actionable or good for an average person's well-being. And if enough ransom demands turn out to be AI-generated scams such that the real ones are overall far less successful, it's possible that fewer people are kidnapped for ransom in the first place. I'm not saying I know any of this, but rather the opposite.
- yeck 4y ago
  In a world where people need to be constantly paranoid, I think many will develop true paranoia, or just give up trying to understand what is going on and defer to who has the power already. Certainly it is not possible for individuals to seek truth on all things on their own, even if it was their full time responsibility to attempt so.
onemoresoop 4y ago
> If you were afraid of your elderly relatives being scammed by people pretending to be policemen or grandchildren, now more tech-concious people will get scammed by the voice and look of their relatives. Are we really approaching the reality where we need 2FA to trust the other person is really who they are? I thought I was immune to certain scams where the scammer accent was a dead giveaway that they were scammers. Now with capabilities of AI I feel somewhat vulnerable again though Im far from being elderly.
ZainRiz 4y ago
The past century has really been an anomaly when it came to trusting news, with photos and videos seeming to be reliable proof. Before that, societies had to be structured very differently to account for the lack of proof. You'd hear people from out of town talking about what they saw in a neighboring city. You'd need to judge how trustworthy the person was. People would expect a chain of narration, to understand how _that_ person came to learn a bit of information (or if they claimed to witness it first hand). As AI generated content becomes more popular, I predict we as a society are going to go back to relying more and more on the reputation of the speakers in question. Who might you consider reputable? People you've met personally, people who your community respects, and of course, influencers who you follow who's persuasive words match your pre-existing world views
- Georgelemental 4y ago
  This change will likely advantage established institutions, no? It will be much harder for a whistleblower, random, etc to prove something that the powers-that-be would prefer be kept quiet.
MagicMoonlight 4y ago
It’s going to be great. The internet will be a lifeless sack of shit full of bots and we’ll all have to meet up in person for true conversation. We can have parties again and have friends instead of depression.
bcrl 4y ago
Of course the dominant 2FA methods will be those that are easily forged, so the general public will still get swindled (I'm looking at you banks that are insisting on using SMS to "verify" my identity). It's as if nobody cares about the world programmers are creating.
1letterunixname 4y ago
This Michael Bay movie script writes itself: 1. A not exactly rocket scientist POTUS takes office in 2024. Let's suppose they were a populist, triumphalist, religious, agro, law-and-order DINO to make it interesting and fictional. 2. Rogue AI launches a social engineering communication offensive against CENTCOM impersonating generals, cols, and ltcs against enlisted and lower ranks to carry out a first strike against China with a digital hallucination that Taiwan is being "attacked". It uses details gleaned by wiretapping the upper echelon to circumvent normal N-person keying rules and authentication protocols. WarGames 2: Joshua Wins

morkalork 4y ago

Is hornswoggled a common word in the USA?

jacquesm 4y ago
I read it as 'to be cheated'. And I'm nowhere near native level, I think the link for me is the swoggled mentally links it to swindled.
smolder 4y ago
I've never heard it anywhere as a USian. It seems the type of obscure and redundant word that people bring out once per decade. Without looking it up, I could guess horn refers to phone and swoggled is another swindled, so to hornswoggle is to phone-scam.
nostromo 4y ago
No. Scammed, tricked, or even bamboozled would be much more common.
the_jesus_villa 4y ago
No, not common at all
oidar 4y ago
Not with the current generation of people. I can't speak to any other generation, but as a native speaker I did understand it. It seemed archaic to me.
- bentley 4y ago
  I wouldn’t call it archaic, but more folksy. I think most people would understand it, but would rarely encounter it outside a humorous context. It’s pretty similar to bamboozle, except that’s gained kind of a “cute” connotation in recent years (due to pet memes, etc.) that hornswoggle doesn’t have.
sharps1 4y ago
Not that I know of. I’d guess probably came across it randomly and liked the word. Seems like it comes up in crosswords https://crosswordtracker.com/clue/hornswoggle/ https://crosswordtracker.com/clue/hornswoggle/ Also funnily not related is a WWE stage wrestler name who had a YouTube video done in them a year ago. https://simple.wikipedia.org/wiki/Hornswoggle https://simple.wikipedia.org/wiki/Hornswoggle.

LinuxBender 4y ago

I would personally always demand proof of life regardless if fake or real and I would ask my family member to answer a question only they would know the answer to. We would of course have code words ahead of time. Maybe this is a sad anecdote, but the code phrase my mom and I had for duress was "I love you"

urbandw311er 4y ago
What would you say if you just wanted to say “I love you”?
- LinuxBender 4y ago
  I would just give her a hug in person.

PragmaticPulp 4y ago

From the article: > DeStefano found the voice simulation particularly unsettling given that “Brie does NOT have any public social media accounts that has her voice and barely has any,” per a post on the mom’s Facebook account. > “She has a few public interviews for sports/school that have a large sampling of her voice,” described Brie’s mom. and > Then, DeStefano remembered that her 15-year-old daughter Brie was on a ski trip, so she answered the call to make sure nothing was amiss. No public social media accounts combined with the timing of the fraud happening during a ski trip implies that somebody close to the family was in on it. It's possible that they used the public sports interview and got randomly lucky with the trip timing, but I'd be looking much closer to home for a suspect.

jacquesm 4y ago
It always pays off to look for suspects close to home but I would definitely not rule out an outsider that monitors social media.
- PragmaticPulp 4y ago
  > but I would definitely not rule out an outsider that monitors social media. The part I quoted claims that she doesn't have any public social media, so that wouldn't work. That's assuming we're getting the real story, of course.
  - sebzim4500 4y ago
    I still think it is way more likely that someone posted about he trip on social media. There is basically no benefit to targeting someone you know with this sort of scam, and the risk is higher.
  - jacquesm 4y ago
    When people say they 'don't have social media' it usually is a good idea to check what their definition of 'social media' is. If they have a web presence that they regularly update it doesn't really matter what the label is, it can likely be used to infer a whole bunch of stuff. I know quite a few people who claim not to have 'social media' and yet they are on LinkedIn etc. Maybe I should have used 'web presence', but I think that what I intended to convey is clear: if you have a regularly updated online identity then you are giving an impostor a lot of useful information. If there is absolutely nothing about this person online, including that they don't use any of the video chat services, no tiktok videos etc then yes, the circle of suspects would narrow accordingly. But most teens are quite active online, even if their parents aren't always aware of it.

xen2xen1 4y ago

Was literally wondering about that while watching a Netflix show. Nothing can be trusted.

awb 4y ago

Proof of identity is going to be a huge opportunity. Images, text and voice can now be spoofed with minimal cost and effort. With the progress of deepfakes and text to video, how much longer until you can spoof video calls? Meeting in person is not practical in many scenarios. Anyone know of any promising ideas or companies in this space of digital trust?

Nathanba 4y ago
it's not any harder than any website login. The only thing is that this isn't available as a simple service to normal people, maybe this is a feature Facebook should offer to have more value: your kids can sign in via biometrics on their phone and then it shows their parents on their facebook account that the child used biometrics X seconds ago to provide proof of identity.
- justrealist 4y ago
  > it's not any harder than any website login Have you ever gone though a password reset process with a bank or 401k? They can't just say "oh, you lost access to your email? sorry, your money is gone". They rely on phone calls, documents, security questions... all things which are very susceptible to programmatic social engineering.
  - Nathanba 4y ago
    I dont see how it matters, this is for the usecase that there is already an established and verified account with e.g Facebook. Many betting or crypto sites do passport and video verifications, Facebook can do the same. Then all the child has to do is login via the Facebook app as usual, on iPhones via biometrics but it doesn't have to be biometrics. It can be any kind of 2fa or tripple verification. Authenticator code + email + sms + app login, simply cutting down on the likelihood that someone is a fake person. Then the parent can see which verifications were done by the child on the Facebook dashboard to see if the kid is alive and able to do verifications instead of whatever else the attacker claims. I think you meant to say that the bank verification process can be socially engineered... well first of all to actually do that you also need a lot of fake documents and expertise and the bank has to utterly fail at their job. Which hopefully prompts governments to require in person meetings at the offline location for future bank accounts.

larsiusprime 4y ago

My question is if this will eventually backfire on kidnappers, because if this escalates enough if might create the expectation that it's not real, so a genuine kidnapper can't distinguish themselves credibly from a virtual kidnapper, because what previously was credible proof of life (the voice of the victim) is no longer so.

dmix 4y ago
Do random people even get kidnapped for ransom anymore? It seems like technology and risk/reward killed that one pretty quickly. At least in the west and places with stable governments/law enforcement.
- Beltiras 4y ago
  It's hard to find good data from the US due to how statistics are collected. upwards of a million children go missing each year.
  - dmix 4y ago
    That stat doesn't provide much insight when asking about ransom or the sort of criminal schemes we're talking about.
    - Beltiras 4y ago
      I know. If only the data were collected at all so we could have a discussion about kidnapping in the USA....
  - from 4y ago
    That is almost entirely runaways and of the small fraction that isn't they are usually abducted by other relatives. About 99% of them return home [safely](https://www.ojp.gov/pdffiles1/ojjdp/196465.pdf#page=6 https://www.ojp.gov/pdffiles1/ojjdp/196465.pdf#page=6). Taken (2006) style kidnappings basically do not happen. Edit: https://www.missingkids.org/ourwork/impact https://www.missingkids.org/ourwork/impact says there were 142 "nonfamily abductions" in 2021 and 121 of those were resolved.

alex_young 4y ago

AI and deepfakes are very popular subjects right now, but is there any evidence that either was used here?

yeck 4y ago
I am not sure it even matters either way. It is certainly possible to do things like this right now, and "kidnapped" or "arrested" family member scams are well established as existing. The significance of this report is less that it happened, and more that it is an illustration of what is not only possible, be very cheap and easy to do.

deleted 4y ago

[deleted]

stbullard 4y ago

Nothing in this article gives any reason to believe that the voice on the other end of the phone was AI-generated. Occam’s razor suggests it was more likely a human pretending to be her daughter. I’m guessing after she realized that the voice wasn’t her daughter, the mother convinced herself it must have been a deepfake to explain herself having been so easily convinced.

dontupvoteme 4y ago
This is likely as currently only elevenlabs has the publically available technology to do this and is SaaS only (and i presume requires law enforcement traceable usage after that first weekend and 4chan's fun with it)
- 015a 4y ago
  I've played with elevenlabs' solution; its really good, but I don't think it could believably emulate the voice of someone under panic duress. The voices it produces have a pretty believable, but flat, affect. Maybe it has options I haven't played with, or maybe you could train it to add panic to the voice. Idk, something about this story feels fishy to me. I don't doubt that we're headed toward this kind of future; but we're talking about very high level spear-phishing to accomplish this today (knowing the individual isn't home, having samples of their voice, highly sophisticated AI knowledge to make the voice and add elements of panic duress, to actually succeed you'd also need to compromise lines of communication to the individual, e.g. do this while they're on a trip to the amazonian rainforest or something). This feels more like an AI hit-piece than an actual done-with-AI story.
  - neom 4y ago
    Watch this to the end: https://www.youtube.com/watch?v=hYQmF_QLR38 https://www.youtube.com/watch?v=hYQmF_QLR38
- yeck 4y ago
  I did a quick search and found this among a list of many: https://voice.ai/ https://voice.ai/

neom 4y ago

In the past month, CBC has been full of news reports of Canadians getting screwed by AI voice scams, mostly old folks: https://www.youtube.com/watch?v=ALVb6UlGURk https://www.youtube.com/watch?v=ALVb6UlGURk https://www.businessinsider.com/couple-canada-reportedly-lost-21000-in-ai-generated-voice-scam-2023-3 https://www.businessinsider.com/couple-canada-reportedly-los... https://www.youtube.com/watch?v=hYQmF_QLR38 https://www.youtube.com/watch?v=hYQmF_QLR38 https://www.youtube.com/watch?v=5wGB_jF05JU https://www.youtube.com/watch?v=5wGB_jF05JU https://www.youtube.com/watch?v=SbBTLDDP8Eg https://www.youtube.com/watch?v=SbBTLDDP8Eg https://www.youtube.com/watch?v=Lwylb4cE1b8 https://www.youtube.com/watch?v=Lwylb4cE1b8

deleted 4y ago

[deleted]

motoxpro 4y ago

Just makes me even more upset about that company that YC funded that had a Launch a few weeks ago that let’s you do EXACTLY this. :(. Been telling my family to come up with a word/phrase to authenticate voice. I.e. floorboard

JCharante 4y ago
Adobe demo'd this 10 years ago and never launched the product

ineptech 4y ago

Man, do we need a way to prove you are who you say you are that doesn't involve a giant corporation. I feel like there was a window where we could've organically coalesced around something open source, like Keybase, or a government oauth run by 18F, or MIT's PGP keyring or something, but the moment has passed and the only identity service you're going to realistically get a majority of people to adopt is for-profit social media bullshit, which is incentivized to stay fragmented and not really address problems like this.

elicash 4y ago

Apple/Google should allow people to answer phone calls from strangers with a voice changer. It's way too easy to clone someone's voice and cause harm. Alternatively, they could notify you if they detect a faked voice, but maybe these two ideas are contradictory. Another solution, of course, is to not pick up the phone from unknown numbers unless you're expecting a food delivery or something.

jamilton 4y ago
That would cause more harm than it would prevent. People calling from other people's phones is a common and necessary thing. Trying to detect faked voices would make sense.

jschveibinz 4y ago

Remember to pre-arrange a shibboleth (code word or phrase) with your families just in case this happens to you.

deleted 4y ago
[deleted]
segh 4y ago
If you've failed to pre-arrange, you could always just ask something personal. "Sorry Sally I know you've been kidnapped, but before we continue, what was the name of our first dog?"
- sebzim4500 4y ago
  Can you come up with a question that definitely isn't answered on social media while you are in a high pressure situation?
dontupvoteme 4y ago
[flagged]

kleer001 4y ago

For those who worry this is the end of safety for all mankind I'd like to point you towards this quote from the Victorian era "Rogues are very keen in their profession, and know already much more than we can teach them." Alfred Charles Hobbes - 1851

mquirion 4y ago

This is very concerning, but I think my bigger, more immediate worry - and one I suspect will happen at a much larger scale in the US, if not the world - is the use of these tools in bullying, particularly among school-aged kids. The detrimental effects on kids (and thus, eventual adults) due to the misuse of these tools could be cataclysmic.

deleted 4y ago

[deleted]

linsomniac 4y ago

Looks like it's time to set up a family sign/countersign for authenticating each other.

deleted 4y ago

[deleted]

IvyMike 4y ago

Reminds me of Greg Egan's "A Kidnapping" from "Axiomatic". (I'd like to describe it, but it would spoil it. Just go read Axiomatic, it's got some of the most mind-blowing stories I've read.)

deleted 4y ago

[deleted]

acmegeek 4y ago

As others have mentioned, for all families, friendships, and relationships, is a good idea to establish a word or phrase that can verify someone is real and not a faked/AI voice. As the resources necessary to carry out a scam like this race towards near trivial, this will happen more frequently. I like the term "realword", like a password to determine if real. And of course, this word must be said in person, not over chat/text/email/etc. For most, over phone or videochat should be fine as well. Maybe we need some kind of public service campaign to add visibility to this threat and mitigation options like the realword? Maybe also encourage a spaced repetition habit to establish it?

Rudism 4y ago

I find it a little odd that an article which is, at least incidentally, about the dangers of sharing too much information about yourself publicly on the internet in this dawning age of AI-assisted scams, is itself so heavily plastered with unrelated random facebook and instagram photos of the victims.

yeck 4y ago
Yeah, I found that to be in poor taste and very unnecessary.

hn2017 4y ago

Why not link to the original source? https://www.wkyt.com/2023/04/10/ive-got-your-daughter-mom-warns-terrifying-ai-voice-cloning-scam-that-faked-kidnapping/ https://www.wkyt.com/2023/04/10/ive-got-your-daughter-mom-wa...

tonylemesmer 4y ago

Whats the dogs name? Max Hey Janelle. Whats wrong with Wolfy? Wolfy’s fine honey. Wolfy’s just fine. Where are you? Your foster parents are dead. (T2 judgement day)