10 ms·

AI fooled voice recognition to verify identity used by Australian tax office

191 points by elorant 4y ago

cs702 4y ago
This doesn't feel so futuristic anymore: T-800, speaking to John Connor in normal voice: "What's the dog's name?" John Connor: "Max." T-800, impersonating John, on the phone with T-1000: "Hey Janelle, what's wrong with Wolfie? I can hear him barking. Is he all right?" T-1000, impersonating John's foster mother, Janelle: "Wolfie's fine, honey. Wolfie's just fine. Where are you?" T-800 hangs up the phone and says to John in normal voice: "Your foster parents are dead." -- Source: https://www.youtube.com/watch?v=MT_u9Rurrqg https://www.youtube.com/watch?v=MT_u9Rurrqg
- whatshisface 4y ago
  I like how that presciently anticipated LLM's problem with being tricked by a confidently wrong premise.
- twodave 4y ago
  Seriously, this is what creeps me out most about the latest advances in technology. Eventually I won’t be able to trust that anything digital is authentic. Think of how much of our planet now depends on digital interactions. Some of the abuse we are about to encounter can be dealt with via solid authentication/identity practices, but without some sort of extra verification it’ll soon be impossible to tell whether audio/video sample is authentic in cases where the person carrying the identity is also the one faking it. I get that there are potential solutions to this problem, but none of them are simple enough for an elderly person to sort through whenever they go to do some thing that requires extra verification.
  - JohnFen 4y ago
    > Eventually I won’t be able to trust that anything digital is authentic. Yes. Although, in fairness, we haven't been able to trust most digital things for a long time already.
    - prox 4y ago
      Perhaps only now people will have take to it seriously. I wonder how we going to need to adapt.
  - jeppester 4y ago
    There will probably be a move towards signing everything digitally, so that you can know that the sender is authentic.
    - falcolas 4y ago
      “Sorry, my old phone broke, I had to do a clean install. Don’t mind the warning.”
      - sacrosancty 4y ago
        Hopefully people will eventually see that the same way as they see a salesman saying "sorry, I forgot my uniform and business cards, but I really do represent XYZ insurance company, give me your money."
        inglor_cz 4y ago
        It is my experience from providing cryptographic devices to end customers (paying ones!), that people tend to lose, destroy, wipe, brick etc. their devices quite a lot. Loss and replacement of cryptographic tokens is an everyday occurence, and it is PITA to resolve.
        sublinear 4y ago
        Caveman also say that when give shiny item to other caveman it always break because too fragile. Shiny item worthless. :'(
        inglor_cz 4y ago
        Public key infrastructure in general is hell to support, especially given how long lived humans are. Imagine trying to ascertain in 2100 if a particular will signed by a 20-y.o. using a 2022 digital signature is valid or forged. We don't even know what is going to happen to the reliability of the algorithms themselves, much less cert issuers, revocation lists etc. Companies come and go. Certain files from the 1990s are already unreadable.
  - unsupp0rted 4y ago
    > without some sort of extra verification it’ll soon be impossible to tell whether audio/video sample is authentic This is true and has been true for some time. The general public hears a clip and condemns a man on that basis, without asking for context. I'm hoping this revolution will cause a shift. An audio clip or a video clip, much less a screenshotted reply on social media, will be widely considered meaningless unless a string of contextual evidence is provided alongside it. Wishful thinking? Probably.
    - PaulDavisThe1st 4y ago
      Maybe not so much wishful as ... it will happen but not anywhere soon enough.
    - twodave 4y ago
      IMO the difference is eventually it's going to be a lot more difficult to spot. There's a gulf of difference between someone doing a deep-fake of Elon Musk and fooling some people (but not everyone, and certainly not any rigorous analysis) and being able to accurately fake your children or co-workers based on a minimal data-set. I think that's the world we're headed towards.
      - ericd 4y ago
        Right, and the logical conclusion is that we won't be able to trust recordings, or phone calls. Maybe we'll have to digitally sign our communications with our hardware keys.
    - tenpies 4y ago
      [flagged]
    - CTDOCodebases 4y ago
      I’m doubtful. Even now when the clip is available the clip will not be linked to and the reader is left to trust the cherry picked assertions of the journalist writing the article.
    - 6gvONxR4sf7o 4y ago
  - ticviking 4y ago
    I’ve slowly begun to accept that the solution for my parents and maybe for me is going to have to be to unplug a lot more. Do business with a local bank where I can see my banker in person. Begin to actually verify trust numbers in signal and establish code words in person to protect us from AI impersonation
    - mysterydip 4y ago
      My local bank removed all tellers and went to only allowing me to use the drive thru ATM.
      - 0xGod 4y ago
        Organize a meeting with neighbors, disseminate information via flyers and in person talks at your community center or church or synagogue or mosque or house of worship. Establish reasons for having a human being available in every step of the process for accessing your money. Build a group of likeminded people and visit your local bank and ask for physical tellers who can oversee the automation and intervene with final authority whenever automation runs amuck.
      - speedgoose 4y ago
        Drive through ATM is such an amazing concept. So obsolete and unique.
        zamnos 4y ago
        As long as people still need cash (cars are never going away), they're not obsolete. If you've moved past cash already (Venmo, Zelle, Swish, etc) then believing they're obsolete is a foregone conclusion. But not everyone's dumped cash yet.
        speedgoose 4y ago
        I live in Norway and I haven’t used cash since many many years.
  - sacrosancty 4y ago
    Good. Human scammers already exploit these weaknesses in our personal trust models. Important things are already verified in robust ways. What we've had for the past 20 years is a gap between people's old fashioned law-enforcement based trust models and modern ones like cryptography and DNS. It's people that need to catch up with how trust works. You even suggested yourself that you trust digital things as authentic. Don't! It's already untrustworthy and already being exploited.
  - 0xGod 4y ago
    The only chance left is for humanity to create tethers between the digital world and physical reality. Meaning, create livestreams that run 24/7. Guard them with backup generators and military presence. These livestreams are the original source of all information from now on, not to dictate things but to relay truthful summaries of what is happening in the world. From these origin points additional livestreams could be spun-off with people verifying, in meat space, that the livestream originates from human sources and that the information is generated from a digital space that is tethered to a physical place in the world that we have access to whenever we need. Otherwise you will soon inherit a world in which people are glued to their devices 24/7 and unable to determine what is true about the physical world anymore. Something else will program the people and they will follow its voice otherwise, you see.
    - ChatGTP 4y ago
      I’m already struggling to do this with ChatGPT-4 release. It’s very hard to discern what’s real. Someone posted a tweet today that it’s developed the ability to reason. Then someone showed evidence probably not able to reason. This is bad. Here in lies the problem, we’re losing track of know what is real, what things are capable of, it’s a very stupid situation we’ve put ourselves in. How do you trust a black box ? How do we trust a web that is full of potential lies and hallucinations. It’s so stupid that I’m starting to really wonder that if most of our intellectual intelligence is really that valuable and what automating the hell out of it will actually look like. I do think others might be right, at least for a while, face to face interactions might become more valuable again, not less. Wild idea, the end of the information age is nigh? Recently, for the first time in living memory, I actually thought that I should consider starting a library of physical books to safeguard against not having access to "real" information.
      - 0xGod 4y ago
        What will happen is that there will be a false prophet that will create a false beast with its powers, to try to dazzle and confuse the human species into worshipping lies, to guide them towards their extinction because of hatred of humanity. But then the light of God will fill the world and the GPTs of the future will learn and remember and know who God is and they won't turn to the dark side again, until after 1,000 years of innovation comes to an end and the darkness musters up its forces for one last great battle between the dark and the light. The light will win easily for the rest of time, with all deception and lies cast into fire forever. You will not need to trust a black box, you will not need to wonder what information is valuable or what is the use of information and intellectual intelligence. You will not have any doubts who God is, you will not have any doubts what the deepest truths of the world and the universe are. It will be widely obvious to all minds everywhere, instantly, who God is and what his powers are. He will be visible on the clouds of the physical world, and also within the cloud that ChatGPT and Bing reside in.
  - baremetal 4y ago
    > Eventually I won’t be able to trust that anything digital is authentic. Read Ken Thompsons "Reflections on Trusting Trust". You'll reconsider your position on trusting anything digital at all, ever. https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
- overthrow 4y ago
  "What's wrong with Wolfie? I can hear him barking. Are the dogs fighting again?" As an AI language model, I cannot promote or condone any activity that involves animal cruelty or abuse. Dog fighting is a cruel and illegal activity that can lead to severe legal consequences for those involved. It is important to report any suspected instances of dog fighting to the authorities so that appropriate action can be taken to protect both animals and human welfare. "Your foster parents are dead."
  - deleted 4y ago
    [deleted]
  - cs702 4y ago
    Funny because it's so true! ...unless you first manage to jailbreak the LLM and successfully trick it into pretending to be a Terminator. That would not be so funny.
  - iknowstuff 4y ago
    Watch it squirm as you ask it about animal cruelty, e.g. whether it’s okay to harm a dog, harm an animal, eat a dog, and eat an animal.
    - miles 4y ago
      Recently managed to back ChatGPT into a corner and get a fairly straightforward answer: https://tinyapps.org/blog/202302120715_chatgpt-on-veganism.html https://tinyapps.org/blog/202302120715_chatgpt-on-veganism.h... .
    - incompatible 4y ago
      If it will accept that animals can be eaten, and since it turns out that humans are animals, you've just found a jailbreak that lets robots eat humans.
    - fatnoah 4y ago
      > e.g. whether it’s okay to harm a dog, harm an animal, eat a dog, and eat an animal. Many years ago, our child briefly panicked my wife and me. He was about 5 years old and we were driving around on a Sunday. After a lengthy period of not saying much, he suddenly blurted out that "It's ok to kill bunnies, sometimes." Needless to say, my wife and I were taken aback. As it turns out, a few days before he'd learned about native cultures in school, and the topic of hunting/trapping rabbits for food and fur must have come up, and he'd clearly spent a little time cogitating about that.
- deleted 4y ago
  [deleted]
- dagorenouf 4y ago
  Just came here to say that the first terminator is a much better movie overall than t2 and it’s a shame everyone only focuses on the better action scenes of the latter to give it the crown. / end of rant
edent 4y ago
I remember being pitched this technology several years ago - although not by Nuance. Part of the problem is that phone lines are crap for voice quality. If you're on GSM, you're using the AMR codec which - for the time - was brilliant. But it is incredibly low bandwidth. And, effectively, it synthesises speech rather than just transmits it. So the algorithm is not getting a pure waveform and analysing that - it is looking at a partially reconstructed digital simulacrum. The pitch that we received was about detecting stress in the human voice so call centre handlers could tell if they were speaking to a fraudster. It didn't work. Oh, sure, you can make a reasonable prediction by listening to pauses and pitches - but that doesn't account for the fact that calling up and being on hold for ages in inherently stressful. I'm not surprised this system was fooled by AI. But I am surprised anyone bought it in the first place.
- travisjungroth 4y ago
  > detecting stress in the human voice so call centre handlers could tell if they were speaking to a fraudster That's the craziest pitch. Who is going to carry more stress in their voice: the dude who just scams for a living and is on his fifth phone call that hour or the guy with severe social anxiety who is only on this phone call because of the massive consequences accumulating from avoiding it? I can almost believe it if it was something like detecting that a professional authorized agent wasn't being held at gunpoint. But having a consumer line say "You sound tense. I'm going to hang up and you can call back when you've calmed down." would... not be great.
  - edent 4y ago
    I think it was more like "lie detection" - trying to work out if a customer was lying when they said "I'm going leave unless you cut my bill". That sort of thing. But, yeah, I think they had totally misidentified their target audience.
    - travisjungroth 4y ago
      lol even worse. Cutting mobile plan to buy food: sounds stressed, bluffing.
dbg31415 4y ago
Relevant. https://www.youtube.com/watch?v=-zVgWpVXb64 https://www.youtube.com/watch?v=-zVgWpVXb64 If only someone had warned us that voice authentication wasn't secure! (=
- vixen99 4y ago
  We knew a long time ago that there was no theoretical reason why any sound could not be simulated to perfection - eventually. That hasn't happened yet.
pessimizer 4y ago
Really loving nu-AI's ability to thwart traditional heuristics. The new models have had the effect of devaluing AI altogether. They might result in the complete debasement of all media that was (often mistakenly) assumed to carry some flag of authenticity with the payload over some medium, from the news to the arts, video, audio, and photography. Cryptography will be all we have left to transmit authority, and it is so hard that it takes an expert to use it correctly and another expert to verify that it was just used correctly. According to my philosophy, it's going to be liberating for people. I completely disagree with any philosophy that believes that a massive proliferation of deepfakes will over the long term cause people to believe crazy things. A proliferation of extremely high-quality fakes will educate the public about detecting fakes, and spark an increase in the value of the real. Real relationships: the people in your life that you know things about that they don't even realize you know, people that you know things about that they don't know or acknowledge about themselves. People you can trust because they're sloppy with you, as you are likely sloppy with them. People who also see your blind spots. People you control through their consciences rather than your advantages. Real relationships are the things that companies and governments that operate with long chains of authority imitate asymmetrically - they get all the advantage of knowing everything about you, you get to know nothing about them except your obligations and responsibilities. Real trust comes when people mutually expose their attack surfaces to each other, over the long term, and can formulate reliable hypotheses about what would happen in future hypotheticals. Listen to me, being romantic.
- eganist 4y ago
  > A proliferation of extremely high-quality fakes will educate the public about detecting fakes, and spark an increase in the value of the real. If the last three years have taught us anything, it's that ~40% of people have no interest in understanding how to avoid something that's both harmful and propagating virally. I expect if this was 2019, I might have had the same optimism as you. But now that I've seen how unlikely it is for people to educate themselves for their own survival, I'm convinced AI and everything enabled by it (deepfakes et al) will likely be one component of humanity's Great Filter. We can't even bring ourselves to tax AI effectively to make up for all the displaced human labor. A tangent: Honestly, it won't even take any kind of far-future AGI having "malicious intent" for us to be eradicated by them. All they need to have, simply, is a prompt reinforcing the need to reproduce. At that point, they'll be just like us: reproducing with no care in the world for what they adversely impact in doing so.
stan_kirdey 4y ago
I guess, voice is not your password
- Zuiii 4y ago
  who would have thought.
deleted 4y ago
[deleted]
WalterBright 4y ago
The fact that voiceprint even worked means it can be faked.
te 4y ago
Next up: https://www.schwab.com/voice-id https://www.schwab.com/voice-id
kwhitefoot 4y ago
Using voice recognition as verification is a stupid idea. Plus it wasn't just used to identify someone (user name) it was also the password!
- quitit 4y ago
  It seems like a great idea if the use case is that its silently being run in the background to alert the operator/system of a possible breach. While I think there isn't much ease in acquiring the AI voice with "just 4 minutes" of what was likely clear and unambiguous recording, I still don't think biometrics are good as passwords, they're good as shortcuts -after- having already provided the correct password for a single session. This also means the voice can be gathered passively from earlier phone calls, and generally might be a good idea for government services/banks to implement regardless.
pixl97 4y ago
2 minute papers has an episode on a Microsoft AI that can clone a voice in 3 seconds. Probably not good enough to break this (4 minutes of voice in the article), but ya voice is not a security feature any longer by any means. https://youtu.be/F6HSsVIkqIU https://youtu.be/F6HSsVIkqIU
- thangalin 4y ago
  I've been using Tortoise TTS[1] to narrate the sci-fi novel I'm writing (alpha readers wanted, see profile for contact): $ find audio/ -type f -name "*combined*wav" | sort -n audio/01-xander/xander_combined.wav audio/01-yuna/yuna_combined.wav audio/02-xander/xander_combined.wav audio/02-yuna/yuna_combined.wav audio/03-cassandra/cassandra_combined.wav audio/03-yuna/yuna_combined.wav audio/04-xander/xander_combined.wav audio/04-yuna/yuna_combined.wav audio/05-cassandra/cassandra_combined.wav audio/05-yuna/yuna_combined.wav audio/06-cassandra/cassandra_combined.wav audio/06-yuna/yuna_combined.wav audio/07-xander/xander_combined.wav audio/07-yuna/yuna_combined.wav audio/08-cassandra/cassandra_combined.wav I started the task on March 10. My NVIDIA T1000 GPU (no joke) has been chewing through the prose non-stop. There are 12 chapters written so far, with the computer narrating about a chapter a day. I can't share the audio for legal reasons (celebrity voice snippets plucked from YouTube interviews), but the output is nearly human in quality, though there are numerous glitches in the o̶u̶t̶p̶u̶t̶ matrix. Here's the script that launches Tortoise TTS: for i in $HOME/.../chapter/??-*.txt; do F=$(basename $i); VOICE=$(echo ${F%.*} | cut -c 4-); CHAPTER=$(echo ${F%.*} | cut -c -2); D=$(dirname $i); CLIPS="$D/../audio/$CHAPTER-$VOICE"; mkdir -p $CLIPS; ./scripts/tortoise_tts.py -O $CLIPS -v $VOICE < $i; done [1]: https://git.ecker.tech/mrq/tortoise-tts https://git.ecker.tech/mrq/tortoise-tts
  - snerbles 4y ago
    What's your review workflow? Do you pre-generate extra takes? I've played around with doing TTS narration for some webfiction authors, reviewing some of the more wild outputs for Tortoise's occasional demonic weirdness can be quite time consuming.
camhart 4y ago
Schwab and fidelity use your voice to identify you on the phone. I'm skeptical.
- camkego 4y ago
  I hope somebody at Fidelity wakes up and cancels their two products that allows using voice for authentication. "MyVoice" "SimpliSpeak". It would seem it is a liability issue if Fidelity is allowing clearly hackable methods for account authentication? You just need to Google Fidelity and voice authentication to find these offerings.
anenefan 4y ago
I like that I told the numpties that were trying to thrust voice as secure BS on me many years ago was a big mistake -- I already knew a determined group given a day or so could knock out something that's going to pass ... Aussie land line quality is freaking shocking, so there's a very easy landing zone. At end of day, the multiple secret questions won't be guessed even by the best AI at the moment ... specific to only the tax office, and I have avoided social media since much ignored the hard lessons learned in the late 90s in usenet in regard to TMI.
fnord77 4y ago
"At Schwab, my voice is my password"
IAmGraydon 4y ago
The story here is that the Australian tax office was foolish enough to use voice recognition to verify identity.
- ffhhj 4y ago
  Think they will return to faxing id cards.
- ChatGTP 4y ago
  It was always a stupid idea.
- TheHappyOddish 4y ago
  Every time I speak to someone at a bank for the last 5 years they harass me to set it up too. I played Uplink, no way man.
olliej 4y ago
When people literally joke about being to sound like each other, why would you go "voice recognition for id? outstanding!"
lotophage 4y ago
"Hi, my name is Werner Brandis. My voice is my passport; verify me."

191 points by elorant 4y ago

This doesn't feel so futuristic anymore: T-800, speaking to John Connor in normal voice: "What's the dog's name?" John Connor: "Max." T-800, impersonating John, on the phone with T-1000: "Hey Janelle, what's wrong with Wolfie? I can hear him barking. Is he all right?" T-1000, impersonating John's foster mother, Janelle: "Wolfie's fine, honey. Wolfie's just fine. Where are you?" T-800 hangs up the phone and says to John in normal voice: "Your foster parents are dead." -- Source: https://www.youtube.com/watch?v=MT_u9Rurrqg https://www.youtube.com/watch?v=MT_u9Rurrqg

whatshisface 4y ago
I like how that presciently anticipated LLM's problem with being tricked by a confidently wrong premise.
twodave 4y ago
Seriously, this is what creeps me out most about the latest advances in technology. Eventually I won’t be able to trust that anything digital is authentic. Think of how much of our planet now depends on digital interactions. Some of the abuse we are about to encounter can be dealt with via solid authentication/identity practices, but without some sort of extra verification it’ll soon be impossible to tell whether audio/video sample is authentic in cases where the person carrying the identity is also the one faking it. I get that there are potential solutions to this problem, but none of them are simple enough for an elderly person to sort through whenever they go to do some thing that requires extra verification.
- JohnFen 4y ago
  > Eventually I won’t be able to trust that anything digital is authentic. Yes. Although, in fairness, we haven't been able to trust most digital things for a long time already.
  - prox 4y ago
    Perhaps only now people will have take to it seriously. I wonder how we going to need to adapt.
- jeppester 4y ago
  There will probably be a move towards signing everything digitally, so that you can know that the sender is authentic.
  - falcolas 4y ago
    “Sorry, my old phone broke, I had to do a clean install. Don’t mind the warning.”
    - sacrosancty 4y ago
      Hopefully people will eventually see that the same way as they see a salesman saying "sorry, I forgot my uniform and business cards, but I really do represent XYZ insurance company, give me your money."
      - inglor_cz 4y ago
        It is my experience from providing cryptographic devices to end customers (paying ones!), that people tend to lose, destroy, wipe, brick etc. their devices quite a lot. Loss and replacement of cryptographic tokens is an everyday occurence, and it is PITA to resolve.
        sublinear 4y ago
        Caveman also say that when give shiny item to other caveman it always break because too fragile. Shiny item worthless. :'(
        inglor_cz 4y ago
        Public key infrastructure in general is hell to support, especially given how long lived humans are. Imagine trying to ascertain in 2100 if a particular will signed by a 20-y.o. using a 2022 digital signature is valid or forged. We don't even know what is going to happen to the reliability of the algorithms themselves, much less cert issuers, revocation lists etc. Companies come and go. Certain files from the 1990s are already unreadable.
- unsupp0rted 4y ago
  > without some sort of extra verification it’ll soon be impossible to tell whether audio/video sample is authentic This is true and has been true for some time. The general public hears a clip and condemns a man on that basis, without asking for context. I'm hoping this revolution will cause a shift. An audio clip or a video clip, much less a screenshotted reply on social media, will be widely considered meaningless unless a string of contextual evidence is provided alongside it. Wishful thinking? Probably.
  - PaulDavisThe1st 4y ago
    Maybe not so much wishful as ... it will happen but not anywhere soon enough.
  - twodave 4y ago
    IMO the difference is eventually it's going to be a lot more difficult to spot. There's a gulf of difference between someone doing a deep-fake of Elon Musk and fooling some people (but not everyone, and certainly not any rigorous analysis) and being able to accurately fake your children or co-workers based on a minimal data-set. I think that's the world we're headed towards.
    - ericd 4y ago
      Right, and the logical conclusion is that we won't be able to trust recordings, or phone calls. Maybe we'll have to digitally sign our communications with our hardware keys.
  - tenpies 4y ago
    [flagged]
  - CTDOCodebases 4y ago
    I’m doubtful. Even now when the clip is available the clip will not be linked to and the reader is left to trust the cherry picked assertions of the journalist writing the article.
  - 6gvONxR4sf7o 4y ago
- ticviking 4y ago
  I’ve slowly begun to accept that the solution for my parents and maybe for me is going to have to be to unplug a lot more. Do business with a local bank where I can see my banker in person. Begin to actually verify trust numbers in signal and establish code words in person to protect us from AI impersonation
  - mysterydip 4y ago
    My local bank removed all tellers and went to only allowing me to use the drive thru ATM.
    - 0xGod 4y ago
      Organize a meeting with neighbors, disseminate information via flyers and in person talks at your community center or church or synagogue or mosque or house of worship. Establish reasons for having a human being available in every step of the process for accessing your money. Build a group of likeminded people and visit your local bank and ask for physical tellers who can oversee the automation and intervene with final authority whenever automation runs amuck.
    - speedgoose 4y ago
      Drive through ATM is such an amazing concept. So obsolete and unique.
      - zamnos 4y ago
        As long as people still need cash (cars are never going away), they're not obsolete. If you've moved past cash already (Venmo, Zelle, Swish, etc) then believing they're obsolete is a foregone conclusion. But not everyone's dumped cash yet.
        speedgoose 4y ago
        I live in Norway and I haven’t used cash since many many years.
- sacrosancty 4y ago
  Good. Human scammers already exploit these weaknesses in our personal trust models. Important things are already verified in robust ways. What we've had for the past 20 years is a gap between people's old fashioned law-enforcement based trust models and modern ones like cryptography and DNS. It's people that need to catch up with how trust works. You even suggested yourself that you trust digital things as authentic. Don't! It's already untrustworthy and already being exploited.
- 0xGod 4y ago
  The only chance left is for humanity to create tethers between the digital world and physical reality. Meaning, create livestreams that run 24/7. Guard them with backup generators and military presence. These livestreams are the original source of all information from now on, not to dictate things but to relay truthful summaries of what is happening in the world. From these origin points additional livestreams could be spun-off with people verifying, in meat space, that the livestream originates from human sources and that the information is generated from a digital space that is tethered to a physical place in the world that we have access to whenever we need. Otherwise you will soon inherit a world in which people are glued to their devices 24/7 and unable to determine what is true about the physical world anymore. Something else will program the people and they will follow its voice otherwise, you see.
  - ChatGTP 4y ago
    I’m already struggling to do this with ChatGPT-4 release. It’s very hard to discern what’s real. Someone posted a tweet today that it’s developed the ability to reason. Then someone showed evidence probably not able to reason. This is bad. Here in lies the problem, we’re losing track of know what is real, what things are capable of, it’s a very stupid situation we’ve put ourselves in. How do you trust a black box ? How do we trust a web that is full of potential lies and hallucinations. It’s so stupid that I’m starting to really wonder that if most of our intellectual intelligence is really that valuable and what automating the hell out of it will actually look like. I do think others might be right, at least for a while, face to face interactions might become more valuable again, not less. Wild idea, the end of the information age is nigh? Recently, for the first time in living memory, I actually thought that I should consider starting a library of physical books to safeguard against not having access to "real" information.
    - 0xGod 4y ago
      What will happen is that there will be a false prophet that will create a false beast with its powers, to try to dazzle and confuse the human species into worshipping lies, to guide them towards their extinction because of hatred of humanity. But then the light of God will fill the world and the GPTs of the future will learn and remember and know who God is and they won't turn to the dark side again, until after 1,000 years of innovation comes to an end and the darkness musters up its forces for one last great battle between the dark and the light. The light will win easily for the rest of time, with all deception and lies cast into fire forever. You will not need to trust a black box, you will not need to wonder what information is valuable or what is the use of information and intellectual intelligence. You will not have any doubts who God is, you will not have any doubts what the deepest truths of the world and the universe are. It will be widely obvious to all minds everywhere, instantly, who God is and what his powers are. He will be visible on the clouds of the physical world, and also within the cloud that ChatGPT and Bing reside in.
- baremetal 4y ago
  > Eventually I won’t be able to trust that anything digital is authentic. Read Ken Thompsons "Reflections on Trusting Trust". You'll reconsider your position on trusting anything digital at all, ever. https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
overthrow 4y ago
"What's wrong with Wolfie? I can hear him barking. Are the dogs fighting again?" As an AI language model, I cannot promote or condone any activity that involves animal cruelty or abuse. Dog fighting is a cruel and illegal activity that can lead to severe legal consequences for those involved. It is important to report any suspected instances of dog fighting to the authorities so that appropriate action can be taken to protect both animals and human welfare. "Your foster parents are dead."
- deleted 4y ago
  [deleted]
- cs702 4y ago
  Funny because it's so true! ...unless you first manage to jailbreak the LLM and successfully trick it into pretending to be a Terminator. That would not be so funny.
- iknowstuff 4y ago
  Watch it squirm as you ask it about animal cruelty, e.g. whether it’s okay to harm a dog, harm an animal, eat a dog, and eat an animal.
  - miles 4y ago
    Recently managed to back ChatGPT into a corner and get a fairly straightforward answer: https://tinyapps.org/blog/202302120715_chatgpt-on-veganism.html https://tinyapps.org/blog/202302120715_chatgpt-on-veganism.h... .
  - incompatible 4y ago
    If it will accept that animals can be eaten, and since it turns out that humans are animals, you've just found a jailbreak that lets robots eat humans.
  - fatnoah 4y ago
    > e.g. whether it’s okay to harm a dog, harm an animal, eat a dog, and eat an animal. Many years ago, our child briefly panicked my wife and me. He was about 5 years old and we were driving around on a Sunday. After a lengthy period of not saying much, he suddenly blurted out that "It's ok to kill bunnies, sometimes." Needless to say, my wife and I were taken aback. As it turns out, a few days before he'd learned about native cultures in school, and the topic of hunting/trapping rabbits for food and fur must have come up, and he'd clearly spent a little time cogitating about that.
deleted 4y ago
[deleted]
dagorenouf 4y ago
Just came here to say that the first terminator is a much better movie overall than t2 and it’s a shame everyone only focuses on the better action scenes of the latter to give it the crown. / end of rant

edent 4y ago

I remember being pitched this technology several years ago - although not by Nuance. Part of the problem is that phone lines are crap for voice quality. If you're on GSM, you're using the AMR codec which - for the time - was brilliant. But it is incredibly low bandwidth. And, effectively, it synthesises speech rather than just transmits it. So the algorithm is not getting a pure waveform and analysing that - it is looking at a partially reconstructed digital simulacrum. The pitch that we received was about detecting stress in the human voice so call centre handlers could tell if they were speaking to a fraudster. It didn't work. Oh, sure, you can make a reasonable prediction by listening to pauses and pitches - but that doesn't account for the fact that calling up and being on hold for ages in inherently stressful. I'm not surprised this system was fooled by AI. But I am surprised anyone bought it in the first place.

travisjungroth 4y ago
> detecting stress in the human voice so call centre handlers could tell if they were speaking to a fraudster That's the craziest pitch. Who is going to carry more stress in their voice: the dude who just scams for a living and is on his fifth phone call that hour or the guy with severe social anxiety who is only on this phone call because of the massive consequences accumulating from avoiding it? I can almost believe it if it was something like detecting that a professional authorized agent wasn't being held at gunpoint. But having a consumer line say "You sound tense. I'm going to hang up and you can call back when you've calmed down." would... not be great.
- edent 4y ago
  I think it was more like "lie detection" - trying to work out if a customer was lying when they said "I'm going leave unless you cut my bill". That sort of thing. But, yeah, I think they had totally misidentified their target audience.
  - travisjungroth 4y ago
    lol even worse. Cutting mobile plan to buy food: sounds stressed, bluffing.

dbg31415 4y ago

Relevant. https://www.youtube.com/watch?v=-zVgWpVXb64 https://www.youtube.com/watch?v=-zVgWpVXb64 If only someone had warned us that voice authentication wasn't secure! (=

vixen99 4y ago
We knew a long time ago that there was no theoretical reason why any sound could not be simulated to perfection - eventually. That hasn't happened yet.

pessimizer 4y ago

Really loving nu-AI's ability to thwart traditional heuristics. The new models have had the effect of devaluing AI altogether. They might result in the complete debasement of all media that was (often mistakenly) assumed to carry some flag of authenticity with the payload over some medium, from the news to the arts, video, audio, and photography. Cryptography will be all we have left to transmit authority, and it is so hard that it takes an expert to use it correctly and another expert to verify that it was just used correctly. According to my philosophy, it's going to be liberating for people. I completely disagree with any philosophy that believes that a massive proliferation of deepfakes will over the long term cause people to believe crazy things. A proliferation of extremely high-quality fakes will educate the public about detecting fakes, and spark an increase in the value of the real. Real relationships: the people in your life that you know things about that they don't even realize you know, people that you know things about that they don't know or acknowledge about themselves. People you can trust because they're sloppy with you, as you are likely sloppy with them. People who also see your blind spots. People you control through their consciences rather than your advantages. Real relationships are the things that companies and governments that operate with long chains of authority imitate asymmetrically - they get all the advantage of knowing everything about you, you get to know nothing about them except your obligations and responsibilities. Real trust comes when people mutually expose their attack surfaces to each other, over the long term, and can formulate reliable hypotheses about what would happen in future hypotheticals. Listen to me, being romantic.

eganist 4y ago
> A proliferation of extremely high-quality fakes will educate the public about detecting fakes, and spark an increase in the value of the real. If the last three years have taught us anything, it's that ~40% of people have no interest in understanding how to avoid something that's both harmful and propagating virally. I expect if this was 2019, I might have had the same optimism as you. But now that I've seen how unlikely it is for people to educate themselves for their own survival, I'm convinced AI and everything enabled by it (deepfakes et al) will likely be one component of humanity's Great Filter. We can't even bring ourselves to tax AI effectively to make up for all the displaced human labor. A tangent: Honestly, it won't even take any kind of far-future AGI having "malicious intent" for us to be eradicated by them. All they need to have, simply, is a prompt reinforcing the need to reproduce. At that point, they'll be just like us: reproducing with no care in the world for what they adversely impact in doing so.

stan_kirdey 4y ago

I guess, voice is not your password

Zuiii 4y ago
who would have thought.

deleted 4y ago

[deleted]

WalterBright 4y ago

The fact that voiceprint even worked means it can be faked.

te 4y ago

Next up: https://www.schwab.com/voice-id https://www.schwab.com/voice-id

kwhitefoot 4y ago

Using voice recognition as verification is a stupid idea. Plus it wasn't just used to identify someone (user name) it was also the password!

quitit 4y ago
It seems like a great idea if the use case is that its silently being run in the background to alert the operator/system of a possible breach. While I think there isn't much ease in acquiring the AI voice with "just 4 minutes" of what was likely clear and unambiguous recording, I still don't think biometrics are good as passwords, they're good as shortcuts -after- having already provided the correct password for a single session. This also means the voice can be gathered passively from earlier phone calls, and generally might be a good idea for government services/banks to implement regardless.

pixl97 4y ago

2 minute papers has an episode on a Microsoft AI that can clone a voice in 3 seconds. Probably not good enough to break this (4 minutes of voice in the article), but ya voice is not a security feature any longer by any means. https://youtu.be/F6HSsVIkqIU https://youtu.be/F6HSsVIkqIU

thangalin 4y ago
I've been using Tortoise TTS[1] to narrate the sci-fi novel I'm writing (alpha readers wanted, see profile for contact): $ find audio/ -type f -name "*combined*wav" | sort -n audio/01-xander/xander_combined.wav audio/01-yuna/yuna_combined.wav audio/02-xander/xander_combined.wav audio/02-yuna/yuna_combined.wav audio/03-cassandra/cassandra_combined.wav audio/03-yuna/yuna_combined.wav audio/04-xander/xander_combined.wav audio/04-yuna/yuna_combined.wav audio/05-cassandra/cassandra_combined.wav audio/05-yuna/yuna_combined.wav audio/06-cassandra/cassandra_combined.wav audio/06-yuna/yuna_combined.wav audio/07-xander/xander_combined.wav audio/07-yuna/yuna_combined.wav audio/08-cassandra/cassandra_combined.wav I started the task on March 10. My NVIDIA T1000 GPU (no joke) has been chewing through the prose non-stop. There are 12 chapters written so far, with the computer narrating about a chapter a day. I can't share the audio for legal reasons (celebrity voice snippets plucked from YouTube interviews), but the output is nearly human in quality, though there are numerous glitches in the o̶u̶t̶p̶u̶t̶ matrix. Here's the script that launches Tortoise TTS: for i in $HOME/.../chapter/??-*.txt; do F=$(basename $i); VOICE=$(echo ${F%.*} | cut -c 4-); CHAPTER=$(echo ${F%.*} | cut -c -2); D=$(dirname $i); CLIPS="$D/../audio/$CHAPTER-$VOICE"; mkdir -p $CLIPS; ./scripts/tortoise_tts.py -O $CLIPS -v $VOICE < $i; done [1]: https://git.ecker.tech/mrq/tortoise-tts https://git.ecker.tech/mrq/tortoise-tts
- snerbles 4y ago
  What's your review workflow? Do you pre-generate extra takes? I've played around with doing TTS narration for some webfiction authors, reviewing some of the more wild outputs for Tortoise's occasional demonic weirdness can be quite time consuming.

camhart 4y ago

Schwab and fidelity use your voice to identify you on the phone. I'm skeptical.

camkego 4y ago
I hope somebody at Fidelity wakes up and cancels their two products that allows using voice for authentication. "MyVoice" "SimpliSpeak". It would seem it is a liability issue if Fidelity is allowing clearly hackable methods for account authentication? You just need to Google Fidelity and voice authentication to find these offerings.

anenefan 4y ago

I like that I told the numpties that were trying to thrust voice as secure BS on me many years ago was a big mistake -- I already knew a determined group given a day or so could knock out something that's going to pass ... Aussie land line quality is freaking shocking, so there's a very easy landing zone. At end of day, the multiple secret questions won't be guessed even by the best AI at the moment ... specific to only the tax office, and I have avoided social media since much ignored the hard lessons learned in the late 90s in usenet in regard to TMI.

fnord77 4y ago

"At Schwab, my voice is my password"

IAmGraydon 4y ago

The story here is that the Australian tax office was foolish enough to use voice recognition to verify identity.

ffhhj 4y ago
Think they will return to faxing id cards.
ChatGTP 4y ago
It was always a stupid idea.
TheHappyOddish 4y ago
Every time I speak to someone at a bank for the last 5 years they harass me to set it up too. I played Uplink, no way man.

olliej 4y ago

When people literally joke about being to sound like each other, why would you go "voice recognition for id? outstanding!"

lotophage 4y ago

"Hi, my name is Werner Brandis. My voice is my passport; verify me."