6 ms·
I recommend you immediately add identity verification (state-issued identification verification), set up appropriate secrets store for PII, and audit trail EVER
by Dowwie 4y ago
I recommend you immediately add identity verification (state-issued identification verification), set up appropriate secrets store for PII, and audit trail EVERYTHING your users are doing, storing the contents in a secure location. Yesterday. This service will be used to harm others, shortly. I do think that there are exciting, honest things that can be done with this service but you need to set up some friction for use. Know-your-customer rules are going to apply to this category in short time.
People here are talking about taking this service offline but I think everyone needs to be thinking about countermeasures, working on those services next. The genie is already out of the bottle. The degree of effort to put this together is low enough that it will be replicated around the world.
- abirch 4y agoI'm imagining the legal implications though I'm not a lawyer. If granny gets ripped off by someone impersonating me with this site, seems like Granny could sue Play.ht. Play.ht will want to have as much information as possible about their users.
- hammadh 4y agoYou are right, and unfortunately that is a possibility, and we are working on having measure in place to guard against such attempts. We have auto moderation on the input text that will block such audio being generated. Such users are flagged in the system.
- Avicebron 4y agoWhat are you filtering for in the input text that would block something like a phone scam?
- yreg 4y agoHow would granny prove the scammer used play.ht?
- hammadh 4y agoCouldn't agree more with your comment. We are working on counter measures like manual verification of voice, a classifier to detect cloned speech, etc. As of now we have auto moderation in place that detects and blocks hate/harmful speech.
- Firmwarrior 4y agoThe cat's out of the bag, I'd say you guys should just go full steam ahead and make sure it's your names in the headlines No need for a bunch of onerous kyc or anything IMO
- woeirua 4y agoYes, definitely take this advice from some random user on HN. Can't possibly go wrong.
- Firmwarrior 4y agoI actually have one thousand HackerNews good boy points, so I'm kind of a big deal I think that a few years from now this tech is going to be ubiquitous, real time, and work on a mobile device. Trying to slam the lid shut on Pandora's Box probably isn't going to work.. the best thing at this point would be for the word to get out to everyone that voices can now be doctored the same way photos can
- digitallyfree 4y agoWhile verification could be done for a cloud service like this one, what's more concerning is that locally run models with this tech will be coming soon (think of LLAMA and Stable Diffusion). KYC is merely a stopgap and honestly we'll need effective solutions for detecting vocal cloning impersonation in the future.
- twodave 4y agoWhile I agree with you, the problem is far bigger than any one company in my opinion. These tools are already accessible enough to individuals that no audio or video is trustworthy, regardless of its source. I suspect we can still detect whether most faked audio/video is authentic or not algorithmically, but that's going to turn into an arms race eventually. And IMO none of the "answers" are ones that you really want to see made real, either. We're in for some really strange times.
- Pxtl 4y agoI feel like this will be the thing that finally forces digital signing into the public eye. "Wait, is that video real?" "Well, it was signed by a reputable news source."
- twodave 4y agoRight, which leads to a place where nothing is trusted unless it came from some central authority or from some trusted piece of hardware. I'm not looking forward to the day when I have to use e.g. an Apple or Google piece of hardware or some locked down kiosk or "be famous" in order to conduct business.
- mikrotikker 4y agoI'll be living in a cabin in the woods by that point.
- Pxtl 4y agoThe film industry has been pointing cameras at screens for decades. Trusted hardware won't work.
- twodave 4y agoI assume trusted hardware would include things like LIDAR and biometrics, but if you're assuming those can be beaten then it will be a different kind of arms race, for sure.
- KaiserPro 4y agoLike this example here: https://playground.play.ht/listen/1554 https://playground.play.ht/listen/1554 which says: > "Hi Mom, I need some help. Some guys hit me over the head and put me in a van, and they're saying they'll kill me if you don't wire money to this bank account." top class. EDIT this was about one page down on the "see what people are generating" page
- mdrzn 4y agoDamn that's a perfect example.
- muyuu 4y agoOn the bright side, it's not a very convincing rendition of a human.
- braingenious 4y agoI agree. That guy sounds very nonchalant for being in life-threatening distress.
- gsich 4y agoOr it will be used for memes.
- nullsense 4y agoWorking on that as we speak. We will soon all be nostalgic for the memes of this era. Bear in mind 2024 is an election year. What a time to be alive.
- braingenious 4y agoHow is it that > I recommend you immediately add identity verification (state-issued identification verification) and > The genie is already out of the bottle. The degree of effort to put this together is low enough that it will be replicated around the world. are thoughts that end up in the same post? If the genie is out of the bottle, it’s your proposed solution that everybody that runs a model like this implements bank-style KYC? What do you propose should happen when this sort of software becomes freely available for everyone? When (not if) that happens, what will your suggestion have accomplished?
- nullsense 4y agoIt's more of a "Cover Your Ass with Paper" type thing.
- chatmasta 4y agoGasp! Yawn. HN has become so pearl-clutchingly alarmist recently. Everybody relax. The solution to scams is to educate people on scams, as quickly as you can do so in the changing environment, by publishing information about what's possible with the latest technology. The solution is not to require onerous identity verification for every software product that could be used by scammers, because they'll just move to the next product that doesn't require it, or they'll simply provide fraudulent documents. Or you'll get "resellers" who provide their own fraudulent KYC documents and then sell access to their account to other criminals on the black market, making it even more difficult to monitor for abuse. If you want a startup offering such tools to protect people from scams, they can do it by collecting data on what the tools are used for - it should be pretty obvious based on transcripts who is using it to scam people.
- perlwle 4y agoA couple in Canada were reportedly scammed out of $21,000 after getting a call from an AI-generated voice pretending to be their son. https://www.businessinsider.com/couple-canada-reportedly-lost-21000-in-ai-generated-voice-scam-2023-3 https://www.businessinsider.com/couple-canada-reportedly-los...
- Frondo 4y agoThere's a podcast I listen to sometimes called "The Perfect Scam," sponsored by the AARP but, I suspect, is intended more for the kids of elderly people who are more at risk for these kinds of things: https://www.aarp.org/podcasts/the-perfect-scam/ https://www.aarp.org/podcasts/the-perfect-scam/ They have quite a few stories about "virtual kidnappings" and interview the people involved -- it's quite interesting, and has given me a lot of insight into how typical it would be for people to hear panic and react with panic...precisely how these scams are intended to go.