26 ms·
Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)
- lelag 4y agoI predict Kitboga will have a lot of fun with this one...
- Lapsa 4y agooh my... happy porcupines
- isaacfrond 4y agofrom the article: Real-time, controllable deepfakes ready for virtual camera injection. Created for performing penetration testing against e.g. identity verification and video conferencing systems, for the use by security analysts, Red Team members, and biometrics researchers. Reminds one of the vulnerable world hypothesis. It's only a matter of time before a technology comes along that is both destructive and so simple any fool can use it.
- nonrandomstring 4y agoIn cybersecurity the bad guys always win. There are two very rough "proofs of a kind" I use to explain this to students, both well known in military and counter-terrorism analysis. One is called Blotto analysis. As the number of fronts grows large (>12 is an important point) the game favours the attackers. Defenders have to always win on all fronts, but terrorists only have to win once on any of n fronts. Another scenario comes from finite board games like Go. Defenders lose when their back is against a wall. Because the space of practical human security solutions is smaller than the inventiveness of attackers, eventually the attackers win. Biometrics is the poster child for this kind of folly. There are only two edge cases. One is to create proven trustless systems. Such systems cannot involve humans in any way shape or form, and therefore fail by definition as human security solutions. The other is to eliminate all attackers. This does indeed create a fountain of fascinating ethical conundrums with regard to security research and the possibility for strategic limitation of knowledge hazards.
- jacquesm 4y agoSpot on, and given that there seem to be no real guidelines for any of this beyond 'responsible disclosure' the question for me is whether we are accelerating the arms race with all these neat point-and-click tools.
- pid-1 4y ago> Defenders have to always win on all fronts, but terrorists only have to win once on any of n fronts It depends on how you design systems and processes. Let Mn denote the nth defense mechanism of a system. We can build architectures in which the probability of breaches is calculated as: (M0 is pwned) AND (M1 pwned) AND (M2 is pwned) AND (M3 is pwned)..... You can see the total probability can become small quickly, as it's computed as the product of all probabilites. The collorary is: it's possible to design secure systems from insecure building blocks (as long as those are reasonably uncorrelated!) Insecure systems are generally designed the way you suggest in your post: if a single defense mechanism is breached, the whole system will be breached. So you basically are as secure as your weakest link.
- nonrandomstring 4y agoYes, I haven't mentioned defence in depth here. And as you say, serious real world applications are more than a single layer. But of course what we see with DiD is polynomial growth of complexity, so policy, monitoring, rebuild time and so much else gets hard to manage - upshot is someone, somewhere falls back to simpler configurations - and there's the entry point. But sure, thank goodness we've moved beyond a single blacklist firewall these days :)
- giorgiop 4y agoHi there, one of the maintainers here. Thanks for sharing & AMA!
- jacquesm 4y agoEven if you claim that you are legally not responsible for any abuses of this software do you feel any moral responsibility?
- lovemenot 4y agoI too would like to imagine whether or not I would release something like this in similar circumstances. Yet this question seems unfairly loaded. At the least - premature. Though we can easily imagine it, afaik there is no evidence of any actual abuse to date.
- jacquesm 4y agoTools like these will end up in the hands of those that are trying to harden an installation and in the hands of those that will end up using them to try to break into such installations. I've often wondered whether the net effect of the whole security research community is a net positive or a net negative and I honestly do not know the answer. So yes, it is a loaded question. But asking yourself if what you can do is actually the right thing to do is always good, especially if you - as these authors imply - know up front that there is a large chance of abuse.
- lovemenot 4y agoThank you for a considered response. I share those feelings. I just hope the person offering AMA will not be daunted by a good but blunt question from a voice that carries weight around here.
- jacquesm 4y agoI've been in that position, which is one of the reasons why I'm asking. When I came up with 'live streaming video on the www' I never for one second sat down to think about the abuse potential. Color me hopelessly naive. And when confronted with the various abuses over the years I've always had a problem with that, this was the direct consequence of me just 'scratching my itch' and it caused a huge amount of misery. Oh, say the defenders, but if you had not done it then somebody else would have. This is true, but then that moral weight would be on their shoulders and not on mine. Hence my question. Because I do feel that weight and it has caused me to carefully consider the abuse potential of the stuff that I've released since then and I've only released those things that I feel have none that I can (easily) discern.
- agilob 4y agohttps://github.com/alievk/avatarify-desktop https://github.com/alievk/avatarify-desktop
- giorgiop 4y agoDot was used for performing vulnerability assessments on many biometric KYC vendors on 2022. The Verge covered this study in this article https://www.theverge.com/2022/5/18/23092964/deepfake-attack-facial-recognition-liveness-test-banks-sensity-report https://www.theverge.com/2022/5/18/23092964/deepfake-attack-...
- pragmatick 4y agoThat article is linked in the second paragraph of the readme.
- Planktonne 4y ago> Authors and contributing developers assume no liability and are not responsible for any misuse or damage caused by the use of this program. Anything that can be created, will be created. However, that doesn't free you from all moral culpability. If you create something, make it freely accessible and easy to use, then I think you are partly responsible for its misuse. I'm not saying that they shouldn't have created this, or that they don't have the right to release it. But to create it, release it, and then pretend that any misuse was entirely separate to you is at best naive.
- davidguetta 4y agoHaving morals when the other side doesn't is a weakness. Make the tools available to everyone. As Elon musk says, sun is the best disinfectant.
- Planktonne 4y agoHaving morals when the other side doesn't is the best possible reason to be on different sides.
- davidguetta 4y agoMorals are cultural.. in some countries it means burning gay people. Power however, is amoral and neutral
- ls15 4y agoUnless we cut off the chain of responsibility somewhere, the creators of the programming language, the computer and its components, as well as the people who have designed the computer and those who mined the required minerals are responsible as well.
- alexjurkiewicz 4y agoGood thing nobody is suggesting we *checks notes* jail silica miners for building the Deepfake Offensive Toolkit.
- Nowado 4y agoI'm absolutely loving the idea of deepfake-detection security SASS developing an easy to use live deepfake system. Crassus tradition is truly immortal.
- xiphias2 4y agoGovernments and bank branches should offer physical identification as a (potentially payed) service globally, and provide a digital signature that a person is owning a physical device (which should be valid until the device is reset or invalidated by the person).
- 2OEH8eoCRo0 4y agoI always imagined the USPS fulfilling this need.
- xiphias2 4y agoI don't know how serious is USPS, but in my country the post office doesn't check if the signature looks the same for example. Banks face and handle fraud on a much higher level, and have a much better training in identificacion.
- 2OEH8eoCRo0 4y agoMy signatures never look the same. Signatures aren't a good judge of authenticity. My point is that a bad actor would have to show up in person and lie to a federal entity. It's not that they won't get away with it- it's that they cannot hide on the other side of the globe.
- giorgiop 4y agoOne the author here. Please read this https://www.biometricupdate.com/202205/sensity-alleges-biometric-onboarding-providers-downplaying-deepfake-threat https://www.biometricupdate.com/202205/sensity-alleges-biome...
- Bibbo 4y agoThis is huge!
- Chang22 4y agoIs that something related to the big attack happened last year in China? https://findbiometrics.com/nine-of-top-10-liveness-detection-systems-vulnerable-deepfakes-report-052003/ https://findbiometrics.com/nine-of-top-10-liveness-detection...
- SXX 4y agoTo all people who ask moral side of this tech and that it can be abused. What do you think about tools like Kali Linux? NMAP? Basically any pentesting software can be abused.
- jacquesm 4y agoIt's not what we think about this, it is what the creators think about it that matters.
- yosito 4y agoIt doesn't matter what anyone thinks about it.
- jacquesm 4y agoI actually disagree with that. I think software and ethics are closely intertwined and too often we see them as disjoint. It's interesting to try to find a software project that does not have an ethically dubious angle to it, that isn't all that simple.
- imiric 4y agoWhile it's an interesting discussion, would you ask the same question to a knife or sword maker? Your tone in this and the AMA thread is combative, and wouldn't lead to a productive discussion. It would be much more interesting to discuss the technical merits of this project than any particular moral concerns you might have, especially on this forum.
- jacquesm 4y ago> Your tone in this and the AMA thread is combative, and wouldn't lead to a productive discussion. That's mostly your impression. But that may be because you believe that I have a position with regards to this particular software, which I do not, I just see this as an opportunity to gain insight in the position of the author which I find interesting and which may help guide me in similar decisions in the future, because it's something that I've been wrestling with for a long, long time. Since 1995 in fact. > It would be much more interesting to discuss the technical merits of this project than any particular moral concerns you might have, especially on this forum. What you find more interesting and what I find more interesting do not necessarily have to be the same things, and you are totally welcome to ask your technical merits questions. Finally, I don't like to be told what I can and can not discuss, especially not in a thread started by someone who wrote 'AMA' where the third A stands for 'Anything'.
- jl6 4y agoWhich requires more computation: a real-time deepfake or real-time deepfake detection? That will determine the future balance of power.
- tgv 4y agoYou don't need to detect a deepfake in the same time as it takes to generate one, do you?
- dividedbyzero 4y agoBut you may have to go through orders of magnitude more material
- darkwater 4y agoIf you get a Zoom call from your CFO telling you to issue a money transfer to that bank account, that it is 100% legit, how much time do you have to detect a deepfake?
- garblegarble 4y agoDoes your company instantly execute payment instructions to new supplier accounts based on a remote verbal instruction?
- throw457 4y agoDeepfakes are for deceiving humans. There are still way too many artifacts in the outputs that are easy to detect in code.
- coldcode 4y agoThe problem comes that even with extremely good deepfake detection, political operations can still use the deepfakes to fool the average voter into believing that is it real, irrespective of any successful analysis. You and I might not be fooled but we don't matter if 10000X people are fooled and affect our lives anyway in a negative way.
- yosito 4y agoA few months ago, the IRS made me verify my identity using some janky video conferencing software where I had to hold up a copy of my passport. The software was so hard to use, that I can't believe average people manage to do it. Now, real-time deep fakes are literally easier to create than using the video verification software itself. This will have interesting societal implications.
- deleted 4y ago[deleted]
- aimor 4y agoI mistyped my SSN this year and I wound up doing something similar: I had to take off my glasses and hold my face exactly in the center of the camera, while repeatedly squinting (hopelessly) to try and read the error messages as they alternated between "TOO CLOSE" and "TOO FAR AWAY". I gave up and, luckily, a few hours later found the mistake.
- Abishek_Muthian 4y agoIn India, Digital Signature issuing companies use webcam video to authenticate the applicant as well(I don't think even holding document is required); That digital-sign is used everywhere from signing tax filing to paying taxes. I hope deep-fake detection software can compete with deep-fake generation software, I've been tracking this need-gap on my problem validation forum for a while now[1]. That said, There are ethical usages of deep-fake videos as well; In fact I might checkout this very tool to see if I can use it for 'smiling more in the videos', remembering to smile during videos is exhaustive for me. There are other ethical usages like limiting the physical effort needed to produce video content for those with disability(like myself)[2]. [1] https://needgap.com/problems/21-deep-fake-video-detection-fakenews-machinelearning https://needgap.com/problems/21-deep-fake-video-detection-fa... [2] https://needgap.com/problems/20-deep-fake-video-generating-sw-for-ethical-usage-deepfake-machinelearning https://needgap.com/problems/20-deep-fake-video-generating-s...
- kalal 4y agoSerious question. What is the chance this video has been deepfaked? https://www.youtube.com/watch?v=ENlL-Uru-cM&ab_channel=CNBCTelevision https://www.youtube.com/watch?v=ENlL-Uru-cM&ab_channel=CNBCT...
- throw9871928 4y agoI work at Axis, which makes surveillance cameras.[0] This comment is my own, and is not on behalf of the company. I'm using a throwaway account because I'd rather not be identified (and because surveillance is quite controversial here). Axis has developed a way to cryptographically sign video, using TPMs (Trusted Platform Module) built into the cameras and embedding the signature into the h.264 stream.[1] The video can be verified on playback using an open-source video player.[2] I hope this sort of video signing will be mainstream in all cameras in the future (i.e. cellphones etc), as it will pretty much solve the trust issues deep fakes are causing. [0] https://www.axis.com/ https://www.axis.com/ [1] https://www.axis.com/newsroom/article/trust-signed-video https://www.axis.com/newsroom/article/trust-signed-video [2] https://www.axis.com/en-gb/newsroom/press-release/axis-communications-launches-open-source-project-for-video-authentication-to-ensure-the-integrity-of-surveillance-video https://www.axis.com/en-gb/newsroom/press-release/axis-commu...
- Tepix 4y ago> it will pretty much solve the trust issues I'm not so sure. How will you verify a signature if you see a video on TV or on social media? Do you believe these devices are 100% secure and the keys will never be extracted?
- throw9871928 4y agoNothing is 100% secure, but the point of using a TPM is to prevent extraction of the keys.
- Diggsey 4y agoTPMs are great at preventing extraction of keys via the exploitation of software bugs. They are not so good at preventing extraction if you have physical access to the machine, and especially not if you don't need the machine to survive intact, and can afford to destroy as many machines as you need to get the keys out...
- Tepix 4y agoI agree that it's better than nothing. Sooner or later deepfakes will be pretty much perfect and detection will be more or less impossible.
- mustyoshi 4y agoI'm glad they released this. I'm sick snd tired of seeing big companies and orgs (Google is the most recent) publish an amazing application of ML but refuse to release the trained model because the model is biased and may be used in a bad way.
- IshKebab 4y agoI suspect that's mostly an excuse and they just want to keep it to themselves for commercial reasons. I mean I'm sure they are happy not to have to deal with any ethical issues by keeping it private but that's probably a secondary motivation. It's not like they release their state of the art ML stuff when there aren't any ethical issues anyway, e.g. for voice recognition.
- Yuioup 4y agoI see a lot of people decrying its existence, but maybe this tool can help combating deepfakes?
- mensetmanusman 4y agoThis is like releasing an app that unlocks all car doors and telling users to only use it on their car.
- IshKebab 4y agoYou can buy lock picks specifically for cars. The world hasn't ended.
- mensetmanusman 4y agoAll cars…
- sva_ 4y agoTo those who ask about the ethics of releasing something like this, I'd say that this technology already exists, and bad actors probably already can get access if they really want to and are sophisticated enough. Making this available to the general public will spread awareness of the existence of such tools, and can then possibly have a preventive effect.
- belter 4y agoYou are right. I saw some guy that looked like Matt Damon trying to sell me some crypto coins...
- karmakurtisaani 4y agoIt would be so ridiculously crazy that an established A-list actor would willingly promote those crypto scams. Must have been a deep fake!
- HWR_14 4y agoFor what it's worth, actors seem to be securing themselves against this using IP rights.
- bko 4y agoI agree with everything you said, but I we shouldn't deny that opportunistic bad actors don't exist. Or it might get on their radar and be exploited. Open source tools also tend to be better maintained, documented and reliable, so the bad guys will have a better tool. That being said, bringing it to light also has benefits like you said. If the tool is out in the open and state of the art techniques are used, technology to detect its use will also benefit.
- blagie 4y agoAs someone with a stalker, I can't emphasize this enough. A stalker will go to all sorts of lengths to do bizarre shit. People don't believe it. I would guess governments will do some equivalent there-of. Democratizing access to things -- including bad things -- has a preventative effect: 1) I can guard against things I know about 2) People take me seriously if something has been democratized The worst-case scenario is if my stalker got her hands on something like deep fake technology before the police / prosecutor / jury didn't knew it existed. I'd probably be in jail by now if something like that ever happened. She's tried to frame me twice before. Fortunately, they were transparent. She'll try again. Best case scenario is that no one has access to this stuff. Worst case scenario is only a select group have access, and most people don't know about it. Universal access is somewhere in between.
- hrgiger 4y agoThere is also work going on for deepfake detection challenges since years, I think forensics benchmark was the oldest one and other two placed at 2019 , I dont know if forensics still going on though http://kaldir.vc.in.tum.de/faceforensics_benchmark/ http://kaldir.vc.in.tum.de/faceforensics_benchmark/ https://www.kaggle.com/competitions/deepfake-detection-challenge/leaderboard https://www.kaggle.com/competitions/deepfake-detection-chall... https://ai.facebook.com/blog/deepfake-detection-challenge-results-an-open-initiative-to-advance-ai/ https://ai.facebook.com/blog/deepfake-detection-challenge-re... edit*: looks like yes forensics top submission 25 Nov, 2021
- scoutt 4y agoI can imagine more scenarios other than "bad actors" (scammers and such) for this software... For example: what about using this in an interview for a remote job (possibly intercontinental)? It could tip the balance in some situations (due to biases). For example, beating ageism by projecting a 25 years old version of ourselves. Or removing a tatoo or birth defect (and possibly other changes too, you can imagine).
- registeredcorn 4y agoI'm really excited to see what could be done with this! I think the primary benefits of this being released are two fold: 1) It will give security researchers more freely available technology to work with in order to try and fight the malicious use of deepfakes. (I saw some interesting comments in this thread about TPM. It'd be interesting to see what other solutions are out there.) 2) It would raise the overall awareness of the general population about the existence and advancements that deepfake technology has made. I would argue that a small subset of the overall population know what the term "deepfake" means, and even fewer are aware at how far it has progressed in only a few short years. (I'm not super well versed in the topic myself, I just know that I've heard a lot of progress has been made.) I think that since this tech is already actively being used by bad actors, the best course of action that we can take until at least a somewhat good counter to it has been adopted (and then quickly defeated) is to make as many people aware that this is something that could affect them, or their families. That this is something that could be used to get someone fired, or hurt, or killed. I think that the more that people are aware of its existence, the less impactful the overall effect of deepfakes becomes. People learn to look twice before making a call on something, because of how easy it has become to fake audio and video.
- wafriedemann 4y agoWell, genius of this guy. Create the threat then sell the cure. The old school business model we know from anti-virus software. "I am Cofounder and CEO at Sensity, formerly called Deeptrace, an AI security startup detecting and monitoring online visual threats such as “deepfakes”." (one of the contributors of this repo)
- nicce 4y agoWell, this kind of threat was just a matter of time, if did not exist already, and public knowledge is for a greater good.
- madrox 4y agoThe moral lens people apply to deepfakes feels myopic to me. There are lots of tools that have been built that have arguably done more harm that no one talks about, like port scanning tools. Perhaps because the the negative consequences are so visually obvious and require few intuitive leaps.
- momojo 4y ago> Perhaps because the the negative consequences are so visually obvious and require few intuitive leaps. Exactly this. At some point it's not feasible to calculate the impact of a technology's trickle down. Unless the impact is large enough. I'm not surprised people have such a viscerally moral reaction to deepfakes. > There are lots of tools that have been built that have arguably done more harm that no one talks about, like port scanning tools I don't think this is a fair argument against current tools. If we had some way of accurately measuring the secondary effects of port-scanners, should we start caring? Should we retroactively remove any tools that we later regret? (I don't have good answers to these either)
- madrox 4y agoI agree with everything you've said. I suspect this also applies to many criticisms of crypto and NFTs as bad for the environment. The impact is simply easy to calculate...not outsized to other industries. We certainly don't evaluate, say, the video game industry based on its environmental impact in the same way.
- leetrout 4y agoHas anyone followed the install instructions successfully on an m1 mac?
- giorgiop 4y agoWe haven't really tested it on M1. You are welcome to give a try and open an Issue if any problem
- BLO716 4y agoTime for a Teams / Zoom plugin
- giorgiop 4y agoYou can already use this in video conferences, by the way
- micromacrofoot 4y agoThis seems like a good way to get enough attention around deepfakes to make them illegal. How long before a major media personality or politician is catfished by this?
- gambler 4y agoJust because you think some issues is "inevitable" in the future doesn't mean you should spend academic resources to make it happen faster, while simultaneously making it easier to exploit.
- touristtam 4y agoI wished we would be in a position where OBS was not needed.
- giorgiop 4y agoWhy so? I am genuinely interested. I am one of the authors. Other virtual cameras would work too, by the way. OBS is the README as it's the easiest to set up.