29 ms·
Bumble releases open-source A.I. feature to help combat unsolicited nudes
- monkpit 4y agoThe term “Cyberflashing” is ridiculous. It makes it sound innocent and cutesy. We already have a term for this, it’s sexual harassment.
- nerdponx 4y agoEh, it's pithy and evocative.
- monkpit 4y agoThat’s kind of the problem I have with it, I guess. It’s not something to be pithy about, it comes across as a euphemism that softens the impact of the violating act it actually is.
- deltasevennine 4y agosexual harassment is too extreme. 20 years of prison in some places. Cyber flashing is indeed the more appropriate term. I will be absolutely unequivocally clear about my opinion on this: Cyber flashing does not deserve jail time. It deserves being banned from bumble. Anyone who is so low and so cruel to jail someone for cyber flashing... that person is the one who deserves jail time. Also logistically you'd be jailing millions of people if it was sexual harassment. In short, dick pics are a problem with people being stupid. Not a problem with people being criminals. Let's keep the wording appropriate to the severity of the infraction. It's cyberflashing... NOT sexual harassment.
- monkpit 4y agoIf you did this in public you’d be sent to jail. Why is it ok on the internet? Unsolicited indecent exposure is sexual harassment.
- deltasevennine 4y agoYou're relying on logical alignment to regulate your morals. Well it's biased because it can go both ways. For the direction opposite to your query one can logically say: "Dick pics on the internet are ok why isn't flashing your dick ok in public?" You simply chose the direction that fits with your bias. The reality is common sense is the best choice here. There is no alignment when obviously your gut tells you flashing your dick in public is much worse then a dick pic. That gut feeling that regulates that feeling is the same feeling that tells us that while porn is ok to be all over the internet, it's not ok for porn scenes to be played in public.
- cloudking 4y agoGitHub repo: https://github.com/bumble-tech/private-detector https://github.com/bumble-tech/private-detector
- motohagiography 4y agoI wonder how well it detects sandwiches.
- cloudking 4y agoOr hot dogs?
- motohagiography 4y agoThis is a non-trivial question.
- notjulianjaynes 4y agoI saw this only had 5 comments, and was excited I might get to make this reference/joke. Rats!
- tuatoru 4y agoRats, too. Also orchids.
- version_five 4y agoI'm most interested in how they handle it for the users. It says potential "nudes" are blurred and can be "reported" or viewed. Will it tell the sender they're sending something that's going to be blurred? (which on one hand would let them resolve a false positive but also maybe let them find an adversarial image).
- XorNot 4y agoThis is an interesting question, because it can be unpleasant to know that the attempt had even happened. I suspect given the option, most people would turn on an option for "block dick pics silently" and leave it on. Similar to how I never actually check my spam folder anymore.
- monktastic1 4y agoReally? I would have guessed most people would want to know who the "toxic" people are in their life so that they can take appropriate action instead of finding out in a worse way later.
- dinosaurdynasty 4y agoI think the idea is "automatically block people who send dick pics" and not "silently remove dick pics but allow other messages from the same person". Considering what Bumble/Tinder are maybe this makes sense?
- joking 4y agotinder doesn't allow to send pics, I think that this is actually the reason they don't.
- monktastic1 4y agoAh, thanks. My parent comment makes way more sense to me in that light.
- fshbbdssbbgdd 4y ago
- WrtCdEvrydy 4y ago
- anonym29 4y agoI'm almost certain your comment is tongue-in-cheek, but it did pique my curiosity - what do you think the actual ratio of unwanted explicit photos from each sex is? I have a hard time imagining it being less than 99:1.
- diebeforei485 4y agoThere is a lot of DM spam on Twitter, TikTok, and Facebook/Instagram/Whatsapp involving explicit photos of female genitalia that would meet the criteria of obscene material under California's law[1] around sending unsolicited images by electronic means (SB 53). 1. https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202120220SB53 https://leginfo.legislature.ca.gov/faces/billTextClient.xhtm...
- Affric 4y agoAre they sent by the women who own the genitalia? I have never once received an unwanted nude from a woman but as spam whereas with men it has happened enough times for me to believe without statistical justification that unwanted images of male genitalia would be more common.
- diebeforei485 4y agoYes, women who have paid content on OnlyFans or similar platforms. In some cases it's IRL sex workers. I am not sure how this is relevant to building a classifier, however. Or relevant to California law.
- chongli 4y agoIt does not matter who owns the body parts in the images. Unwanted genitalia is unwanted genitalia, no matter the sex or the purpose. I think a lot of unsolicited female genitalia is sent for spam/scam proposes. A particularly pernicious sort is in the form of sextortion [1]. Images of female genitalia may be sent by a criminal to entice a (typically young or teenage) boy into believing he’s engaging in an online romantic relationship with a real girl. When he reciprocates with images of his own body (as requested to do so), the criminal responds by using those images to blackmail him into paying money with the threat of exposing him to friends and family. This sort of attack has also been used to target girls, though recently the targeting of boys has grown much faster. If I had to speculate, criminals likely see young boys as easier targets who can be enticed with images of the female body. [1] https://en.wikipedia.org/wiki/Sextortion https://en.wikipedia.org/wiki/Sextortion
- anonym29 4y agoSeeFood says: not hotdog
- deleted 4y ago[deleted]
- minimaxir 4y agoI wonder if there is a way to use this to finetune an image generating AI to generate images that the AI would think is lewd but is not. (similar to what was done with a Yahoo model in 2016: https://news.ycombinator.com/item?id=12756462 https://news.ycombinator.com/item?id=12756462 ) Archive of the original post (NSFW?): https://web.archive.org/web/20161022003102/https://open_nsfw.gitlab.io/ https://web.archive.org/web/20161022003102/https://open_nsfw...
- Filligree 4y agoOh, absolutely. The only question is whether or not it can be done efficiently.
- firefoxkekw 4y agoAdversarial attacks can be done 'easily' if you have access to the loss function, harder if not. Two minute papers - 1 pixel attack: https://www.youtube.com/watch?v=AOZw1tgD8dA https://www.youtube.com/watch?v=AOZw1tgD8dA Decent papers: https://arxiv.org/abs/1812.00181 https://arxiv.org/abs/1812.00181 https://arxiv.org/pdf/1807.00051 https://arxiv.org/pdf/1807.00051
- lern_too_spel 4y agoMore likely is people making adversarial dick pics to get past the filter, which is very quick to do when you have the parameters of a differentiable model. I noticed they were careful to state that they are releasing a model instead of the model they are using though.
- comboy 4y agovenn
- tomcam 4y ago> adversarial dick pics Well, I wasn’t planning on encountering that phrase today, perhaps ever to be honest
- andrewflnr 4y ago
- fragmede 4y agoI can't help but wonder how much the open sourcing of Stable Diffusion contributed to the decision to do this release.
- teej 4y agoDick picks are the only infinitely renewable resource on this planet. Pretty sure Stable Diffusion isn’t the reason.
- ghostk 4y agoProbably not much at all. They used a pretty straightforward and easy to run model, EfficientNetV2. Something that doesn't give out the best results, but will offer very quick inference and train time.
- arianvanp 4y ago(Full disclosure; I work for a Bumble competitor. And I applaud Bumble for open sourcing this. It's great if we can work together as an industry to protect people from sexual abuse). There's some questions I have about this release the interest is in the model itself. They released the pre-trained model as a zip that you can download from their website but they do not mention the license for using this model anywhere. It is unclear from the documentation and license under what conditions you're allowed to use it. The zipfile itself doesn't contain any terms of use inside. Though they give instructions on how to tweak the model with your own additional input images; again it's unclear from the license if you can create derivatives of the Model. One might assume it falls under the Apache license that's included in the repository. But the model itself does not live in the repository (is not bundled with the source code) so it's not clear. Also Apache isn't really suited for assets and models. it's more meant for source code. It would be nice if they would adopt a license that explicitly mentions how the model can be used like StableDiffusion does.
- mdaniel 4y agoI wonder if marketing put the kibosh on the original title, since the URL slug differs substantially from the article's title While doing a little view-source: spelunking, I also noticed they're straight-up lying about their alternate language link tags, which I guess is only slightly less interesting than if they had, in fact, translated that blog post into the 26 languages they pretend to support
- user- 4y agoCan they open source their algorithirm so I can figure out why I get no matches
- oneoff786 4y ago
- logicalmonster 4y agoI realize that this isn't exactly Reddit, but good God man, has the capacity for a simple self-deprecating joke been lost here?
- nickmooney 4y agoThe issue isn't that it's self-deprecating, it's that it's not funny, esp. given the context. "Read the room" type of thing.
- dymk 4y agoSpeak for yourself, I found it chuckle worthy
- oneoff786 4y agoI don’t get it? Are they suggesting then that their normal photos are being tagged as lewd?
- deleted 4y ago[deleted]
- adrr 4y agoHave you tried asking people to send you pictures of hotdogs.
- hsavit1 4y agodoes anyone actually use bumble and enjoy it? serious question
- Lammy 4y agoTotal speculation of course, but I wonder if they called this "Private Dick" internally before the trademark filing :p https://tsdr.uspto.gov/#caseNumber=79343576&caseType=SERIAL_NO&searchType=statusSearch https://tsdr.uspto.gov/#caseNumber=79343576&caseType=SERIAL_... https://www.quora.com/How-did-the-term-private-dick-for-a-private-investigator-originate https://www.quora.com/How-did-the-term-private-dick-for-a-pr...
- mdaniel 4y agoTheir "drawing" of the trademark is :chefs_kiss: https://tsdr.uspto.gov/documentviewer?caseId=sn79343576&docId=DRW20220630112123#docIndex=3&page=1 https://tsdr.uspto.gov/documentviewer?caseId=sn79343576&docI...
- deleted 4y ago[deleted]
- thom_ 4y ago
- alberth 4y agoHot Dog / Not Hot Dog.
- ackbar03 4y agoArgh you beat me to it
- deleted 4y ago[deleted]
- jedberg 4y agoWhat I wonder is if they actually came up with the idea because of that. The timing is about right.
- cercatrova 4y agoIn the show, Jian Yang sells the app to a social media company for precisely detecting this sort of phallic imagery, so I think Mike Judge knew exactly what he was doing.
- nielsbot 4y agoOT--But Mike Judge was interviewed recently on Fly On The Wall, the podcast from David Spade and Dana Carvey (who doesn't have a podcast these days?)... and it was a great episode. There's a part where he explains where the Butthead's guffawing laugh comes from.. so good. The podcast is focused on interviewing SNL alums.. but has recently branched out a bit. Anyway, I didn't know this, but Mike Judge did some Milton (the character who later appeared in Office Space) short for them. I recommend it.
- smrtinsert 4y agoMike Judge wears his genius so comfortably you don't even notice it sometimes.
- tgtweak 4y agoSurprised to find this gem so far down on the page
- fastball 4y agoA lot of their copy seems to indicate the priority is detecting dick picks. I realize those are the vast, vast majority of "unsolicited lewd images", but I do wonder how capable their model is at detecting pussy pics, if at all.
- subroutine 4y agoThe dev team had a large training set for male anatomy due to availability and a large training set for female anatomy due to determination.
- bergenty 4y agoThat model will probably be used for entirely different purposes.
- deleted 4y ago[deleted]
- DerSaidin 4y ago> AI tool to detect unsolicited lewd images Does this mean it can also detect "send nudes" to avoid false positives on solicited lewd images? Or does the UI have a on/off toggle for currently-accepting-lewd-images?
- arglebargle123 4y agoIf it does I hope they were considerate enough of our Australian friends to have it detect "sapnu puas" as well.
- jffry 4y agoIf it thinks an image is lewd, it blurs it, and you have the choice to view it, report it, or view it and then report it: https://bumble.com/en-us/help/why-am-i-seeing-a-blurred-image https://bumble.com/en-us/help/why-am-i-seeing-a-blurred-imag...
- returnzero 4y agoMany years ago (ca. 2009?) at a previous employer we had an enterprise email scanner which included a content filtering feature which could block "pornographic images". We ended up turning it off as it had too many false positives since it was an insurance company that dealt with injuries so a lot of legitimate injury photos would get classed as porn and blocked. I think it was basically looking at how much of an image was skin tones. The more interesting thing about that feature was that it A) only blocked nude images of white people, and B) would aggressively block photos of golden retrievers much to the annoyance of someone in the claims team who bred them and occasionally received photos to her work email.
- bogota 4y ago> aggressively block photos of golden retrievers That gave me a good laugh
- ramesh31 4y agoThere was an XKCD comic around that timeframe: https://xkcd.com/1425/ https://xkcd.com/1425/ And it really sticks out as one of the few paradigm shifts we’ve had in the last 10 years. In 2009, it really was an impossible task to do that kind of image recognition. But by 2012, it was available in a library for anyone.
- w-ll 4y agoEven outside of image recognition, this is my most common referenced XKCD comic, with non tech managers trying to guess timelines of work.
- mikepurvis 4y agoThat honour goes to 927 in my circles, and it's not even close.
- nl 4y agoSmall correction - it wasn't until 2014 that it was widely available in libraries. AlexNet was September 2012. FlickR did their XKCD1425 bird detector in Oct 2014: https://code.flickr.net/2014/10/20/introducing-flickr-park-or-bird/ https://code.flickr.net/2014/10/20/introducing-flickr-park-o...
- runlevel1 4y agoAround 2007, there was an image board type of site that was struggling to build a system that could reliably detect these sort of images. Their solution ended up being to run it through face detection first -- which was pretty reliable even back then. It turns out, people who sent unsolicited lewd photos tended to do so without their face in frame. So if a face was detected, it was significantly less likely to be lewd. Just an amusing anecdote. Wish I could find the blog post describing it.
- lucasfcosta 4y agoAn excellent example of cutting the Gordian Knot: https://en.wikipedia.org/wiki/Gordian_Knot https://en.wikipedia.org/wiki/Gordian_Knot
- z9znz 4y agoAre you making some kind of Freudian Slit?
- nerdponx 4y agoAlso a great example of effective feature engineering. The success of gradient boosting and deep learning lies largely in the ability of those models to "learn" sophisticated high-order features from raw data. Their particular success in working with images, audio, and video lie in their ability to construct more (and more sophisticated) features, beyond what a team of smart humans with domain knowledge could have constructed by hand given huge resources. In a sense, the only difference between model architectures is the feature space that they embed the raw data into. That said, sometimes you just need to give the model a good starting place, and sometimes you can obviate A sophisticated model largely or entirely, if you can come up with highly "explanatory" features on your own. To this day, I think the ability to come up with high-impact features is one of the differentiating factors of really good industry data scientist.
- DonHopkins 4y agoThere should be a feature to warp and bend the naughty bits so they're tiny and shriveled and encrusted with warts and scabs and spiders, and post a copy back to the offender, to discourage them.
- dtdynasty 4y agoFunnily enough one of my SFW pictures got detected recently on bumble and my working theory is that they detected my phone as a bulge. It might be fun to try to test what type of content it misidentifies. Edit: missing "to"
- tonymet 4y ago20 years ago when candy bar phones were popular i designed a dildo-shaped phone case – a sort of “push up bra” for men. Not sure why my product never took off.
- eclipticplane 4y agoTiming the market is tough: 20 years hence from your invention and you could have been /u/rightcoastguy
- BuyMyBitcoins 4y agoAlternatively: “My cellphone case design had a bad reception”.
- m00x 4y agoYou should have gone to VCs. Maybe could've raised some capital. Could've been huge!
- deleted 4y ago[deleted]
- tintor 4y agoNot all ML jobs are highly sought after.
- onetimeusename 4y agoThis looks like the detector does not run on the phone but I could be wrong about that. If it does not, that implies all messages on this dating app are not private. These conversations and pictures could be very sensitive. our scale allows us to collect a best-in-the-industry dataset of both lewd and non-lewd images For another, it seems like they are saying they collect all these images of people's private parts and use them for their training set.
- ryan_lane 4y agoUnreported profile photos: not lewd Reported profile photos: lewd Reported message photos: lewd Though it's possible they're sourcing the photos in the above way, in general I'd assume that the messages are not private. Unless messages are end-to-end encrypted on a platform, you have to assume the platform can access them.
- jffry 4y agoThis isn't some huge revelation. E2E encryption isn't something that Bumble advertises, and their privacy policy [1] plainly states: > We review the content of messages sent in the App to identify topics, sentiments, and trends across our Users. We will take steps to remove personally identifying information from such messages, prior to reviewing them. We will not share the content of User messages or information we derive from them with any third party. [1] https://bumble.com/en-us/privacy https://bumble.com/en-us/privacy
- onetimeusename 4y agoWe will not share the content of User messages or information we derive from them with any third party. Isn't that what the SavedModel is that they shared on googleapis.com right here? https://storage.googleapis.com/private_detector/private_detector.zip https://storage.googleapis.com/private_detector/private_dete... edit: and for the record if your messages on a dating app are not E2E encrypted then you are making a terrible mistake. I can't believe anyone would voluntarily make this decision.
- 4y ago
- caf 4y agoAn open source dick detector, truly we are living in the future.
- isoprophlex 4y agoA... dicktector.
- akomtu 4y agoImagine a pull request with new unit tests.
- klysm 4y agoI guess bumble was uniquely positioned with a dataset.
- xjay 4y agoMaybe a lewdifier could instead merge lewdness into every picture.
- Imnimo 4y ago>Traversing the trade-offs between state-of-the-art performance and the ability to serve our user base at scale, we implemented (in its latest iteration) an EfficientNetv2-based binary classifier: a convolutional network that has faster training speed and overall better parameters efficiency. It uses a combination of better designed architecture and scaling, with layers like MBConv (that utilizes 1×1 convolutions to wide up the space and depth-wise convolutions for reducing the number of overall parameters) and FusedMBConv (that merges some steps of the vanilla MBConv above for faster execution), to jointly optimize training speed and parameter efficiency. The model has been trained leveraging our GPU powered data centers in a continuous exercise of dataset, network and hyperparameters (the settings used to speed up or improve the training performance) optimization. This is a lot of words to say, "we used an off-the-shelf architecture". Like it's literally just Google's public implementation copied over.
- alexmolas 4y agoI guess it was fun to build the training dataset
- yosito 4y agoI was permanently banned from Tinder for trying to upload a picture of myself holding a pepper from my garden. I can only assume that Tinder's AI thought the pepper looked like an unacceptable body part. I am absolutely in favor of using AI to combat unsolicited nudes as long as it's implemented something like: on the sender's side, show a warning before sending that the image appears to be in bad taste and asking for confirmation, then on the receiver's side, hiding the image behind a warning that it may be unwanted and showing two buttons, "show image" and "unmatch". This is how Bumble appears to have implemented the feature. Though personally, I'm not going to test it because I think sending unsolicited nudes is a losing strategy for dating. Yet, as rare as it might be, there are people of both genders who actually enjoy receiving unsolicited nudes. And people who enjoy sexual pickup lines. The judgement call of what is/isn't going to be attractive or appropriate ought be to left in the hands of the individuals dating as much as possible rather than controlled by a priest at a tech company.
- lm28469 4y ago50% of the pictures I upload to discord which contains my hands/fingers are flagged as porn and automatically deleted, it's a major pain in the ass. On the other hand I uploaded artistic nudes a few times with less issues
- npteljes 4y ago>on the sender's side, show a warning before sending that the image appears to be in bad taste Quick feedback like this is what makes people figure out the system very fast, defeating the protection's purpose. It would work much better, if the warning only showed up on the receiver's side - maybe a blank image and the warning, a heavily blurred image and the warning, and a prompt if received would like it to view or not.
- squarefoot 4y ago> I was permanently banned from Tinder for trying to upload a picture of myself holding a pepper from my garden. I can only assume that Tinder's AI thought the pepper looked like an unacceptable body part. If that was a "peter pepper", the AI could have reacted just like a human. NSFW warning in case you search: yes, it can look very close to the real thing. I tried to grow them in my balcony among other types of chili but this type failed probably because of wrong climate and/or bad soil.
- rich_sasha 4y agoCan I use this classifier in reverse, to generate dick pics? Worse still, can I use to generate dick picks that score negative on their own AI? I'll shut up now.
- qubex 4y agoAs a nudist I find it increasingly frustrating battling all these well-meaning impediments every firm seems to be throwing up everywhere for the sake of combating pornography. Latest discovery is that iOS’s new “press to copy person” doesn’t work if the person is undressed, which makes sense if they’re trying to ensure it doesn’t facilitate copy-and-paste porn of the lowest calibre but… sheesh. I don’t understand why so much effort gets sunk into this arms race between people who want to distribute material featuring naked humans and firms that are doing their darnest to make it impossible.
- Natfan 4y agoI'm reminded of the following description of the electronic thumb from Doulgas Adams' The Hitchhikers' Guide to the Galaxy[0]: --- > *Electronic Thumb* > A short squat black rod, smooth and matte with a couple of flat switches and dials at one end. It allows Hitchhikers to flag down passing spacecraft to hitch a lift. > The Thumb is used by Hitchhikers throughout the Galaxy to steal a ride aboard starships who's operators have little interest in taking along extra passengers. > Half the electronic engineers in the galaxy are constantly trying to find fresh ways of jamming the signals generated by the Thumb, while the other half are constantly trying to find fresh ways of jamming the jamming signals. [0]: https://sites.google.com/site/h2g2theguide/Index/e/339375 https://sites.google.com/site/h2g2theguide/Index/e/339375
- qubex 4y agoIndeed. Either way those of my persuasion are the invisible collateral damage nobody cares enough to even factor into the equation.
- grammarnazzzi 4y ago> I don’t understand why so much effort gets sunk into this arms race between people who want to distribute material featuring naked humans and firms that are doing their darnest to make it impossible. "People who want to distribute material featuring naked humans" have many, many, many options. Bumble just isn't one of them. Bumble does "their darnest to make it impossible" because they want to. Perhaps because their livelihood depends on preventing their dating app from turning into a porn app.
- lhenault 4y agoThat's commendable from them and a step in the right direction to address a real problem. Unsolicited nudes are considered as sexual assault, and illegal in many countries. However, I wonder if that's genuine or some PR stunt, as it's "simply" a binary classifier using an off-the-shelf architecture (EfficientNet v2), and having the trained model being publicly released can now allow adversarial attacks.
- dr_dshiv 4y agoI think all would agree that most men would appreciate unsolicited nudity from potential partners and most women wouldn’t. What’s the most likely explanation for this gender effect?
- Hitton 4y agoThis would be interesting 2 years ago, but considering how long is available https://github.com/infinitered/nsfwjs https://github.com/infinitered/nsfwjs and related https://github.com/nsfw-filter/nsfw-filter https://github.com/nsfw-filter/nsfw-filter this seems just like Bumble's attempt to get into news.
- prh8 4y agoPretty embarrassing the number of people here who seem to have no issue with unsolicited nudes. Has anyone ever talked to anyone who's been harrassed, assaulted, stalked, anything? Yikes