11 ms·
Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either be
by Alir3z4 5y ago
Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc.
Rolled out a machine learning model and trained it on the database. 99% of them vanished.
Next day, the machine didn't work and success rate was around 5%.
Found out, they have learned the trick and now using symbols from different languages to make it look like English.
Trained again, success rate went up again.
Next hour, success rate fallen.
This time, they mixed their content with other valid content of our own blogging platform. They would use content from our own blog or other people posts and mix it to fool the machine learning.
Trained it again and was success.
Once a while such content appear and machine model fails to catch them.
It only takes couple of minutes to mark the bad posts and have the model get trained and redeployed and then boom, bad content is gone.
The text extraction, slicing through good content and bad content, finding out symbols vs sane alphabet and many other thing was at first challenging, but overall pretty excited to make it happen.
Through this we didn't use any platform to do the job, the whole thing was built by ourselves, little bit of Tensorflow, Keras, Scikit-learn and some other spices.
Worth noting, it was all text and no images or videos. Once we got hit with that we'll deal with it.
[0]: https://www.gonevis.com https://www.gonevis.com
edit: Here's the training code that made the initial work https://gist.github.com/Alir3z4/6b26353928633f7db59f40f71c8f67cf https://gist.github.com/Alir3z4/6b26353928633f7db59f40f71c8f... it's pretty basic stuff. Later changed to cover more edge cases and it got even simpler and easier. Contrary to the belief, the better it got, the simpler it became :shrug
- oblio 5y agoDo you have a mechanism for appealing the automated process?
- bliteben 5y agowhat are they google? wait google doesn't even have that
- oblio 5y agoMy comment was targeted at companies trying to take Google's crappy approach at this problem.
- Alir3z4 5y agoNot sure if I understand correctly, but if you mean how re-training and deployment would be. Nothing fancy tbh. For the first several deployment while taking care of edge case and debugging, all manually on my own laptop and shot into cluster as a docker image. Later, when starting to classify more content on the platform itself: - Webhook will trigger the CI to train the model with new ham and spam content. - A new docker image was created and deployed to container. The above would fail if success rate in validation/testing would be below 95%. Still when classifying bad content, the whole process happens automated. Special thanks to Gitlab CI, shell scripts, Python and Docker Swarm.
- nerd_light 5y agoI think they meant (and I am interested in hearing about) appealing a "block" decision that was made by your automation. If I'm a real human and trying to post a "good" post, but the model classifies it as bad and automatically blocks it, how do I appeal that decision? Can I? Or is my post totally blocked with no recourse?
- Alir3z4 5y agoOh got it. Thanks for clarification. When a post gets published, it will be send to machine learning image via REST. If bad, the post will be kept as Draft. A new record gets created in another database table to keep track them, the accuracy rate was recorded as well. This was made to make sure no irreversible action was done on the good content. Blogs with more than 1 year of history would not go through moderation but no action was being taken, just recording the accuracy for future reference. Later, someone from our team (me usually) would check them by eye and pull trigger on them, they would go into make the training better. If something would pass the moderation but it was indeed spam, would go into another iteration. We had to do this for over a month, through the time, the success was around 99%, no blogs would be wiped by machine classification from our database unless confirmed by someone. That time the whole model was trained for that specific content. Later it get into other type of spams. Which we trained different models. Overall, the the machine actions were logged, content/users/blogs would get labeled and bad marks on them. They would be displayed in a report page, until someone make the final decision, through the whole time, the user would be shadow banned (shadow banning didn't help though) and their content would not be published.
- vadfa 5y agoWhat software/libraries have you used for your machine learning moderation system?
- Alir3z4 5y agotensorflow and scikit-learn to train and build the model. On the front FastAPI (behind uvicorn) to accept calls via REST API. Deployed via docker. To be honest, tensorflow and scikit-learn may not be the right fit for everything. Every situation needs different approach and different solution. Worth nothing, the most time consuming part was dealing with data itself and not model training or machine learning. In couple of hours you'd notice you're starting at charts and tuning parameters.
- Y_Y 5y agoIf your eyes can "normalize" a unusual symbols to a common one to make an English word then so can a lookup table. I feel like this isn't a case where you'd reach first for a neural net.
- nradov 5y agoThose are known as homoglyphs. This issue has been studied and best practices are documented. There's no complete solution but it can be mitigated. https://en.wikipedia.org/wiki/Homoglyph https://en.wikipedia.org/wiki/Homoglyph
- dredmorbius 5y agoAll kinds of possibilities there. Unicode Character “;” (U+037E) is a beaut. https://news.ycombinator.com/item?id=28621681 https://news.ycombinator.com/item?id=28621681
- Alir3z4 5y agoYeah, then someone has to create or find that whole table and make. The initial problem wasn't those symbols but the content itself, the symbols and special characters came into the problem later. Later on as mentioned in my original comment, that they would use positive content from other blog posts that were published/passed the moderation to mix up their bad content. Probably could use a different method, but at that time needed something quick and fast and it worked and still works with very little tweaking. Although we don't have massive amount of threats or abusers anymore to exactly know the effect, but again, so far it works. That time, they would coming several thousands per minute, IP blocking, range blocking, USER AGENT, captcha or anything such didn't work on them.
- jffry 5y agoThe good news is that the Unicode consortium has a report on this issue, and the tables already exist for normalization and mapping of confusables to their ASCII lookalikes: https://www.unicode.org/reports/tr39/ https://www.unicode.org/reports/tr39/
- drdebug 5y agoWould you be able to share information about the tools you used, they could be very useful for other blogs and platforms it seems!
- Alir3z4 5y agoBasically https://gist.github.com/Alir3z4/6b26353928633f7db59f40f71c8f67cf https://gist.github.com/Alir3z4/6b26353928633f7db59f40f71c8f... but don't think it will be useful for everyone, each usage and requirements is different and needs different solutions.
- drdebug 5y agoThank you! You'd be surprised how useful a starting point can be, thanks for sharing.
- cowmoo728 5y agoI'm interested in why these people were doing this. Were they hoping to get non-tech-savvy people that were searching for computer help? I guess that's a good audience of unwitting users to attempt to hack, but was the goal to get them to submit to one of the remote tech support scams? Were they embedding malware into your blogging platform, or getting ad revenue out of this somehow?
- Alir3z4 5y ago> Were they hoping to get non-tech-savvy people that were searching for computer help? Yes. They would create this posts and get quickly on search results (The platform is pretty good for making SEO optimization out of the box) and they would write good quality posts as well. They would also share this posts on some other websites, especially social media accounts. We don't have Google analytics or such to see where exactly they would come from. I noticed huge traffic to such pages by looking at the logs. Our nginx log parser was alerting us about sudden spike on certain blogs and pre-defined list of words we have. That's when we noticed something is going on. Didn't take more than couple of hours (while working on the model) that we receive email from data center people about hosting phishing content, again didn't take much longer we received emails from some of those companies as well. > Were they embedding malware into your blogging platform, or getting ad revenue out of this somehow? No. On the blogging platform, we have everything bleached out, nothing would go in without passing through sanitizers. They would simply had people convinced to call those US numbers. I actually called one of those numbers and yeah, it was one of those customer supports some other part of the planet earth and definitely not from the company he was pretending to be and very quickly asked me to install team viewer on my machine. I really wanted to let them access it via the windows on my virtual box and have some fun with them, but well, someone had to fix the moderation issue :D
- neilv 5y agoWhat was the motivation of those attackers?
- YetAnotherNick 5y agoHow do you detect the ground truth for training the model? Do you manually label it?
- Alir3z4 5y agoYes, simple classification. Nothing fancy. Basically, pulled the database into CSV file and anything that was published before the bad content was classified as HAM. We had content that were OK, so marked as HAM and then our new bad content all marked as SPAM. When deployed to production for some hours HAM content got wrongly marked and model got trained on them as well which made so many confusion but the problem was taken care of once the model got properly tuned and safer to let it be automated.
- benjaminjackman 5y agoHmm I wonder if it picked up timestamps as its initial filter.
- tdeck 5y agoFor adversarial problems like this, a shadowban approach can sometimes be necessary. Perhaps people can still see their blogs but GoogleBot gets blocked from indexing them, or they only appear to someone with the spammer's cookies. That way it takes them longer to catch on and evade the model. Of course, that means you'll need to at least spot check your bans because you can't rely on legit users escalating to you.
- mrkurt 5y agoWe shadow ban abusive users on Fly.io and it works great. Everything seems to work right up until they try to connect to a deployed app. It took me a while to realize that ramping up the frustration level is, itself, a helpful deterrent.
- Nextgrid 5y agoAre you doing this because of resource usage? If resources are free then you could even actually deploy their app and either whitelist it for their own IP or only allow very few requests before taking it down. This would be even more frustrating and could ruin whatever they plan to do with their abusive app in the first place. Let's say they deploy their malware/phishing page, test it a couple of times (possibly from a different IP) and it works. They then start spamming the malicious link and waste decent amounts of time/money/processing power, not realizing that the link was dead after the first 10 hits.
- mrkurt 5y agoWe're primarily trying to prevent fraudulent payments combined with expensive VMs. Throttling CPU to almost nothing on high risk accounts sounds delightfully irritating. We also get the less resource intensive, but still harmful abusive apps that port scan the internet. Those are relatively easy to detect. We generally don't want to be a source of port scans so we shut them off pretty quickly.
- Nextgrid 5y ago
- pbreit 5y agoHow did the lousy content affect your legit users?
- Alir3z4 5y agoEvery blog is pretty isolated from other users. It's not that the content will be popped up to everyone when someone posted something. There's a Feed page where you can read what others you follow have published. There's a Explore page where latest content without any filter or categorization would be visible. This is where such content would appear, but only blogs older than 7 days would appear there (we have removed that delay in recent versions). Basically no one noticed them. Although we disclosed the issue we were dealing with some of the old users of platform when they complained about their posts not getting published. That was the first issue in first 15 minutes of the machine learning model classifying wrongly due to being fed mixed content (where bad content was mixed with good content from those exact blogs by spammers.) Other than several bloggers reporting their posts wouldn't go through as expected, no one else got effected and I hope no people were lured with those scammer while their content was published on our platform.
- 5faulker 5y agoTale of humans man...
- HodorTheCoder 5y agoIf you don't mind me asking, what sentence embeddings model (bert/roberta/etc) did you have the best luck with for your classifier? I like the quick retrain that can be done with an approach like this, though I have found that if you throw too many different SPAM profiles at a classifier it starts to degrade, and you might have to build multiple and ensemble them. The embedding backend can help a lot with that.
- Alir3z4 5y agoTried bert but didn't get the proper result, probably wasn't working with it properly. Here's the old source I have on my computer that did the training https://gist.github.com/Alir3z4/6b26353928633f7db59f40f71c8f67cf https://gist.github.com/Alir3z4/6b26353928633f7db59f40f71c8f... This was doing the early work and later changed more to fit other cases. Pretty basic stuff.
- KorematsuFredt 5y agoDid you try rate limiting, shadow banning, ip banning etc?
- nothrowaways 5y agoI would recommend you a different approach, such as using metadata like their location etc.
- Alir3z4 5y agoIt was a coordinated attack from many different locations and IPs. The IP blocking was in place on application level and later into Cloudflare blacklist. Still they would flood in with different IPs and browsers.
- nothrowaways 5y agoStill i would consider working on such metadata. I have seen many interesting results based on this.
- deleted 5y ago[deleted]
- bcrosby95 5y agoWhen we had this problem, we added an input element positioned off-screen and ignored submissions that populated it. Cleaned it all up.
- deleted 5y ago[deleted]
- ummonk 5y agoThis story needs to be turned into a writeup and submitted to HN on its own.
- LuisMondragon 5y ago> Found out, they have learned the trick and now using symbols from different languages to make it look like English. I wonder if you can train a ML model using text as images. For example, taken as strings, "porn" and "p0rn" are not very similar, but visually they are.
- cutemonster 5y agoThat could detect bad text in actual images too
- seanmcdirmid 5y agoNot to mention “pom” in certain fixed width TTY fonts.
- vanderZwan 5y agoAre you familiar with the Xerox copying flaw? Not exactly the same problem and unlikely to apply here but still a fun tangent: https://www.theregister.com/2013/08/06/xerox_copier_flaw_means_dodgy_numbers_and_dangerous_designs/ https://www.theregister.com/2013/08/06/xerox_copier_flaw_mea...
- neiman 5y agoI wonder if it also ends up blocking legit content?
- tootie 5y agoI'm curious why you'd roll your own at all when there are moderation services available. Did you just have a use case that didn't match anything on the market?
- asiachick 5y agoand then your filter has a few false positives and you get cancelled as people claim you're anti <fill-in-the-blank>
- nbardy 5y agoThat sounds like a killer saas startup