5 ms·
Off-topic, but our freemium website is under attack by headless browsers. The freemium service provides access to compute-heavy machine learning models running
by headlessvictim2 5y ago
Off-topic, but our freemium website is under attack by headless browsers.
The freemium service provides access to compute-heavy machine learning models running on GPUs.
Hackers blast 50-100 requests in the same second, which clog the servers and block legitimate users.
We reported IPs to AWS and use Cloudflare "Super Bot Fight Mode" to thwart attacks, but the hackers still break through.
We don't require accounts, but could impose account requirements if this helps.
Any suggestions?
- cmeacham98 5y ago100 requests/second isn't that much, especially if you're fronting your website with Cloudflare. Do you have some unauthenticated endpoint(s) that eat up a ton of server CPU?
- headlessvictim2 5y agoThanks for the reply! The freemium service provides access to machine learning models on GPU instances, served with FastAPI. Each request invokes a compute-intensive ML model, but perhaps there is something wrong with the FastAPI configuration as well?
- tempest_ 5y agoIt could be. I watch the FastAPI repos a lot and tones of people do not understand how async python works and put their models with sync code in an async context.
- headlessvictim2 5y agoConsider us one. :) We tried removing "async" -- thinking it would force sequential processing -- but it unexpectedly seemed to cause parallel processing of requests, which caused CUDA memory errors. Before removing "async", this is the weird behavior we observed: * Hacker blasts 50-100 requests. * Our ML model processes each request in normal time and sequentially. * But instead of returning individual responses immediately, the server holds onto all responses -- sending responses only when the last request finishes (or a bunch of requests finish). * Normally, request 1 should return in N seconds, request 2 in 2N seconds, but with this, all requests returned in about N50 seconds (assuming batch size of 50). 1. Any suggestions on this? 2. Mind clarifying how sync vs aync works? The FastAPI docs are unclear. Any help would be much appreciated. This has been extremely frustrating.
- azinman2 5y agoWhat are the bots goals? Curious
- headlessvictim2 5y agoTo use premium settings without paying. It appears less like malicious DDoS and more like pragmatic theft.
- Nextgrid 5y agoCould you add a symbolic, one-time fee to the "free" tier to deter multiple accounts and then implement reasonable rate-limits per-account?
- Nextgrid 5y agoAny chance the entire thing can be offloaded to a task queue (Celery/etc)? This would decouple the HTTP request processing from the actual ML task. The memory errors you're seeing could suggest that you may not actually be able to run multiple instances of the model, and even if you could it may not actually give you more performance than processing sequentially. Seems like ultimately your current design can't gracefully handle too many concurrent requests, legitimate or malicious - this is a problem I recommend you address regardless of whether you manage to ban the malicious users.
- pjgalbraith 5y agoYeah this is the way. @headlessvictim2 search for "Asynchronous Request-Reply pattern" if you want more information about this kind of architecture. You will remove any bottleneck from the API server and can easily scale out from the task queue.
- headlessvictim2 5y agoThanks for the suggestion. How would this work with GPU-bound machine learning models? The model processing takes > 30 seconds and would still represent the bottleneck?
- austincheney 5y agoBrowser automation will occur by executing events in the DOM or by calling properties of the page/window. It’s all JavaScript designed for user interaction executed by a bot. The one event that cannot be automated is cursor movement/position. Put a check into your event handlers that check that the cursor is actually over the event target.
- headlessvictim2 5y agoThis is interesting. Thanks for sharing. Are you saying block form submission unless the cursor is over the event target? If so: * How to handle legitimate requests from mobile users? * How to handle form submissions with the "return" key?
- austincheney 5y agoMobile users will use touch events instead of click events and likely your interface will be different and the screen width will be different. Check for these things along with keywords from the user agent string to determine mobile users from other users. Return key on a control in a form will fire a submit event. Check for cursor position in your submit handler.
- darkstar999 5y agoThat sounds like an accessibility problem.
- austincheney 5y agoUse an alternate control for keyboard navigation that is visually hidden and is accessed only by tab focus.
- mesadb 5y agoYou are right every testing solutions out there push UIEvents to the page rather than clicking with an actual mouse. That's why puppeteer, selenium etc are scraping tools not testing tools
- slig 5y agoJust block the AWS ASN on CF, it's nor worth fighting.
- rob-olmos 5y ago+1 and GCP, and many other hosting ASNs
- rabi_molar 5y agoPerhaps Captcha?
- headlessvictim2 5y agoThanks for the suggestion. It is possible, but this degrades the experience for legitimate users. We prefer solving this without impacting/taxing normal users if possible.
- geraldwhen 5y agoRecaptcha v3 doesn’t prompt if it thinks you’re a real user.
- Nextgrid 5y agoThis could have major GDPR implications if that's something the parent cares about. ReCaptcha is basically Google spyware that happens to provide captcha services.
- waynesonfire 5y agoit's obvious the parent is trolling.
- forgotmyoldacc 5y agoWhy not ReCAPTCHA?
- headlessvictim2 5y agoThanks for the suggestion. It is possible, but this degrades the experience for legitimate users. We prefer solving this without impacting/taxing normal users if possible.
- readyplayeremma 5y agoJust add the captcha only for requests coming from the problematic ASNs, like AWS. edit: Actually, since you use CF, just make a firewall rule that forces the captcha for those ASNs before it even gets to your app. They have a field named "ip.geoip.asnum" for that, and an action called "challenge" which will force a captcha.
- deleted 5y ago[deleted]
- synergy20 5y agowhat's your site? would like to play with it
- 1vuio0pswjnm7 5y agoTell freemium users what is the acceptable rate for requests per second. Publish the allowable rate on the website. Ban freemium user IPs that exceed the allowable rate. This can be done using a proxy.
- headlessvictim2 5y agoA proxy like Cloudflare or a custom proxy that stores data? Are there proxy examples you could point us to? Thanks for your help.
- 1vuio0pswjnm7 5y agohttps://www.haproxy.com/blog/four-examples-of-haproxy-rate-limiting/ https://www.haproxy.com/blog/four-examples-of-haproxy-rate-l...
- headlessvictim2 5y agoThank you for this.
- machiaweliczny 5y ago429