8 ms·
I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a mod
by pfooti 3y ago
I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1.
I was keeping an eye on it, because we are hard-capped at 100 QPS to our provider, beyond that and they start dropping our traffic (and it is an outside provider, bundling domain registries like verisign and stuff), which makes regular users break if their traffic gets unlucky.
Anyway, after a week of 40qps, they start spiking to 200+, and we pull the plug on the whole thing: now each request to our endpoint requires a recaptcha token. This is not great (more friction for legit users = more churn) but it is successful. IF they had only kept their QPS low, nobody would have cared. I wanted to send some kind of response code like, "nearing quota".
FTR before people ask: it was quite difficult to stop this particular attack, since it worked like a DDOS, smeared across a _large_ number of ipv4 and ipv6 requesters. 50 QPS just isn't enough quota to do stuff like reactively banning IP numbers if the attacker has millions of IPs available.
- pierat 3y agoQPS seems to be "queries per second". I first thought it was something like "quadrillion bytes per second" or some newer over-the-top data measurement :)
- vaylian 3y agoI haven't heard of that abbreviation before. But there is a stub on Wikipedia if anyone cares about improving that article: https://en.wikipedia.org/wiki/Queries_per_second https://en.wikipedia.org/wiki/Queries_per_second
- sbierwagen 3y agoTemporarily add 500ms of latency to all ipv6 users, backoff timers for ipv4 addresses. Since there's only 4 billion v4 addresses, it's easy enough to just track them all in a sqlite db.
- Taek 3y agoI don't think that works if your attacker has millions of IPs and is only using 200 per second.
- akiselev 3y agoHow are they getting millions of ipv4 addresses? IIUC that’s at least the equivalent of a /12 block. Do those shady residential proxies really operate at that scale? If they’re ipv6 address wouldn’t they be safe to block across large ranges?
- slivanes 3y agoMy guess through compromised boxes (or some apps) that act as proxies.
- reaperman 3y agoIt is trivially easy to get millions of IPv6’s, even spread out across a thousand ranges. It is also trivially “easy” to get past reCAPTCHA, but it costs more. My guess is that a domain name checker tool isn’t worth the cost per request to bypass reCAPTCHA (approximate 0.02 cents per session)
- jszymborski 3y agoRight, but this is why the GGP's post suggested throttling IPV6. Or at least that was my understanding.
- hattmall 3y agoCaptcha seems like overkill. Were you not able to implement a JS fingerprinting / bot check before captcha.
- reaperman 3y agoMost of the time, the new CAPTCHAs do that and then never show an actual interactive element. The interactive gimmicks are a fallback in cases of high uncertainty. The webmaster doesn’t need to worry about it, the anti-bot services handle who gets what difficulty of challenge. But the webmaster can specify whether they’d like to be more or less strict/difficult than usual.
- pfooti 3y agoYeah, "recaptcha" in this case is shorthand for a lot of stuff we did to harden the endpoint that ultimately represents a minor shift in balance from no friction to some friction for otherwise legitimate users, but a pretty significant drop in illegitimate traffic. The main idea here is that at some point just leaving the scrape running in a way that didn't overwhelm my backend would have resulted in me not caring enough to do that. But now they get nothing. Even if you're borrowing bandwidth and not paying, you should be a good neighbor is all.
- fragmede 3y agodid you ever look inside those queries? were they the same, just repeated ad nauseam just polling in case their status had changed?
- jb_gericke 3y agoWhy not put auth on the endpoint and enforce quotas and rate limiting (an api gateway like kong could handle this for you).
- pfooti 3y agoEndpoint was invoked in our signup funnel, so there was a bootstrapping problem for quota enforcement, the attackers weren't making a whole signup, just getting to the point where the domain search ran.
- boulos 3y agoRate limit and return a 429?
- pfooti 3y agoSee sibling, but the endpoint was part of a signup funnel, so short of rearchitecting it completely to put that check after customer creation, there's no real persistent key to rate limit on. Any one IP ended up getting rate limited to 5 requests per hour on that API, but the attack was incoming from what looked like a botnet, so it was tricky.
- rapnie 3y agoMaybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher respect for privacy and improved a11y compared to hCaptcha). [0] https://mcaptcha.org/ https://mcaptcha.org/
- toastal 3y agoIf it doesn’t involve my free labor training machine learning models like reCAPTCHA and hCAPTCHA and I can still visit sites from a non-Western IP, then it’s already an improvement.
- 8organicbits 3y agoReally interesting! How efficiently does a web browser compute the PoW? I'm concerned that a bot would use an efficient GPU implementation while real users would run an inefficient JS/webcrypto version.
- jchw 3y agoI tried to implement PoW in browser for the same concept. I think it's probably at least marginally useful, but practically you're limited by WebCrypto I think, which outperformed anything I could find in pure JS or WASM. The disadvantage of WebCrypto is that there's a limited set of algorithms you can use and also, understandably, calculating hashes is async, so if you want a lot of rounds, you'll spend a lot of time jumping in and out of the event loop. It's still probably a useful speed bump or price increase for expensive operations: what might cost a few seconds on a typical phone or desktop is probably at least enough to act as a speed bump for attackers to perform it quickly (especially combined with other measures like throttling and progressively increasing PoW difficulty for an IP.) Maybe WebGPU could change this, but I'm weary of relying on all users having a fast GPU enabled in their browser just for this use case. Though thinking about it, I wonder if there is a hybrid: start with a difficulty that's just a few seconds for CPU/WebCrypto and ramp up quickly, but also support WebGPU where possible so that web users on abusive connections may still succeed? I am not sure though, I guess this depends on the feasibility of using WebGPU and etc.
- Szpadel 3y agoI had similar issue for one of our clients. My strategy to not affect legit users was to enable mitigations if global traffic crossed some threshold. eg. in your case this could mean if traffic is above eg. 75QPS then captcha is enabled, and if it's below that it's disabled. I don't know what tech stack you are using, but nice trick that i figured out was to abuse rate limiting to detect global traffic (doing if branch with rate limit with const as client id)
- pfooti 3y agoThat is more or less what I landed on myself. (Not quite, but similar reactive configs based on traffic thresholds). For a while, we had to just set off pagers when global traffic exceeded a threshold and manually toggle the extra hardening, but eventually it became a lot more reactive.
- ricardo81 3y agoUnderstandable. I've seen services that offer residential IP address proxies for as low as $1/GB. FWIW the particular service in mind actually pays the IP owners who opt into it. I guess your tool is asking the registries if domains are registered.
- WA 3y agoI usually make it a two step process: host example.com If this returns with an IP address, no need to talk to a registrar. Only if there is no IP address, I go with whois example.com
- schoen 3y agoYou could also do an in-between step of host -t soa example.com which should give you domains that have any DNS record at all, not just an A record.
- remram 3y agoIs there still a point talking to the registrar then? How common is it for a domain to exist with the registrar but have no SOA record?
- schoen 3y agoHmmm, I'd have to think about this more to be sure. I'm not immediately positive of the answer. If the registrant creates any DNS records at all, then the SOA will need to exist for the zone to be valid; but I don't recall whether the registrar is or isn't required to publish a zone for a registered domain that otherwise contains no records. (Also whether the registrar is required to inform the registry of authoritative nameservers for the domain at all times from the moment that the domain is registered, and then whether that information would lead to the synthesis of a SOA record.) I guess I can either try this (if I could find a no-frills-enough registrar that doesn't create any records at all for add-on hosting services or domain parking) or try to take some more ICANN coursework to find out the answer. Or maybe someone else reading this thread knows whether we can have a domain in practice that is registered but has no published SOA.
- Pmop 3y agoThere's a demand, why not supply it, and make money while you are at it? This reminds me of the RMT driven botting problem in WoW (World of Warcraft). Instead of fighting the neverending game of cat and mouse against botters, Blizzard just decided to supply the long reprimanded demand for in-game currency by creating the WoW token, and they make money while they're at.
- emodendroket 3y agoIf such a simple mitigation as adding captchas worked then it seems like the value is pretty marginal, plus it sounds like the limit is upstream.
- tyingq 3y agoIf you can detect it reliably, poisoning them with bad answers might be enough to cause them to find an easier target.
- dspillett 3y agoBe careful with that. A friend once did something similar to an active scraper who then went off and set his site up for a full blown DDoS. He knew that was the reason for the intentional DDoS due to messages (along the lines of “think it is funny to poison my information do you?”) in the query string of the bulk requests. Like unpleasant fools making a big noise in shops because they aren't served immediately after cutting in line or some such, the entitled can be quite petty when actively taken to task for the inconvenience they cause. Passive defences are safer in that regard, assuming you don't get into an arms race with the scrapers, though are unfortunately more likely to mildly inconvenience your good users.
- tyingq 3y agoThat's fair. Perhaps a middle-ground of poisoned answers that look like overload related errors. Though I did have success once stopping hot-linked images by serving up images that didn't fit well with the sensibilities of the internet forum that was hotlinking them. It had the advantage of looking like the people in the forum posting had deliberately chosen to post the image. Serving different images depending on client ip made for fun too, as they argued with each other about why they posted "that".
- dspillett 3y ago> Serving different images depending on client ip made for fun too, as they argued with each other about why they posted "that". Evil. I like it.
- scotty79 3y agoI think you could make your requests two step. If somebody wants to access endpoint you might send him a challenge first. Random text. The client must append to the text some other text chosen by him, so that when you calculate sha256 on concatenated text, first byte or two of it will be zeros. To access your actual endpoint client needs to send that generated text and you can check it if it results in the required number of zeros. You might demand more zeros in times of heavier load. Each additional bit that you require to be zero increases number of random attempts to find the text by a factor of two. To make stuff easier for yourself the challenge text instead of being random might be a hash of clients request parameters and some secret salt. Then you don't have to remember the text or any context at all between client requests. You can regenerate it when the client sends second requests with answer. Honestly I don't know why this isn't a standard option in frameworks for building public facing apis that are expected to be used reasonably.
- hombre_fatal 3y agoBecause it doesn’t accomplish anything. Things take longer for honest users while botnet abusers don’t even notice that the rented hardware is burning more CPU. Nor does it matter because each request still goes through.
- scotty79 3y agoFor honest users it's not noticeable in normal conditions. But the DDOSer will DDOS themselves. They might make your service slow down for the users when they attack but not because your server gets swamped just because you choose to increase the difficulty of the challenge while you are attacked. DDOSers will notice because they won't be able to make requests as fast. At least not the ones that are costly for you. It's an enforced client side throtling that affects even bad actors. Generating challenges is really cheap. Calculating a single SHA256 or sth out of a querystring+salt (or better yet 128bit SipHash). Generation and validation can be done on separate layer/server so requests without valid PoW won't even register on your main system.
- hombre_fatal 3y ago
- elliotto 3y agoIt's always interesting to see these prisoner's dilemma / tragedy of the commons show up in networking. If they hadn't abused the commons it would have worked out better for everyone.