5 ms·
We had a client who contacted with an issue: site goes down every 15 minutes. After couple of hours of debugging we found the culprit. Some swedish equal rights
by DocG 9y ago
We had a client who contacted with an issue: site goes down every 15 minutes. After couple of hours of debugging we found the culprit. Some swedish equal rights server was "scraping" site every 15 minutes with massive ddos. We put capacha for the server ip only (let them scratch their heads now) and let it be. If you decide to scrape a site dont bring it down or assume its a big server. Keep it civil and without impact on performance and nobody minds.
- akerro 9y agoWhy have you not reported it to them or/and police?
- Geee 9y agoYou don't need police in environments where you can defend yourself.
- fao_ 9y agoOk, so for them it is no longer a problem, but what about other people? It is their ethical duty to report them to the police, otherwise they were an accessory in the crime (In some countries there are laws to enforce this)
- tokenizerrr 9y agoIf you make a shitty site that can't handle a few concurrent requests this is your problem. It's not a DDoS. Not even a DoS. Just trying to request some pages you make available, and you failing at doing so. Or do you also plan on suing Google once they scrape your site?
- fao_ 9y agoLooks like somebody didn't read the parent. > "scraping" site every 15 minutes with massive ddos Google doesn't scrape every 15 minutes. It was implied that they scraped the site often enough to cause a DDoS.
- tokenizerrr 9y agoLooks like someone has never ran a website with even a slight amount of traffic. A few requests every 15 minutes is nothing and your site should be able to handle it. Additionally, it clearly wasn't a DDoS. Quoting from the parent: > Some swedish equal rights server was "scraping" site every 15 minutes with massive ddos. We put capacha for the server ip only (let them scratch their heads now) and let it be If there is only one server ip, it is by definition not distributed. Just a misuse of the term DDoS.
- thephyber 9y ago> Google doesn't scrape every 15 minutes. Google has a variable timing crawler. Google News crawls in almost realtime, but all Google crawlers back off if the site responds slow. Additionally, Google respects `/robots.txt` (which allows the site owner to define a crawl delay) and uses sitemaps for hints, so it doesn't have to re-crawl every single HTTP object "every 15 minutes".
- DocG 9y agoWe were contacted as outsiders to come in and have a look. We managed to stop site from crashing with configuring but basically the site was bombarded with too many dynamic requests and having no cache it took its toll. Server did not wait for responses, it just sent us bunch of requests all at once.
- tokenizerrr 9y agoAny decent webserver can throttle requests per source ip.
- robalfonso 9y agoYes, agreed though you must also take into account the idea of the "straw that breaks the camels back".
- DocG 9y agoWe gave the info to the client. We were in no position to report it.
- tenkabuto 9y agoOn this note, does anyone know of a drip/gradual scrape process for not overloading a host's servers, such that the process could run over the span of a day/week? I'm not worried about the content changing rapidly.
- gwbas1c 9y agoUhhh, have a brain and common sense when you write your scraper? The simplest thing to do is NOT parallelize your scraper, and sleep for a few seconds between requests.
- deadbunny 9y agoHow is a single server a ddos? Did you mean "We couldn't handle the traffic caused by their scraping which lead to a denial of service"?
- DocG 9y agoYou are correct. Dos was correct term
- jackschultz 9y agoCorrect call on keeping it civil. I know for all the scraping I do I put my name and contact info in the header so if people try to look and find an issue they know how to get in contact. But I also make sure I'm not making requests that would ever affect their servers. Gotta stay nice.