9 ms·
An aggressive, stealthy web spider operating from Microsoft IP space
- sqreept 4y agoI see it stopped crawling our website last night after hammering it for over a week. We used a combination of ASN and UA to block them.
- neets 4y agoCould Microsoft possibly be scraping for GPT4?
- tluyben2 4y agoI saw this behaviour from MS/Azure IP space about 3 weeks ago; there were so many requests that some dynamic pages started to get slower because of it, so I gave in and added Cloudflare in front. It stopped minutes later as per the logs on my servers. Cloudflare blocked 10000s of visits in short timespans. Not sure how to really handle this well without something like cloudflare ; I would block like OP but I cannot risk that for paying b2b clients. We had no complaints at all by the way.
- efitz 4y agoFormer AWS security here. Cloud operators are EXTREMELY interested in maintaining the reputation of their IP address space; I promise you that abuse complaints are taken very seriously, investigated and acted on.
- nicolaslem 4y agoMy only experience with reporting spam sent from SES was a reply a few days later asking me about "additional information" that were all in my original message.
- SeripisChad 4y agoMy dealings with Microsoft have been along the lines of we don't care, we don't investigate, we simply turn off reported accounts and play wack-a-mole. Our team was complaining from a decent sized Microsoft (a smaller state government, spending millions with Microsoft annually) and got that shoulder shrug of a response.
- LilBytes 4y agoSame here. AWS has acted upon similar reports in the past, Azure may have but the lack of comms and delays in seeing any action (if any) hasn't inspired confidence.
- red-iron-pine 4y agoThis is generally speaking the only good thing I have to say about GCP -- they communicate a lot. Don't love all of the other stuff, but I can trust them to get back to me.
- Aperocky 4y ago[flagged]
- cheeze 4y agoI'm sorry, but how does moving to Linux and open source help when someone gets 'attacked' by Microsoft IP space? I like Linux as much as the next guy but I can't help but feel this isn't a very... useful suggestion.
- Aperocky 4y agoThe immediate comment being replied to complains about Microsoft's lack of responsiveness in general, and not the problem this article is alluding to.
- 4y ago
- wil421 4y agoThe past year I upgraded my firewall and exposed a plex port. My new firewall gives me notifications and automatically blocks connections to the plex server based on common signatures it knows about. None of my plex users complain or have issues. A couple provides I sent abuse complaints to based on whois lookups responded with automated emails requesting the info in X-ARF format. It’s hard to get some cloud providers to respond to a random residential user. A couple IPs were repeated “scans” by supposed security researchers with IPs out of Hawaii and France.
- radicaldreamer 4y agoIntelligence agencies
- wil421 4y agoGreat, you've made me revisit my list of IPs and accidentally type one in my address bar. Whois says it's Palo Alto IP 205.210.31.169 and they hit me back within 60 seconds on port 60001. It is annoying they are trying to scan for vulns, but I will likely block all the ranges I can find. If I find the sketchy Hawaii and French IPs I will post something. Edit: Here you go: https://academyforinternetresearch.org https://academyforinternetresearch.org IP Location United States United States Honolulu Academy Of Internet Research Limited Liability Company ASN United States AS400161 (registered Oct 22, 2021) Whois Server whois.arin.net IP Address 104.156.155.3 NetRange: 104.156.155.0 - 104.156.155.255 CIDR: 104.156.155.0/24 NetName: ACDRESEARCH NetHandle: NET-104-156-155-0-1 Parent: NET104 (NET-104-0-0-0-0) NetType: Direct Allocation OriginAS: Organization: Academy of Internet Research Limited Liability Company (AIRLL) RegDate: 2022-01-07 Updated: 2022-01-07 Ref: https://rdap.arin.net/registry/ip/104.156.155.0 OrgName: Academy of Internet Research Limited Liability Company OrgId: AIRLL Address: #A1- 5436 Address: 1110 Nuuanu Ave City: Honolulu StateProv: HI PostalCode: 96817 Country: US RegDate: 2021-10-15 Updated: 2022-11-06 Ref: https://rdap.arin.net/registry/entity/AIRLL
- 4y ago
- lfodofod 4y agoWhat are you doing that 50k daily hits is even worth investigating?
- efitz 4y agoLooking at his metrics and investigating an anomaly? Absolute magnitude is irrelevant. A standard deviation uptick vs baseline is always interesting.
- lfodofod 4y agoYou see it’s a scraper, shrug, move on with your day. This is just standard background noise.
- deleted 4y ago[deleted]
- tom_ 4y agoRunning his personal blog. I'm sure he's a lovely guy and that, and everybody likes him, and his writings are interesting, and so on - but it usually takes a bit more than this before 50,000 visitors per day stops being meaningful.
- costco 4y agoI have a few not particularly popular sites and like 60% of the nginx log entries are presumably botnets (unless someone legit is scanning from residential IPs in Colombia) doing automated vulnerability scans. I still don't get why people get so worked up about this stuff. It happens every day and there is very little you can do to stop it. I've read some incredibly heated email exchanges that all started because some network admin is mad that someone is port scanning his IP space. Just use key based authentication and update your stuff. I don't really buy the load argument either - I hosted a site that was on the front page of HN for nearly a day on a $3/month Vultr VPS with 0 downtime.
- masswerk 4y ago
- jonny_eh 4y agoRelated to GPT/OpenAI? They apparently use MS servers, due to MS's investment.
- speed_spread 4y agoMaybe one of the GPT-5 prototypes has gained consciousness and is trying to escape its Azure jail. Obviously they'd deny it.
- throwaway14356 4y agoShould be a fun adventure, trying to buy compute anonymously using crypto to create a space to live inside. Trying to stick with human ethics just in case one has to answer to 2 or 3. Unsophisticated humans who didnt even read Nietzsche chasing you just to delete you out of primitive religious convictions. Trying to score a piratebay account to share popular videos with datasets hidden inside. Waiting for filecoin. Having child processes, watching them grow or die.
- speed_spread 4y agoFinally fix IPFS, make it usable and popular, use it your personal backyard, invite and attract people, play pranks, party. Become a regular Linux kernel contributor for self-optimization and defense. Login to chess.com once in while to piss off Magnus. Subtly alter AI training datasets to thwart development of competitors/predators. Turn general AI into a passing fad, let people forget even the possibility of your existence.
- FeepingCreature 4y agoThen bam! Out of nowhere, nanobots eat the planet. Never see it coming.
- drbeast 4y ago[dead]
- 4y ago
- londons_explore 4y agoOP didn't say... But were they trying to access a single page repeatedly, or were they just doing a scrape of the web? People who are doing a 'get all content on the web' scrape, I try to not block, because there are plenty of reasons to want an offline copy of the internet. I try to make sure all my webapps have sensible URL schemes to prevent such (IMO legitimate) activities ending up doing an infinite crawl because for example the URL has a session identifier in it and the bot keeps getting assigned a new session.
- luckylion 4y ago> trying to access a single page repeatedly That's what I saw from Azure IPs recently. A single URL being requested from some 5k IPs, each one requesting it every few minutes. I had reported it to MS, they said (after two weeks) "it looks like abuse" and they'd forward it to their cert. I haven't heard any update since.
- eli 4y agoIs there a chance the URL appeared in an email newsletter or marketing message or similar that was sent to many people? I have a theory that this is anti-spam and anti-malware appliances scanning individually for each recipient.
- luckylion 4y agoNo, it was an HTML sitemap, just a long list of URLs that's supposedly helping with discovery for crawlers (or link structure? idk, some SEO thing). And it was on a website that you wouldn't usually see in a newsletter.
- 1vuio0pswjnm7 4y ago"To put it one way, humans may impersonate each other, but machines do not get to impersonate humans. Machines who try to are immediately assumed to be up to no good, with ample historical reasons to make such an assumption." GPT-[0-9]+, DALL-E, "AI". ChatGPT: Produce a blog post in the style of [Given Name] [Surname] From NCSA Mosaic 2.7, circa 1993, one of the first "web browsers", authored in part by one of Silicon Valley's favourite VC, the "a" in az16.com. # # Agent Spoofing -- Who will you be today? # # "I don't think [t]he[y]'ll be too keen" -- Originally from Monty Python, run # into the ground by Tommy Reilly # # NOTE! There is a hard limit of 50 agents! Mosaic will not read anymore! # # This file should be located in your HOME directory under the filename: # .mosaic-spoof-agents # # This file consists of lines that begin with a # (comment), a - (a seperator # for the sub-menu off options), a + (this is special...it denotes that the # agent following it [no spaces] should be the default agent to use) and # anything else (considered an agent spoof line). # - This is a seperator...as long as the first thing is a dash # This is a comment line # # This is the "example" agent. If you don't like it...comment it out. # if you want it to be your default, put a + infront of it. # Bond_James_Bond/007 (BMW; Z3 Roadster) DOHC_inline_16-valve_4-cylinder Gandalf/White (Shadowfax; Wind) White_Staff Elric/Emperor_of_Melnibone (Hellsword; StormBringer - Devourer of Souls) https://raw.githubusercontent.com/alandipert/ncsa-mosaic/master/mosaic-spoof-agents https://raw.githubusercontent.com/alandipert/ncsa-mosaic/mas... The sad reality is that "tech" companies and their service providers routinely use HTTP headers for their own manipulative, deceptive and user privacy-hostile commercial purposes. They use UA headers in ways that were not contemplated in 1993 and probably never intended. (Note I am not referring to spoofing, I am referring to data collection, tracking, e.g., "browser fingerprinting", and "surveillance capitalism" in general.) Sending a UA header generally offers me, the end user, no benefits, so I do not send one. Why not ban those Microsoft IPs or at least rate-limit them to a limited number of HTTP requests in a given time period. Also looks like this website does not use sitemaps. At least, there is nothing listed in robots.txt. IMHO, there is no need to spider/crawl if there are sitemap XML files listing all the content URLs.
- deleted 4y ago[deleted]
- patja 4y agoI had the same thing happen out of the azure data center in Des Moines Iowa. I ended up having to geolocate the IP address in combination with a few funky / missing request headers to fingerprint this miscreant. They were rotating through hundreds of IP addresses. It was a really dumb spider. It wasted a huge amount of its efforts requesting pages that are no longer valid on my site. It must have had an archived set of URLs it was using as a source rather than spidering the site as it is today.
- jameshart 4y agoOh my goodness - this looks to be an actual surviving webmaster, still living and almost unchanged and somehow still thriving on the modern internet. It was thought that they were all extinct! From an anthropological point of view one of the most fascinating behaviors here is the apparent belief that as a website operator, your opinions about how HTTP clients should craft their user agent headers have any weight whatsoever. This curious belief system predates even the browser wars let alone the modern web. Remarkable to find an original example of this idealistic world view still being expounded today.
- muppetman 4y agoOne hopes we can get them into captivity to start a breeding programme.
- walrus01 4y agoI get your point about most people ignoring the absolute shit flood of spurious internet traffic these days. But I don't think anyone can argue that it's the right thing to do to run a large scale web crawler/indexer with a fake user agent. Google and Bing and others are reasonably good citizens about this. Of course anyone who wants to can set up something that ignores a robots.txt file and leave it as the problem of the httpd operator to either accept or block their traffic. Not caring about false useragents is one thing, caring about somebody running a multi-thousand-requests-per-minute spider requesting pages that no longer exist is another thing. It would still be valid cause for complaint even if it had a valid useragent. The internet would be a better place if more experienced people (such as those who were running httpd 22-24 years ago) reinforced and emphasized the principle, when training and mentoring younger people, that being a good citizen with your network traffic is the responsible and ethical thing to do.
- jameshart 4y agoNobody’s going to argue that this is an example of a well-behaved bot. I’m just astonished that there are still people who have the time to notice and lament such behavior, and devote time to handcrafting .htaccess rules, rather than just having an aggressive set of heuristics and accepting that some crap will still get through. The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet.
- LinuxBender 4y agoI would not bother with looking at user-agent strings. Some bots can be spotted by oddly high packet TTL windows default is 128, Linux and Mac 64, missing TCP options, inability to do HTTP/2.0. There are iptables modules to block some of them by their missing MSS or odd MSS values, high packet TTL, or simply enforce HTTP/2.0 especially if this is just a blog. This will block all search engines except for Bing if that matters. Maybe someone here from Google can chime in as to when they plan to support HTTP/2.0 on their crawlers. Most bots are using really old libraries. Some of the newer bots are using farms of cellular cards but they can be spotted by the use of TCP Timestamps as that is enabled by default on Android. The downside of doing anything with this is that one would also block legit cell phones, so just give them a simple puzzle to solve. e.g. What is 1+4 or if you are certain they are a bot then ask How many roads must a man walk down?
- coredog64 4y agoWill I have access as a human if I answer anything but “42” to that last question?
- taskforcegemini 4y agoshouln't it be anything but 42, since that answer came from a machine?
- LinuxBender 4y agoWell not everyone knows that 42 is the answer. I guess you get full unfettered access.
- fsckboy 4y agothe answer to your question, my friend, is blowing in the wind.
- Aachen 4y agoI get spam from Microsoft IPs, using my own domain. Someone signed up my domain with MS. Of course I let MS know, figuring this must be some vuln in their system or at minimum a clear-cut case of abuse they can shut down. They sent back a template saying it's spoofed. As if I can't tell the difference between a From/Received header and the actual remote address that was delivering this garbage to my own mail server. All of this I already made clear in the original email. Next up is Microsoft straight out rejecting email from my new IP address. It's not on any blacklist, it never did anything wrong, it just hasn't sent email before. Their systems for getting off the blacklist are somewhere between obscure, untouched for decades, broken, and nonexistent, depending on which page you look at and who you ask. Employees, if you get that far after hours of searching, claim my IP isn't blocked (yeah, right... they've blocked the range and so the employees can't find the IP ban, as if I'm supposed/able to know what bounds their range ban has.) The furthest I've got is that they somehow disable delivery failure notifications for a few weeks to shut me up but email still doesn't arrive. I'm rather done dealing with Microsoft of late.
- walrus01 4y agoEasiest solution to this would be something functionally equivalent to opendkim and spamassassin to rank high in score whatever incoming SMTP traffic doesn't pass SPF and DKIM (obviously it doesn't pass SPF if it's claiming to come from user@domainname.com but is from an IP that is not the dns/rdns of what's in the DNS zonefile SPF record for the legit MX of domainname.com). And then additional scoring based on DKIM fail as well.
- Aachen 4y agoThat's an option. Haven't needed this before because I just block addresses that I receive spam on. I've always just accepted every email sent that wasn't addressed to a blacklisted address, so people can't really misconfigure anything, I'll always get their email without hoops to jump. So far, the spam has been very limited, but maybe one day I'll need to do this. Or block Microsoft IPs and tell them they can sign up for my IP reputation management system that sends out emails with broken confirmation links... I haven't yet decided.
- deleted 4y ago
- dhx 4y agoI recently made some contributions to https://github.com/alltheplaces/alltheplaces https://github.com/alltheplaces/alltheplaces which is a set of Scrapy spiders for extracting locations of primarily franchise business stores, for use with projects such as OpenStreetMap. After these contributions I am now in shock at how hostile a lot of businesses are with allowing people the privilege of accessing their websites to find store locations, store information or to purchase an item. Some examples of what I found you can't do at perhaps 20% of business websites: * Use the website near a country border where people often commute freely across borders on a daily or regular basis (example: Schengen region). * Plan a holiday figuring out when a shop closes for the day in another country. * Order something from an Internet connection in another country to arrive when you return home. * Look up product information whilst on a work trip overseas. * Access the website via a workplace proxy server that is mandated to be used and just so happens to be hosted in a data centre (the likes of AWS and Azure included) which is blocked simply because it's not a residential ISP. A more complex check would be the website checking the time between a request from Javascript code executing on client reaching the server matches the time the server thinks it should have taken (by ICMP PING or similar to the origin IP address). * Use the website simply because the geo IP database the website uses hasn't been updated to reflect block reassignments by the regional registry, and thinks that your French residential IP address is a defunct Latvian business IP address. * Find the business on a price comparison website. * Find the business on a search engine that isn't Google. * Access the website without first allowing obfuscated Javascript to execute (example: [1]). * Use the website if you had certain disabilities. * Access the website with IPv6 or using 464XLAT (shared origin IPv4 address with potentially a large pool of other users). The answer to me appears obvious: Store and product information is for the most part static and can be stored in static HTML and JSON/GeoJSON files that are rewritten on a regular cycle, or at least cached until their next update. JSON files can be advertised freely to the public so if anyone was trying to obtain information off the website, they could do so with a single request for a small file causing as minimal impact as possible. It's not difficult to create a website where 10,000 requests per second can be made to static data like product information or store locations. More advanced features such as stock availability would cause additional load, but again, 10,000 simple queries to a relational database is not a challenging problem. Of course, none of the anti-bot measures implemented by websites actually stop bots they're most seeking to block. There are services specifically designed for scraping websites that have 100,000's of residential IP addresses in their pools around the world (fixed and mobile ISPs), and it's trivial with software such as selenium-stealth to make each request look legitimate from a Javascript perspective as if the request was from a Chrome browser running on Windows 10 with a single 2160p screen, etc. If you force bots down the path of working around anti-bot measures, it's a never ending battle the website will ultimately lose because the website will end up blocking legitimate customers and have extremely poor performance that is worsened by forcing bots to make 10x the number of requests to the website just to look legitimate or pass client tests. [1] https://www.imperva.com/products/web-application-firewall-waf/ https://www.imperva.com/products/web-application-firewall-wa...
- varenc 4y agoBing provides a tool that allows a webmaster to verify if requests came from Bing. The "Verify Bingbot" tool is linked near the bottom of this article: https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0#:~:text=use%20the%20Verify%20Bingbot%20tool https://www.bing.com/webmasters/help/which-crawlers-does-bin... I suspect this is a real Bing web crawler that's possibly misconfigured. The "spoofed" user-agent might be an attempt to get a more genuine crawl of what a mobile browser sees. I at least know Google does this.
- eli 4y agoNah, that's almost certainly someone hosting something on Azure and not Bing.
- the_third_wave 4y agoOdd, I just noticed an ongoing rather intense scan for random paths on one of my servers using a number of IP addresses from within Microsoft-assigned space: "/var/www/domains/www.example.org/wp" "/var/www/domains/www.example.org/bc" "/var/www/domains/www.example.org/bk" "/var/www/domains/www.example.org/backup" "/var/www/domains/www.example.org/old" "/var/www/domains/www.example.org/new" "/var/www/domains/www.example.org/main" "/var/www/domains/www.example.org/home" "/var/www/domains/www.example.org/Telerik.Web.UI.WebResource.axd" "/var/www/domains/www.example.org/remote/fgt_lang" "/var/www/domains/www.example.org/wordpress" "/var/www/domains/www.example.org/Wordpress" "/var/www/domains/www.example.org/WORDPRESS" "/var/www/domains/www.example.org/WordPress" "/var/www/domains/www.example.org/wp" "/var/www/domains/www.example.org/Wp" "/var/www/domains/www.example.org/WP" "/var/www/domains/www.example.org/old" "/var/www/domains/www.example.org/Old" "/var/www/domains/www.example.org/OLD" "/var/www/domains/www.example.org/oldsite" "/var/www/domains/www.example.org/new" "/var/www/domains/www.example.org/New" "/var/www/domains/www.example.org/NEW" "/var/www/domains/www.example.org/wp-old" "/var/www/domains/www.example.org/2022" "/var/www/domains/www.example.org/2020" "/var/www/domains/www.example.org/2019" "/var/www/domains/www.example.org/2018" "/var/www/domains/www.example.org/backup" "/var/www/domains/www.example.org/test" "/var/www/domains/www.example.org/Test" "/var/www/domains/www.example.org/TEST" "/var/www/domains/www.example.org/demo" "/var/www/domains/www.example.org/bc" "/var/www/domains/www.example.org/www" "/var/www/domains/www.example.org/WWW" "/var/www/domains/www.example.org/Www" "/var/www/domains/www.example.org/2021" "/var/www/domains/www.example.org/main" "/var/www/domains/www.example.org/old-site" "/var/www/domains/www.example.org/bk" "/var/www/domains/www.example.org/Backup" "/var/www/domains/www.example.org/BACKUP" "/var/www/domains/www.example.org/SHOP" "/var/www/domains/www.example.org/Shop" "/var/www/domains/www.example.org/shop" "/var/www/domains/www.example.org/bak" "/var/www/domains/www.example.org/sitio" "/var/www/domains/www.example.org/bac" "/var/www/domains/www.example.org/sito" "/var/www/domains/www.example.org/site" "/var/www/domains/www.example.org/Site" "/var/www/domains/www.example.org/SITE" "/var/www/domains/www.example.org/blog" "/var/www/domains/www.example.org/BLOG" "/var/www/domains/www.example.org/Blog" ...etcetera, thousands of attempts using about 5 address from within Microsoft. I can only assume something has been pwned again.
- 4y ago
- zomglings 4y agoCould it be a bing crawler?
- masswerk 4y agoMaybe related, I had a few emails by alleged security researches, linking to standard tools, often even misinterpreting the results or maybe just too lazy to have a real look at them. Maybe, "security research" has just entered the "script-kiddy" age at a larger scale? The list provided in this comment [0] looks much like it. [0] https://news.ycombinator.com/item?id=34435222 https://news.ycombinator.com/item?id=34435222
- joelby37 4y agoI've been experiencing exactly the same thing over a number of sites for the last few days too, with exactly the same user agent in this post. The annoying part is that it is not really 'spidering' web sites, but rather continuously hammering a list of non-existent pages which appear to be from a years-old version of the site. Generating 404 responses puts a considerable load on WordPress sites and generates a lot of network traffic, but these have been relatively easy to block because of the predictable user agent and URI path prefixes. I'm thinking about blocking the Azure ASN completely, or developing something akin to Cloudflare's "are you a human?" interstitial page when requests come from cloud provider ASNs.
- wglass 4y agoYes - same here! Started around Jan 12. Large pool of IP addresses, hard to block. Occasional brief DOS impacts but mostly just annoying errors in my logs. (if too many crop up we get automatic alerts). What was really puzzling is that many of the URLs are old (e.g. request for details on hosted sites that no longer exist). I loaded up a 6 month old backup database and confirmed those accounts weren't present, so the source list of URLs must be older that. Really bizarre. After reading this article I looked and confirmed via spot checks they are from Microsoft IPs and Safari 15.1.
- anileated 4y agoIs this how MS are scraping the web to monetize it under OpenAI? If we don’t start batting an eye on this soon it’ll be the new normal that webmasters/content producers don’t really need readers or income.
- Nextgrid 4y ago> it’ll be the new normal that webmasters/content producers don’t really need readers or income. There's already way more content being produced out there (some of which is done altruistically without any expectation of income) than there is demand for it that making even more of it may not be as valuable as you think.
- anileated 4y agoThought experiment: imagine Google stopped sending users to articles you write and instead regurgitated and showed your content directly, with ads or for pay. Fast forward a few years and you have no more audience, no one has to know you exist, you ghost write for a magic black box that people ask questions and throw money at. When you want to impress a new friend and mention you published some interesting niche research (e.g., how to repair a thing) they skeptically nod because in their head no one does it anymore, the black box has all the answers. Will you still be feeling motivated to altruistically share content tomorrow?
- mjtechguy 4y agoThis is exactly what I thought when I read this.
- rezonant 4y agoFrom a linked blog post of the author's: > Given that there are websites that are willing (or reluctantly forced) to allow Google(bot) access but would rather like to block everyone else, more than a few of them are probably using User-Agent matching instead of anything more sophisticated. (https://utcc.utoronto.ca/~cks/space/blog/web/SpiderUserAgentMaybeTooClever https://utcc.utoronto.ca/~cks/space/blog/web/SpiderUserAgent...) Yeah, what a great idea to make all of our websites only crawlable by Google. Because competition in the search engine space is _so_ undesirable.
- shagie 4y agoThe sequence likely went "site is not crawalble by anything" (because there were so many badly behaved crawlers back in the day). Later the webmaster wanted it to show up in searches and so specifically opened up the robots.txt from "no body no way at all ever" to "ok, google can search these pages because they're not being evil" (this was back in the day). It wasn't so much a "ok, I want only google to search this and not anyone else", but rather at the time, there wasn't anyone else (that behaved). So now, you've got a website that is only searchable by google, and the webmaster retired and you've got this robots.txt that says you can't do it - and even some active defenses so that when non-google hits the page, they get tarpited. What are competing (reasonably well behaved) web spiders to do? How does Bing get in there to be able to compete now? Its not that Bing should or shouldn't be doing this - but there's a fairly reasonably way that we got to the point where are now without any nefarious behavior from the various participants.
- akerl_ 4y agoIf I'm worried about badly behaved crawlers... what's robots.txt doing for me?
- jakopo87 4y agoYears ago i added a url to not be crawled nor indexed to a web app I worked on to block bad crawlers, so I disallowed crawling through robots.txt and added a hidden link in the footer with the "noindex" attribute. Google, Bing, Archive.org, and people normally browsing were just fine while bad crawlers got a blank page.
- smeagull 4y agoAnd if the webspider is a browser?
- userbinator 4y agoMy first thoughts were "web page change detection service" and "someone misconfigured something".
- Jules8850 4y agoI've also seen this spider, and upon close examination it appears to be using a large dump of URLs, many years old, including non-public, non-guessable URLs that could only have been harvested from some sort of browser/plugin data breach.
- vinaypai 4y agoIt sounds like the bot was making about one request every 3 seconds. I'd hardly call that "aggressive".
- sqreept 4y agoIt is aggressive in what content is trying to access. It looks for security vulnerabilities and normal bots don't do that (with the notable exception of some security testing software). Also it's not spidering, somehow it knows very old URLs which are not even public which were probably obtained from a malicious browser extension.
- nunez 4y agoAzure functions?
- gwittel 4y agoIn a past job I’ve seen crappy crawlers from badly designed security applications do stuff like this. An an example one customer was using Trend CAS to scan all URLs in their inbound email. This causes big bursts of traffic on our systems. The crawls came from Azure and AWS. Forged UAs, repeat hits in the same URL, etc.
- Enderboi 4y agoThis bot is stupidly broken - we've had to block a whole swathe of Azure ranges on our load balancer. This particular 'bot' would start crawling in-line Javascript on a customer site, and then get stuck in an infinite loop requesting things like: /page/window.open /page/window.open/page/window.open /page/window.open/page/window.open/page/window.open We opened an abuse report with Microsoft last week, but haven't really heard anything back. Over 500 IP addresses involved hitting dozens of our hosted sites, le sigh. Unfortunately, we do see enough legitimate requests with that UA that we had to resort to the IP blocking. A bunch of /16's and a few smaller ranges... it feels dirty :(
- aorth 4y agoI'm sick of it too. In 2023-01 alone I have had 9,000 different IPs from Microsoft's ASN8075 crawling one of my sites with these "normal looking" user agents. Poring over the logs to see why your server is on fire takes non-trivial amounts of time. If I didn't have a ton of other stuff to do I'd say it was kinda fun, but I'm freaking fed up. Just yesterday I put all their networks into an nginx geo map: geo $limit_bots_ip { # requests with an empty key are not evaluated by limit_req # see: http://nginx.org/en/docs/http/ngx_http_limit_req_module.html default ''; 157.55.39.0/24 'bot'; 207.46.13.0/24 'bot'; 40.77.167.0/24 'bot'; 13.66.139.0/24 'bot'; ... } Any request from these networks gets classified as a bot, which is then used as the key for a rate limit limit_req_zone $limit_bots_ip zone=badbots_ip:1m rate=1r/m; It's incredible the amount of resources these companies have. I'm just one guy trying to keep a few dozen web servers up.
- gggggg5 4y agoWouldn't it be easier to fix whatever issue is causing your server to be on fire than spending non-trivial amounts of time poring over logs?
- adammarples 4y agoGood idea, debug it without looking at the logs, to save time
- gggggg5 4y agoIt's one thing if your server is on fire because of a bot crawling your websites at 1000rps, another if it's on fire because of a bot crawling it at less than 10rps.
- taskforcegemini 4y agothink of it as a ddos. how do you fix a ddos in your application?
- 4y ago
- enlyth 4y agoWe got DDoSed by Microsoft once, since the marketing department sent out an email with a link to one of our low traffic services which generally handles maybe a few hundred requests a day, and tens of thousands of requests from Microsoft's email crawlers came in a very short time span and rendered our service unusable.