3 ms·

Google tries to crawl pages that might exist

2 points by parham 13y ago

The Google bot has recently started randomly crawling pages, specially for profile and news pages e.g. /profile/<random_name> or /news/<random_id> Neither the profile or news path prefix are used in my site's urls. Some examples random names were “BobHope” and “suitcase_murphy” these aren't included or related to the site in question at all. I'm not complaining this is just an observation, has this happened to anyone else?

benologist 13y ago
The chance of either of those URLs being present on a site would have to be miniscule. Most likely they followed some outdated link somewhere.
- parham 13y ago
  Very unlikely as it's a development server (I should've mentioned) I doubt anyone linked back.
  - benologist 13y ago
    Here's two simple ways links could have happened: - someone having the domain before you - script kiddies machine-generating links to pages on random domains so Google finds exploitable targets for them Google is not randomly looking for pages - that's brute-forcing a site map and it's utterly, impossibly inefficient even to supplement what they find crawling around.
throwaway420 13y ago
Possibly they're checking random URLs to see if a site is erroneously redirecting what should be 404s to random content pages?
- andrewcooke 13y ago
  if you were going to write that code, would you use a url like that or a long, random string that you know will not exist? and why wouldn't you add at the end "this will not exist don't worry"?
ancarda 13y ago
I downloaded all my logs from a public-facing development server. Hitting anything returns "403 Forbidden" except for a few domains which should not be crawled (google does it anyway). Most traffic is usual (hit / and /robots.txt then leave). All I found (in 41 rotated logs) was: access.log.12:66.249.73.132 - - [29/Oct/2013:01:26:15 +0100] "GET /?ac=2 HTTP/1.1" 403 135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" access.log.12:66.249.73.132 - - [29/Oct/2013:03:24:12 +0100] "GET /?tag=lazy HTTP/1.1" 403 135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" access.log.3:66.249.66.57 - - [06/Nov/2013:12:12:42 +0100] "GET /?ac=2&slt=8&slr=1&lpt=1 HTTP/1.1" 403 135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" access.log.4:66.249.75.57 - - [05/Nov/2013:19:14:19 +0100] "GET /?cat=1 HTTP/1.1" 403 135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" access.log.5:66.249.66.109 - - [04/Nov/2013:15:54:01 +0100] "GET /?ac=2&slt=8&slr=1&lpt=1 HTTP/1.1" 403 135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" access.log.5:66.249.75.18 - - [05/Nov/2013:03:08:27 +0100] "GET /?ac=2&slt=8&slr=1&lpt=1 HTTP/1.1" 403 135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" I have no idea what "?ac", "?cat" or "?tag" are suppose to do. Nothing on the server responds to GET params (I use url rewrites and POST only) so I don't think I ever made a link (not even accidentally). I found nothing for "/profile" or "/news"
- parham 13y ago
  My server is also a public-facing development server. I think they might be trying to find pages that aren't linked to directly in a hope to find more unlined pages... just a theory
- andrewcooke 13y ago
  i've found an example of a similar url - http://stackoverflow.com/questions/19674788/redirect-url-having-query-string-to-404-page-using-htaccess http://stackoverflow.com/questions/19674788/redirect-url-hav... - so i am wondering if you have an ip that previously belonged to someone using whatever software that is (some kind of index or pagination i assume) and which had links with a numeric ip address....
deleted 13y ago
[deleted]

2 points by parham 13y ago

benologist 13y ago

The chance of either of those URLs being present on a site would have to be miniscule. Most likely they followed some outdated link somewhere.

parham 13y ago
Very unlikely as it's a development server (I should've mentioned) I doubt anyone linked back.
- benologist 13y ago
  Here's two simple ways links could have happened: - someone having the domain before you - script kiddies machine-generating links to pages on random domains so Google finds exploitable targets for them Google is not randomly looking for pages - that's brute-forcing a site map and it's utterly, impossibly inefficient even to supplement what they find crawling around.

throwaway420 13y ago

Possibly they're checking random URLs to see if a site is erroneously redirecting what should be 404s to random content pages?

andrewcooke 13y ago
if you were going to write that code, would you use a url like that or a long, random string that you know will not exist? and why wouldn't you add at the end "this will not exist don't worry"?

ancarda 13y ago

I downloaded all my logs from a public-facing development server. Hitting anything returns "403 Forbidden" except for a few domains which should not be crawled (google does it anyway). Most traffic is usual (hit / and /robots.txt then leave). All I found (in 41 rotated logs) was: access.log.12:66.249.73.132 - - [29/Oct/2013:01:26:15 +0100] "GET /?ac=2 HTTP/1.1" 403 135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" access.log.12:66.249.73.132 - - [29/Oct/2013:03:24:12 +0100] "GET /?tag=lazy HTTP/1.1" 403 135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" access.log.3:66.249.66.57 - - [06/Nov/2013:12:12:42 +0100] "GET /?ac=2&slt=8&slr=1&lpt=1 HTTP/1.1" 403 135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" access.log.4:66.249.75.57 - - [05/Nov/2013:19:14:19 +0100] "GET /?cat=1 HTTP/1.1" 403 135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" access.log.5:66.249.66.109 - - [04/Nov/2013:15:54:01 +0100] "GET /?ac=2&slt=8&slr=1&lpt=1 HTTP/1.1" 403 135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" access.log.5:66.249.75.18 - - [05/Nov/2013:03:08:27 +0100] "GET /?ac=2&slt=8&slr=1&lpt=1 HTTP/1.1" 403 135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" I have no idea what "?ac", "?cat" or "?tag" are suppose to do. Nothing on the server responds to GET params (I use url rewrites and POST only) so I don't think I ever made a link (not even accidentally). I found nothing for "/profile" or "/news"

parham 13y ago
My server is also a public-facing development server. I think they might be trying to find pages that aren't linked to directly in a hope to find more unlined pages... just a theory
andrewcooke 13y ago
i've found an example of a similar url - http://stackoverflow.com/questions/19674788/redirect-url-having-query-string-to-404-page-using-htaccess http://stackoverflow.com/questions/19674788/redirect-url-hav... - so i am wondering if you have an ip that previously belonged to someone using whatever software that is (some kind of index or pagination i assume) and which had links with a numeric ip address....

deleted 13y ago

[deleted]