5 ms·

Google bots follow SQL-injected URLs from third party sites

75 points by cubictwo 13y ago

rshm 13y ago
I think the cause here is google following the injected urls from third party sites.
- axblount 13y ago
  That's exactly what the article says.
mrtksn 13y ago
So, this is basically using the Google bot as a proxy. I don't see what Google can do about it, or does Google have to do anything at all. Just secure your website as if the attacker is not going to use the Google bot as a proxy, because well, nobody can guarantee that the attacker will use the Google bot. Google bot is nothing more than a computer sending HTTP GET request to your server, so if your site is vulnerable to HTTP GET requests why you write like there is something special about Google bot?
- djjaxe 13y ago
  Well it would be nice if google could check the url it was visiting and if there is any sqli in it to not send the request (though this could potentially slow their crawling...)
  - mrtksn 13y ago
    Why? Do you want anybody but Google to hack your site? Why would Google spend resources on unnecessary detection of SQL injection(which probably will not be perfect anyway and may break legit requests) when anybody can hack your website? I just can't justify why do you expect Google to spend resources on not running bogus HTTP GET request when anybody can run those? What is different about being hacked by Google bot and being hacked by an unsuspected user who clicks on a bogus link that was put on the same page where Google found that link to your server? Just doesn't makes sense.
    - Zancarius 13y ago
      > Just doesn't makes sense. Not only that, but it seems to me that it'd be a more efficient use of resources to spend the time hardening your own site rather than lobbying Google to implement something that only mitigates one potential attack vector. Even then, it just seems stupid because I'm sure there are valid GET query strings that might have select, insert, update, delete, or some permutation thereof in them. It seems to me that it's just a punt on poor programming habits...
  - maddddddddddddd 13y ago
    what about "' OR 1=1;--" are you going to ban the word OR?
  - geofft 13y ago
    How is Google supposed to check for what is/isn't "sqli"? The proposal reminds me of Yahoo! Mail's old "medireview" problem, where it filtered emails containing the string "eval": http://en.wikipedia.org/wiki/Medireview#Blocked_emails http://en.wikipedia.org/wiki/Medireview#Blocked_emails Even if you look for somewhat complete SQL strings, if I want to host http://try-sql-in-your-browser.io/?sql=select+foo+from+bar http://try-sql-in-your-browser.io/?sql=select+foo+from+bar, I'd want Google to index it.
    - mikeash 13y ago
      For a hilarious variant trying to protect against "attacks" on humans, use your favorite search engine to search for "buttbuttination".
      - hobs 13y ago
        Awesome, I had seen a few things in the wild like this, but this ended up in some fun reading. Thanks!
    - AYBABTME 13y ago
      Well for one, we could make a standard query value that would say whether or not, the URL is evil. http://try-sql-in-your-browser.io/?evil=true http://try-sql-in-your-browser.io/?evil=true It works well for IPv4.
tylerkahn 13y ago
Spoofing IP headers (specifically the source address) is possible.
- djjaxe 13y ago
  But the only point in this is to take down a site you won't be able to get into useful information back from this request as the request's response will be heading back to the spoofed address... (though if you are using for ddos it is pointless to get the data back...)
  - MichaelGG 13y ago
    You can't make an HTTP request with a spoofed IP without being in position on the network to do a MiTM. The TCP handshake won't complete, so you can't send the HTTP request.
- jamescun 13y ago
  This would only work on connectionless protocols, such as UDP. UDP offers no message reliability and relies on the source address to respond. This is how DNS amplification attacks work, as the main protocol of DNS is UDP. Whereas HTTP goes over TCP which is a connection-oriented protocol. TCP offers message integrity by going back and forth between client and server multiple times to verify the message was retrieved successfully. Without a valid source address the 3-Way TCP Handshake used to establish the connection cannot succeed.
  - hnha 13y ago
    thank you, I always wondered but never asked.
    - AYBABTME 13y ago
      Coursera had a great Computer Network course. I think you can still register and download the videos: https://class.coursera.org/comnetworks-002/ https://class.coursera.org/comnetworks-002/
lazyjones 13y ago
This could also be Google probing for vulnerable software on your website. They're doing it for various Typo3 versions (I know, because we got some false positives in the past - Google Webmaster tools warned us about it, we saw the requests in logs, our fault for replying with Status 200 for some invalid URLs where we just showed our main page), they might be doing it for other software where the only way to check for vulnerabilities is to try an actual (harmless) SQL inject.
iLoch 13y ago
I think the real problem here is that the author is assuming the Google Bot somehow gets special treatment. Why are you concerned that the bot is sending SQL injection attacks? The same thing could be done by a regular user. Are you defending against regular users? If so, then you're defending against Google bots too. Now if the problem is that your firewall is blocking Google bots, well then you're going to have to let them through if you want Google traffic. It's unfeasible for Google to fix this on their end. Your application should be secure from SQL injection without the need for a firewall anyway, so it shouldn't be a technical problem having to make an exception for Google bots.
- bryanlarsen 13y ago
  The problem is not that the Google Bot gets special treatment, but that attackers get special treatment. It's common to blacklist or slow responses to the IP's of potential attackers. A slow response to a false positive is usually acceptable as long as you have a low rate of false positives, but it can seriously affect your Google ranking if you are slowing their bots...
ChuckMcM 13y ago
This is sort of sad, now I have to go dig through crawler code to see what we (Blekko) do when we crawl a site. We do avoid known 'traps' (which are often inadvertent) that result in loops but this is a bit more nefarious. On a related note, if you would like to be the person looking through the crawler code and designing a defense for this, and are willing to work in the bay area. Send me an email :-)
- pyre 13y ago
  Sounds fun, but I'm not moving to the bay area. :P
- jmpe 13y ago
  How would one differentiate between a legitimate query and one that traps? Seriously, suppose I link to a search result page for MySQL [on some site] for info on such a query, wouldn't it look the same? Not looking for work atm, so my advice: this is not your problem.
  - ChuckMcM 13y ago
    A typical crawler trap is a link to a page which adds some sort of "info" the url, so lets say you have a URL like http://not-a-real-host.com/some_page/ And that page has a link for comments which takes you to http://not-a-real-host.com/some_page/?show_comments=1 And the designer didn't neuter the show comments link and if you follow it, it takes you to: http://not-a-real-host.com/some_page/?show_comments=1&show_comments=1 Ad infiniteum. That is a bug but the same sort loop where the cgi arg is going from ?page=1, ?page=2 is valid. Basically folks who love working on web crawlers are people who like figuring out puzzles like this. The number of interesting things web sites can do that make crawling them not work is unlimited I believe. We have lots of regular checks (things like how many unique documents have I pulled from a host, or how many pages have the same MD5 hash, etc.) Designing defenses is part of the fun.
    - ugexe 13y ago
      He is asking how you plan to differentiate when it is ok for the query to be in the url (perhaps it is checked and validated with a regex on the server so it cant be abused) or when its actually capable of doing something malicious the maintainer is not aware of.
vezzy-fnord 13y ago
Reminds me of an anecdote I read on Stack Overflow where a web developer inserted an unprotected database killswitch parameter in his code which ended up being triggered by a Googlebot. This is a pretty interesting attack vector, though. The old Google bomb being used for more nefarious purposes than spoofing page rankings.
badman_ting 13y ago
But why would the attacker want this, how can it benefit them? I don't understand how a third party could make Googlebot hitting a SQLi benefit themselves. I get the cleverness of hiding behind Googlebot, I just don't see how you can "steer" the attack, so to speak, to your own benefit.
- bazzargh 13y ago
  There's a long history of being able to search google for sites with vulnerabilities or infections if you know what to look for. With a black hat on, alter a busy forum's code to generate a link spreading the infection via google to every domain that commenters mention; a tiny fraction of those will be infected, those will show up later in a search. That's a bit weak, and might not pick out decent targets. However, visit the forum as a user, mention some specific sites, and hey presto - google infects them for you, while you disappear in the crowd. Seems a bit airport thriller though. More likely the leaked sqli and google's use of it were accidental.
asm89 13y ago
Wondering how effective this is for getting a competitor website to block for example the google crawler. Maybe that's the motive.

75 points by cubictwo 13y ago

rshm 13y ago

I think the cause here is google following the injected urls from third party sites.

axblount 13y ago
That's exactly what the article says.

mrtksn 13y ago

So, this is basically using the Google bot as a proxy. I don't see what Google can do about it, or does Google have to do anything at all. Just secure your website as if the attacker is not going to use the Google bot as a proxy, because well, nobody can guarantee that the attacker will use the Google bot. Google bot is nothing more than a computer sending HTTP GET request to your server, so if your site is vulnerable to HTTP GET requests why you write like there is something special about Google bot?

djjaxe 13y ago
Well it would be nice if google could check the url it was visiting and if there is any sqli in it to not send the request (though this could potentially slow their crawling...)
- mrtksn 13y ago
  Why? Do you want anybody but Google to hack your site? Why would Google spend resources on unnecessary detection of SQL injection(which probably will not be perfect anyway and may break legit requests) when anybody can hack your website? I just can't justify why do you expect Google to spend resources on not running bogus HTTP GET request when anybody can run those? What is different about being hacked by Google bot and being hacked by an unsuspected user who clicks on a bogus link that was put on the same page where Google found that link to your server? Just doesn't makes sense.
  - Zancarius 13y ago
    > Just doesn't makes sense. Not only that, but it seems to me that it'd be a more efficient use of resources to spend the time hardening your own site rather than lobbying Google to implement something that only mitigates one potential attack vector. Even then, it just seems stupid because I'm sure there are valid GET query strings that might have select, insert, update, delete, or some permutation thereof in them. It seems to me that it's just a punt on poor programming habits...
- maddddddddddddd 13y ago
  what about "' OR 1=1;--" are you going to ban the word OR?
- geofft 13y ago
  How is Google supposed to check for what is/isn't "sqli"? The proposal reminds me of Yahoo! Mail's old "medireview" problem, where it filtered emails containing the string "eval": http://en.wikipedia.org/wiki/Medireview#Blocked_emails http://en.wikipedia.org/wiki/Medireview#Blocked_emails Even if you look for somewhat complete SQL strings, if I want to host http://try-sql-in-your-browser.io/?sql=select+foo+from+bar http://try-sql-in-your-browser.io/?sql=select+foo+from+bar, I'd want Google to index it.
  - mikeash 13y ago
    For a hilarious variant trying to protect against "attacks" on humans, use your favorite search engine to search for "buttbuttination".
    - hobs 13y ago
      Awesome, I had seen a few things in the wild like this, but this ended up in some fun reading. Thanks!
  - AYBABTME 13y ago
    Well for one, we could make a standard query value that would say whether or not, the URL is evil. http://try-sql-in-your-browser.io/?evil=true http://try-sql-in-your-browser.io/?evil=true It works well for IPv4.

tylerkahn 13y ago

Spoofing IP headers (specifically the source address) is possible.

djjaxe 13y ago
But the only point in this is to take down a site you won't be able to get into useful information back from this request as the request's response will be heading back to the spoofed address... (though if you are using for ddos it is pointless to get the data back...)
- MichaelGG 13y ago
  You can't make an HTTP request with a spoofed IP without being in position on the network to do a MiTM. The TCP handshake won't complete, so you can't send the HTTP request.
jamescun 13y ago
This would only work on connectionless protocols, such as UDP. UDP offers no message reliability and relies on the source address to respond. This is how DNS amplification attacks work, as the main protocol of DNS is UDP. Whereas HTTP goes over TCP which is a connection-oriented protocol. TCP offers message integrity by going back and forth between client and server multiple times to verify the message was retrieved successfully. Without a valid source address the 3-Way TCP Handshake used to establish the connection cannot succeed.
- hnha 13y ago
  thank you, I always wondered but never asked.
  - AYBABTME 13y ago
    Coursera had a great Computer Network course. I think you can still register and download the videos: https://class.coursera.org/comnetworks-002/ https://class.coursera.org/comnetworks-002/

lazyjones 13y ago

This could also be Google probing for vulnerable software on your website. They're doing it for various Typo3 versions (I know, because we got some false positives in the past - Google Webmaster tools warned us about it, we saw the requests in logs, our fault for replying with Status 200 for some invalid URLs where we just showed our main page), they might be doing it for other software where the only way to check for vulnerabilities is to try an actual (harmless) SQL inject.

iLoch 13y ago

I think the real problem here is that the author is assuming the Google Bot somehow gets special treatment. Why are you concerned that the bot is sending SQL injection attacks? The same thing could be done by a regular user. Are you defending against regular users? If so, then you're defending against Google bots too. Now if the problem is that your firewall is blocking Google bots, well then you're going to have to let them through if you want Google traffic. It's unfeasible for Google to fix this on their end. Your application should be secure from SQL injection without the need for a firewall anyway, so it shouldn't be a technical problem having to make an exception for Google bots.

bryanlarsen 13y ago
The problem is not that the Google Bot gets special treatment, but that attackers get special treatment. It's common to blacklist or slow responses to the IP's of potential attackers. A slow response to a false positive is usually acceptable as long as you have a low rate of false positives, but it can seriously affect your Google ranking if you are slowing their bots...

ChuckMcM 13y ago

This is sort of sad, now I have to go dig through crawler code to see what we (Blekko) do when we crawl a site. We do avoid known 'traps' (which are often inadvertent) that result in loops but this is a bit more nefarious. On a related note, if you would like to be the person looking through the crawler code and designing a defense for this, and are willing to work in the bay area. Send me an email :-)

pyre 13y ago
Sounds fun, but I'm not moving to the bay area. :P
jmpe 13y ago
How would one differentiate between a legitimate query and one that traps? Seriously, suppose I link to a search result page for MySQL [on some site] for info on such a query, wouldn't it look the same? Not looking for work atm, so my advice: this is not your problem.
- ChuckMcM 13y ago
  A typical crawler trap is a link to a page which adds some sort of "info" the url, so lets say you have a URL like http://not-a-real-host.com/some_page/ And that page has a link for comments which takes you to http://not-a-real-host.com/some_page/?show_comments=1 And the designer didn't neuter the show comments link and if you follow it, it takes you to: http://not-a-real-host.com/some_page/?show_comments=1&show_comments=1 Ad infiniteum. That is a bug but the same sort loop where the cgi arg is going from ?page=1, ?page=2 is valid. Basically folks who love working on web crawlers are people who like figuring out puzzles like this. The number of interesting things web sites can do that make crawling them not work is unlimited I believe. We have lots of regular checks (things like how many unique documents have I pulled from a host, or how many pages have the same MD5 hash, etc.) Designing defenses is part of the fun.
  - ugexe 13y ago
    He is asking how you plan to differentiate when it is ok for the query to be in the url (perhaps it is checked and validated with a regex on the server so it cant be abused) or when its actually capable of doing something malicious the maintainer is not aware of.

vezzy-fnord 13y ago

Reminds me of an anecdote I read on Stack Overflow where a web developer inserted an unprotected database killswitch parameter in his code which ended up being triggered by a Googlebot. This is a pretty interesting attack vector, though. The old Google bomb being used for more nefarious purposes than spoofing page rankings.

badman_ting 13y ago

But why would the attacker want this, how can it benefit them? I don't understand how a third party could make Googlebot hitting a SQLi benefit themselves. I get the cleverness of hiding behind Googlebot, I just don't see how you can "steer" the attack, so to speak, to your own benefit.

bazzargh 13y ago
There's a long history of being able to search google for sites with vulnerabilities or infections if you know what to look for. With a black hat on, alter a busy forum's code to generate a link spreading the infection via google to every domain that commenters mention; a tiny fraction of those will be infected, those will show up later in a search. That's a bit weak, and might not pick out decent targets. However, visit the forum as a user, mention some specific sites, and hey presto - google infects them for you, while you disappear in the crowd. Seems a bit airport thriller though. More likely the leaked sqli and google's use of it were accidental.

asm89 13y ago

Wondering how effective this is for getting a competitor website to block for example the google crawler. Maybe that's the motive.