8 ms·

I sent you an email about this. (A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance becau

by pg 15y ago

I sent you an email about this. (A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory.)

tibbon 15y ago
Could you use some sort of sitemap or other way to provide the data to Google that isn't so damaging to site performance? Or in Google Webmaster tools turn down the rate of crawling? Just realized that this could be a problem for lots of sites, and I'm curious as to what the best solution is, since not everyone has Matt Cutts reading their site and helping out.
- Matt_Cutts 15y ago
  We do have a self-service tool in webmaster tools for people who prefer to be crawled slower.
  - arctangent 15y ago
    Do you support/respect the "crawl-delay" directive? http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...
    - storborg 15y ago
      Nope. As mentioned above, apparently Google thinks people will "shoot themselves in the foot" with the crawl-delay directive, while they won't with Google's special interface (which requires registering and logging in).
      - wanorris 15y ago
        I can't imagine that they are just guessing abut this. I'm sure someone tried implementing it and was horrified at the actual results before they gave up on it.
      - einhverfr 15y ago
        What more can you expect from the World's Largest Adware Company? Seriously, one thing about Google is that they seem to really like ensuring people are logged on, preferably at all times. Fortunately recent changes to Google Apps (promoting apps user accounts to full Google accounts) has made this more complex on my side and probably degraded the level of actionable info they can get out of it.
  - Thieum22 15y ago
    or faster : http://news.ycombinator.com/item?id=2382728 http://news.ycombinator.com/item?id=2382728
pierrefar 15y ago
Hi Paul, A site can be crawled from any number of Googlebot IP addresses, and so blocking all except one doesn't help in throttling crawling. If you verify the site in Webmaster Tools, we have a tool you can use to set a slower crawl rate for Googlebot, regardless of which specific IP address ends up crawling the site. Let me know if you need more help. Edit Detailed instructions to set a custom crawl rate: 1. Verify the site in Webmaster Tools. 2. On the site's dashboard, the left hand side menu has an entry called Site Settings. Expand that and choose the Settings submenu. 3. The page there has a crawl rate setting (last one). It defaults to " Let Google determine my crawl rate (recommended)". Select "Set custom crawl rate" instead. 4. That opens up a form and choose his desired crawl rate in crawls per second. If there is a specific problem with Googlebot, you can reach the team as follows: 1. To the right hand side of the Crawl Rate setting is a link called "Learn More". Click that to open a yellow box. 2. In the box is a link called Report a problem with Googlebot which will take you to form you can fill out with full details. Thanks! Pierre
- aw3c2 15y ago
  I would like to set that crawl rate but do not see why I must register at Google to do so. Why can't Google support the Crawl-Delay directive in robots.txt for this?
  - Matt_Cutts 15y ago
    My plane is about to take off, but very briefly: people sometimes shoot themselves in the foot and get it way, way wrong. Like "crawl a single page from my website every five years" wrong. Crawl-Delay is (in my opinion) not the best measure. We tend to talk about "hostload," which is the inverse: the number of simultaneous connections that are allowed.
    - anonfoobar1 15y ago
      Another great way to shoot yourself in the foot (and getting it way way wrong) is to block all Googlebot IPs except for one.
    - 15y ago
Matt_Cutts 15y ago
Gotcha--thanks, Paul. I'm about to get on a plane, but we'll get this figured out where we're not sending as much hostload toward HN.
- ashishgandhi 15y ago
  Does that mean at Google you can manually set the system to treat different sites differently? I don't mean the ranking but other aspects - like you guys blacklisted some domains which produce low quality content in wholesale. (I don't know if the algorithm was tweaked to detect and filter such sources or if it was a manual thing.)
  - Kiall 15y ago
    No, It sounds like @pg has blocked google bot via a firewall, So as far as google's automated systems are concerned, hacker news was shutdown and thus removed from the index. I believe Matt is simply offering help/advice on figuring out the best solution for HN.
  - seabee 15y ago
    Does that mean at Google you can manually set the system to treat different sites differently? Webmaster Tools has a crawl rate slider which operate on a site-by-site basis, and that's existed for quite a while now. If you're asking if they can manually boost a site's ranking, hopefully that isn't what's being suggested.
- wgx 15y ago
  If only we could all get our Google woes fixed in such a manner.
  - metaprinter 15y ago
    seriously. i can't get a hospital to show up in google maps... no human for me to talk to. HN is number 4 instead of 1 google page one, they're right on it.
wheels 15y ago
Have you considered putting a caching reverse proxy in front of the arc app to keep the backend from having to render all of the old pages? It seems like the only dynamic element of old articles is the "$x days ago" bit and that'd be pretty easy to turn into something static by instead just putting in timestamps in the actual HTML and using Javascript to transform them into how many hours / days ago they were. Then the crawlers would just be pulling out cached, pre-rendered HTML. There's an example of doing such with nginx here: http://serverfault.com/questions/30705/how-to-set-up-nginx-as-a-caching-reverse-proxy http://serverfault.com/questions/30705/how-to-set-up-nginx-a... With that you'd just have to send out the HTTP header from the arc app saying that current articles expire immediately, and old ones don't.
- pg 15y ago
  I believe Rtm has already set one up.
  - wheels 15y ago
    The conspicuous lack of a "Server:" header inclines me to believe that that's probably not the case (most web servers set one indicating the server software and version). Here are the headers that HN sends out from an old post (20 days ago): HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Cache-Control: private Connection: close Cache-Control: max-age=0
    - bascule 15y ago
      My favorite part of HN's headers: the lines are separated by naked LFs instead of CRLF, in violation of the HTTP spec
      - divtxt 15y ago
        This is common violation that everyone accepts. It's definitely done by 'bad' clients - not sure how often servers send bare LF. (I used to telnet to port 80 for testing, and type GET / HTTP/1.0 <enter> <enter>, and that should be LF on Linux & Mac)
jedberg 15y ago
reddit had the same problem. We set up a separate server just for the google crawler with it's own copy of the database, so that the queries for old pages didn't slow down everyone else.

by pg 15y ago

tibbon 15y ago

Could you use some sort of sitemap or other way to provide the data to Google that isn't so damaging to site performance? Or in Google Webmaster tools turn down the rate of crawling? Just realized that this could be a problem for lots of sites, and I'm curious as to what the best solution is, since not everyone has Matt Cutts reading their site and helping out.

Matt_Cutts 15y ago
We do have a self-service tool in webmaster tools for people who prefer to be crawled slower.
- arctangent 15y ago
  Do you support/respect the "crawl-delay" directive? http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...
  - storborg 15y ago
    Nope. As mentioned above, apparently Google thinks people will "shoot themselves in the foot" with the crawl-delay directive, while they won't with Google's special interface (which requires registering and logging in).
    - wanorris 15y ago
      I can't imagine that they are just guessing abut this. I'm sure someone tried implementing it and was horrified at the actual results before they gave up on it.
    - einhverfr 15y ago
      What more can you expect from the World's Largest Adware Company? Seriously, one thing about Google is that they seem to really like ensuring people are logged on, preferably at all times. Fortunately recent changes to Google Apps (promoting apps user accounts to full Google accounts) has made this more complex on my side and probably degraded the level of actionable info they can get out of it.
- Thieum22 15y ago
  or faster : http://news.ycombinator.com/item?id=2382728 http://news.ycombinator.com/item?id=2382728

pierrefar 15y ago

Hi Paul, A site can be crawled from any number of Googlebot IP addresses, and so blocking all except one doesn't help in throttling crawling. If you verify the site in Webmaster Tools, we have a tool you can use to set a slower crawl rate for Googlebot, regardless of which specific IP address ends up crawling the site. Let me know if you need more help. Edit Detailed instructions to set a custom crawl rate: 1. Verify the site in Webmaster Tools. 2. On the site's dashboard, the left hand side menu has an entry called Site Settings. Expand that and choose the Settings submenu. 3. The page there has a crawl rate setting (last one). It defaults to " Let Google determine my crawl rate (recommended)". Select "Set custom crawl rate" instead. 4. That opens up a form and choose his desired crawl rate in crawls per second. If there is a specific problem with Googlebot, you can reach the team as follows: 1. To the right hand side of the Crawl Rate setting is a link called "Learn More". Click that to open a yellow box. 2. In the box is a link called Report a problem with Googlebot which will take you to form you can fill out with full details. Thanks! Pierre

aw3c2 15y ago
I would like to set that crawl rate but do not see why I must register at Google to do so. Why can't Google support the Crawl-Delay directive in robots.txt for this?
- Matt_Cutts 15y ago
  My plane is about to take off, but very briefly: people sometimes shoot themselves in the foot and get it way, way wrong. Like "crawl a single page from my website every five years" wrong. Crawl-Delay is (in my opinion) not the best measure. We tend to talk about "hostload," which is the inverse: the number of simultaneous connections that are allowed.
  - anonfoobar1 15y ago
    Another great way to shoot yourself in the foot (and getting it way way wrong) is to block all Googlebot IPs except for one.
  - 15y ago

Matt_Cutts 15y ago

Gotcha--thanks, Paul. I'm about to get on a plane, but we'll get this figured out where we're not sending as much hostload toward HN.

ashishgandhi 15y ago
Does that mean at Google you can manually set the system to treat different sites differently? I don't mean the ranking but other aspects - like you guys blacklisted some domains which produce low quality content in wholesale. (I don't know if the algorithm was tweaked to detect and filter such sources or if it was a manual thing.)
- Kiall 15y ago
  No, It sounds like @pg has blocked google bot via a firewall, So as far as google's automated systems are concerned, hacker news was shutdown and thus removed from the index. I believe Matt is simply offering help/advice on figuring out the best solution for HN.
- seabee 15y ago
  Does that mean at Google you can manually set the system to treat different sites differently? Webmaster Tools has a crawl rate slider which operate on a site-by-site basis, and that's existed for quite a while now. If you're asking if they can manually boost a site's ranking, hopefully that isn't what's being suggested.
wgx 15y ago
If only we could all get our Google woes fixed in such a manner.
- metaprinter 15y ago
  seriously. i can't get a hospital to show up in google maps... no human for me to talk to. HN is number 4 instead of 1 google page one, they're right on it.

wheels 15y ago

Have you considered putting a caching reverse proxy in front of the arc app to keep the backend from having to render all of the old pages? It seems like the only dynamic element of old articles is the "$x days ago" bit and that'd be pretty easy to turn into something static by instead just putting in timestamps in the actual HTML and using Javascript to transform them into how many hours / days ago they were. Then the crawlers would just be pulling out cached, pre-rendered HTML. There's an example of doing such with nginx here: http://serverfault.com/questions/30705/how-to-set-up-nginx-as-a-caching-reverse-proxy http://serverfault.com/questions/30705/how-to-set-up-nginx-a... With that you'd just have to send out the HTTP header from the arc app saying that current articles expire immediately, and old ones don't.

pg 15y ago
I believe Rtm has already set one up.
- wheels 15y ago
  The conspicuous lack of a "Server:" header inclines me to believe that that's probably not the case (most web servers set one indicating the server software and version). Here are the headers that HN sends out from an old post (20 days ago): HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Cache-Control: private Connection: close Cache-Control: max-age=0
  - bascule 15y ago
    My favorite part of HN's headers: the lines are separated by naked LFs instead of CRLF, in violation of the HTTP spec
    - divtxt 15y ago
      This is common violation that everyone accepts. It's definitely done by 'bad' clients - not sure how often servers send bare LF. (I used to telnet to port 80 for testing, and type GET / HTTP/1.0 <enter> <enter>, and that should be LF on Linux & Mac)

jedberg 15y ago

reddit had the same problem. We set up a separate server just for the google crawler with it's own copy of the database, so that the queries for old pages didn't slow down everyone else.