5 ms·

How long will it take for Google to index this?

44 points by byrneseyeview 18y ago

byrneseyeview 18y ago
Google has gotten much faster at finding new pages lately. It would be interesting to know how much faster.
earle 18y ago
30 to 90 minutes for this site
byrneseyeview 18y ago
Indexed. 25 minutes? In their excerpt, they note that it was posted 21 minutes ago. So that seems to be their score.
dkokelley 18y ago
20 minutes ish. Very good. Keep in mind that as they update the 21 "minutes ago" will get farther, so it may have been even sooner.
merrick33 18y ago
Historically the indexing factor is tied to the amount and authority of a domains inbound links. Not all sites will get the results yielded on this domain.
martian 18y ago
In the spirit of Douglas Hofstadter... nice recursion.
- TrevorJ 18y ago
  If I imbeded the search result in a third webpage and then googled that page...woo, makes my head hurt.
  - ivankirigin 18y ago
    This isn't even that odd. That's just plain recursion. Each notice of indexing should alter the query text based upon the comments at the time.
- mojuba 18y ago
  http://www.google.ie/search?q=%22This+sentence+can+not+be+found+in+Google%22 http://www.google.ie/search?q=%22This+sentence+can+not+be+fo...
pierrefar 18y ago
I've seen some blog posts indexed within 2 minutes. A lot of people have reported sub-5-minute indexing. 20-40min is good and the most common time frame in my experience and from what I understand from other people. Ultra-fast indexing depends on posting just before the crawler hits the index page. A lot of luck is involved to catch the crawler like that.
- tb 18y ago
  If someone uses Google Reader to subscribe to your RSS feed, then your new blog posts are likely to get indexed as soon as Google Reader sees your new post show up in RSS.
- pyroman 18y ago
  Some blog software, like wordpress, will ping services when you post something new. http://codex.wordpress.org/Update_Services http://codex.wordpress.org/Update_Services
jakewolf 18y ago
I couldn't resist checking: http://search.yahoo.com/search?p=%22How+long+will+it+take+for+Google+to+index+this%3F%22&fr=yfp-t-501&toggle=1&cop=mss&ei=UTF-8 http://search.yahoo.com/search?p=%22How+long+will+it+take+fo... http://search.live.com/results.aspx?q=%22How+long+will+it+take+for+Google+to+index+this%3F%22&go=&form=QBHP http://search.live.com/results.aspx?q=%22How+long+will+it+ta... http://www.alexa.com/search?q=%22How%20long%20will%20it%20take%20for%20Google%20to%20index%20this?%22 http://www.alexa.com/search?q=%22How%20long%20will%20it%20ta... All duds for now.
- andreyf 18y ago
  PG talked about blocking all non-Google crawlers in an attempt to cut down on traffic ... not sure if he ever did or not
  - haasted 18y ago
    There doesn't seem to be anything about it in HN's robot.txt. Of course, it may have been implemented in other ways.
    - andrewparker 18y ago
      What are those "x" URLs blocked in the robots.txt for HN?
TrevorJ 18y ago
Google is impressive, that's for sure.
jsteele 18y ago
No offence Tarquin, but what's the purpose of this test? It cannot possibly tell us anything about our own sites, or the sites we work on. Like many of us, I've already tracked the time it takes for Google to index the changes to my own sites - about 14 hours. I work on sites that the Google robots hit only once every 2 weeks - less content, and fewer updates, though some of them earn 10 times what my more 'frequent' sites earn. Good O, Hacker News! Which means sweet F all to me, all of my customers, and pretty much everybody else on the planet. By the way, how long IS a piece off string?
- byrneseyeview 18y ago
  Tarquin? The purpose was to measure a phenomenon I'd previously observed -- that it's getting harder to find answers on Google, because people post their questions on well-trafficked sites, and within a few minutes, those sites rocket to the top for a well-crafted search query trying to answer the question. "By the way, how long IS a piece off string?" I don't know, but strlen(piece) probably does.
lyime 18y ago
Thats pretty impressive. I have noticed my small blog posts get indexed quickly on google.
henning 18y ago
for those of us who have never run high authority websites -- if google indexes you that frequently, how much bandwidth do they use up?
Tichy 18y ago
I recently thought about this: what if the whole internet worked more like Twitter? People are laughing about Twitter and think it is an easy problem, but not even Google manages instant updates! The equivalent to Twitter for Google would be to index all news on the internet immediately. In that sense, I think we can cut Twitter some slack...
- smanek 18y ago
  Except that you're off by (at least) several dozen orders of magnitude. Twitter, in fact, is several thousand times smaller than even GMail, which has ~100,000,000 registered users to Twitter's ~1,000,000 (http://twitterfacts.blogspot.com/2007/10/twitter-number-of-users.html http://twitterfacts.blogspot.com/2007/10/twitter-number-of-u... - and that's assuming twitter's user base has over doubled in the last six months while GMails has remained stagnant for the last year or so). Google manages to route large messages between GMail users instantly, without breaking a sweat, and with 100% uptime (so does Microsoft, Yahoo, and every other major email provider). They have no trouble dealing with users who send or receive thousands of emails a day, while twitter claims this is some sort of pathological, borderline-impossible case to deal with. And GMail is relatively trivial compared to actually indexing the internet. The last decent estimate I saw (2005 - http://www.cs.uiowa.edu/~asignori/web-size/ http://www.cs.uiowa.edu/~asignori/web-size/) suggested the indexable WWW is ~ 12 Billion pages - and god knows how much that has increased in the last four years. Seriously, make sure you're in the right ballpark - hell, in the right galaxy - before talking out off your ass. Twitter's problem should be easy today - it was solved decades ago. The fact that Twitter can't scale is nothing more than a testament to their incompetence.
  - Tichy 18y ago
    Well I was really thinking about twitter replacements, as I said - what if the whole net would work like twitter? I update my blog and I want it to be known instantly. Google does not provide it yet. But I have also thought about mail, perhaps it would be easiest to replace Twitter with "micromailing", and use existing software solutions. Difference is of course the "only receive messages from people I subscribe to" thing, but that should only make it easier. Edit: also, google mail doesn't do polling, so maybe it is very different after all. It's a completely different architecture. Maybe Twitter would work better with an inbox-per-user model, but I wouldn't bet on it. Another difference: most emails don't go to dozens of recipients at the same time. Tweets often have several hundred recipients.
bluelu 18y ago
5 weeks ago, I had a question regarding aws and created a topic in the amazon forum. Afterwards I searched google again, with different keywords and I found my own topic. That was scary. Only a few minutes passed (at most 5)
khafra 18y ago
I posted "Quisquis custodiet ipsos custodes" on reddit, then immediately decided I wanted to check my spelling--I was already the top result in Google.
- khafra 18y ago
  ...and now this one's on top, with the former post in second place. My mind boggles at the possibilities for recursion.

44 points by byrneseyeview 18y ago

byrneseyeview 18y ago

Google has gotten much faster at finding new pages lately. It would be interesting to know how much faster.

earle 18y ago

30 to 90 minutes for this site

byrneseyeview 18y ago

Indexed. 25 minutes? In their excerpt, they note that it was posted 21 minutes ago. So that seems to be their score.

dkokelley 18y ago

20 minutes ish. Very good. Keep in mind that as they update the 21 "minutes ago" will get farther, so it may have been even sooner.

merrick33 18y ago

Historically the indexing factor is tied to the amount and authority of a domains inbound links. Not all sites will get the results yielded on this domain.

martian 18y ago

In the spirit of Douglas Hofstadter... nice recursion.

TrevorJ 18y ago
If I imbeded the search result in a third webpage and then googled that page...woo, makes my head hurt.
- ivankirigin 18y ago
  This isn't even that odd. That's just plain recursion. Each notice of indexing should alter the query text based upon the comments at the time.
mojuba 18y ago
http://www.google.ie/search?q=%22This+sentence+can+not+be+found+in+Google%22 http://www.google.ie/search?q=%22This+sentence+can+not+be+fo...

pierrefar 18y ago

I've seen some blog posts indexed within 2 minutes. A lot of people have reported sub-5-minute indexing. 20-40min is good and the most common time frame in my experience and from what I understand from other people. Ultra-fast indexing depends on posting just before the crawler hits the index page. A lot of luck is involved to catch the crawler like that.

tb 18y ago
If someone uses Google Reader to subscribe to your RSS feed, then your new blog posts are likely to get indexed as soon as Google Reader sees your new post show up in RSS.
pyroman 18y ago
Some blog software, like wordpress, will ping services when you post something new. http://codex.wordpress.org/Update_Services http://codex.wordpress.org/Update_Services

jakewolf 18y ago

I couldn't resist checking: http://search.yahoo.com/search?p=%22How+long+will+it+take+for+Google+to+index+this%3F%22&fr=yfp-t-501&toggle=1&cop=mss&ei=UTF-8 http://search.yahoo.com/search?p=%22How+long+will+it+take+fo... http://search.live.com/results.aspx?q=%22How+long+will+it+take+for+Google+to+index+this%3F%22&go=&form=QBHP http://search.live.com/results.aspx?q=%22How+long+will+it+ta... http://www.alexa.com/search?q=%22How%20long%20will%20it%20take%20for%20Google%20to%20index%20this?%22 http://www.alexa.com/search?q=%22How%20long%20will%20it%20ta... All duds for now.

andreyf 18y ago
PG talked about blocking all non-Google crawlers in an attempt to cut down on traffic ... not sure if he ever did or not
- haasted 18y ago
  There doesn't seem to be anything about it in HN's robot.txt. Of course, it may have been implemented in other ways.
  - andrewparker 18y ago
    What are those "x" URLs blocked in the robots.txt for HN?

TrevorJ 18y ago

Google is impressive, that's for sure.

jsteele 18y ago

No offence Tarquin, but what's the purpose of this test? It cannot possibly tell us anything about our own sites, or the sites we work on. Like many of us, I've already tracked the time it takes for Google to index the changes to my own sites - about 14 hours. I work on sites that the Google robots hit only once every 2 weeks - less content, and fewer updates, though some of them earn 10 times what my more 'frequent' sites earn. Good O, Hacker News! Which means sweet F all to me, all of my customers, and pretty much everybody else on the planet. By the way, how long IS a piece off string?

byrneseyeview 18y ago
Tarquin? The purpose was to measure a phenomenon I'd previously observed -- that it's getting harder to find answers on Google, because people post their questions on well-trafficked sites, and within a few minutes, those sites rocket to the top for a well-crafted search query trying to answer the question. "By the way, how long IS a piece off string?" I don't know, but strlen(piece) probably does.

lyime 18y ago

Thats pretty impressive. I have noticed my small blog posts get indexed quickly on google.

henning 18y ago

for those of us who have never run high authority websites -- if google indexes you that frequently, how much bandwidth do they use up?

Tichy 18y ago

I recently thought about this: what if the whole internet worked more like Twitter? People are laughing about Twitter and think it is an easy problem, but not even Google manages instant updates! The equivalent to Twitter for Google would be to index all news on the internet immediately. In that sense, I think we can cut Twitter some slack...

smanek 18y ago
Except that you're off by (at least) several dozen orders of magnitude. Twitter, in fact, is several thousand times smaller than even GMail, which has ~100,000,000 registered users to Twitter's ~1,000,000 (http://twitterfacts.blogspot.com/2007/10/twitter-number-of-users.html http://twitterfacts.blogspot.com/2007/10/twitter-number-of-u... - and that's assuming twitter's user base has over doubled in the last six months while GMails has remained stagnant for the last year or so). Google manages to route large messages between GMail users instantly, without breaking a sweat, and with 100% uptime (so does Microsoft, Yahoo, and every other major email provider). They have no trouble dealing with users who send or receive thousands of emails a day, while twitter claims this is some sort of pathological, borderline-impossible case to deal with. And GMail is relatively trivial compared to actually indexing the internet. The last decent estimate I saw (2005 - http://www.cs.uiowa.edu/~asignori/web-size/ http://www.cs.uiowa.edu/~asignori/web-size/) suggested the indexable WWW is ~ 12 Billion pages - and god knows how much that has increased in the last four years. Seriously, make sure you're in the right ballpark - hell, in the right galaxy - before talking out off your ass. Twitter's problem should be easy today - it was solved decades ago. The fact that Twitter can't scale is nothing more than a testament to their incompetence.
- Tichy 18y ago
  Well I was really thinking about twitter replacements, as I said - what if the whole net would work like twitter? I update my blog and I want it to be known instantly. Google does not provide it yet. But I have also thought about mail, perhaps it would be easiest to replace Twitter with "micromailing", and use existing software solutions. Difference is of course the "only receive messages from people I subscribe to" thing, but that should only make it easier. Edit: also, google mail doesn't do polling, so maybe it is very different after all. It's a completely different architecture. Maybe Twitter would work better with an inbox-per-user model, but I wouldn't bet on it. Another difference: most emails don't go to dozens of recipients at the same time. Tweets often have several hundred recipients.

bluelu 18y ago

5 weeks ago, I had a question regarding aws and created a topic in the amazon forum. Afterwards I searched google again, with different keywords and I found my own topic. That was scary. Only a few minutes passed (at most 5)

khafra 18y ago

I posted "Quisquis custodiet ipsos custodes" on reddit, then immediately decided I wanted to check my spelling--I was already the top result in Google.

khafra 18y ago
...and now this one's on top, with the former post in second place. My mind boggles at the possibilities for recursion.