8 ms·
Our new search index: Caffeine
- deleted 16y ago[deleted]
- tptacek 16y agoWow is that ever a terrible infographic.
- rbranson 16y agoOne of the worst I've ever seen. It wasn't even worth it if it took 3 minutes to draw it up.
- defen 16y agoI couldn't tell if it was a joke... "magic internet stuff here"
- tptacek 16y agoYou think for a moment that at least maybe the different content types are allocated different colors, but, no, they're all dark blue. One of two dark blue orbits in Caffeine.
- tjmc 16y agoPerhaps they were the best blues of the 41 shades... (cf. http://stopdesign.com/archive/2009/03/20/goodbye-google.html http://stopdesign.com/archive/2009/03/20/goodbye-google.html)
- MikeCapone 16y agoConsidering they had that page with pigeons to explain how pagerank works, I would assume it's not supposed to be very serious.
- inerte 16y agoThe pigeon ranking page was an April 1st joke. It's exactly funny in one day of the year. Saying how you'll rank 85% of web searches is a little bit more serious business.
- pjscott 16y agoThis shouldn't affect the ranking; updated data will just get to the ranking algorithm faster, is all.
- deleted 16y ago[deleted]
- deleted 16y ago[deleted]
- jedsmith 16y agoI'd have to disagree. It shows us that Google has adopted the experimental "Bohr" model of database indexing.
- resdirector 16y agoI'm not sure if I get this...is this just a pithy one-liner, or am I missing something deeper here?
- arethuza 16y agohttp://en.wikipedia.org/wiki/Bohr_model http://en.wikipedia.org/wiki/Bohr_model The right hand side of the diagram.... So you were right about the "pithy one liner".
- resdirector 16y agoThen this comment should be flagged. It (a) reduces the intelligence of the discussions on HN (b) pushes more worthy comments down below the fold, (c) encourages similar pithy comments, and (d) makes those who would otherwise make intelligent comments less likely to comment. This is the kind of one-liner that perhaps Jon Stewart would make...and while there is nothing wrong with The Daily Show, I thought that we, the HN community, are collectively better than that. Sorry for the rant, but I'm just sick of wading through pithy one-liners on HN.
- jedsmith 16y agoI felt a small amount of humor -- indeed, what a small attempt it was, I didn't consider it that amusing -- was welcome in the thread discussing the shortcomings of the accompanying infographic. Unless you'd like to have a serious discussion about the shortcomings of the accompanying infographic? I'm not sure there's much intelligence to be had in a discussion responding to "Wow is that ever a terrible infographic". Can we discuss its poor choice of color? Its lack of visual appeal? The obvious selection of a Home Depot paint swatch to represent Google's old index, and what that means for their database technology? Not really. Not in response to "Wow is that ever a terrible infographic". I'd be hesitant to flag everything that you don't initially understand. Cut HN some slack, we're intelligent people and we enjoy humor as much as the next man. I certainly appreciate when someone makes me chuckle in the comments, and I'd loathe HN if that went away by your hand. Given something like http://news.ycombinator.com/item?id=1403672 http://news.ycombinator.com/item?id=1403672 is it HN that has to change, or the person in the chair? I realize your time may be valuable, but you don't have to read everything and get offended by its presence -- including my comment.
- bshep 16y agoI agree. The depiction of old index looks better, that to me means failure to communicate on their part.
- imagii 16y agoWithout reading the article (and ignoring the captions), you'd think the left side was better. Ordered and color-coded, as opposed to flying around.
- tjpick 16y agoagreed, I immediately thought the old index is structured and layered, the new index is a big ball of mud...
- orionlogic 16y agovery much like itunes Genius bar logo with electrons changed with google sidebar icons. Caffeine is not new and has been talked in SEO community from a quite sometime. I wonder why they make it public now. To silence iPhone 4 news? No no,just coincidence...
- dandelany 16y agoWell, search is their primary service; their web indexing algorithms are sort of their Mother of All Secret Sauces. You wouldn't expect Coke to publish the exact recipe.
- ErrantX 16y agoPerhaps you have to "build search engines for a living like [them]" to understand it.
- dskhatri 16y agoNice to see they reference an iPod instead of a Nexus One. Somehow I feel that if it was a Microsoft announcement, the company would reference Zunes.
- batiudrami 16y agoNexus One total internal storage: 512MB 'the largest iPod': 160GB The Nexus One is a phone, not a device with a large storage capacity. If Google made one of those, I'm sure it would be referenced. edit: the Nexus One box also includes a 4GB micro SD card, but that'd be comparing flash memory with harddrives, and is hardly an intuitive explanation.
- jodrellblank 16y agoThat is an enormous amount of data. I wonder how much is junk?
- mkramlich 16y agoYou're new to the Internetz, right? :)
- jodrellblank 16y agoI meant junk like search engine cheating forum and usenet posts and sites registered to repeat stock phrases and content of spammy aggregator sites, rather than making judgements against lolcats and breakfast time tweets.
- gojomo 16y ago100 petabytes of information (100 million GiB) feeding their index is more than I would have expected.
- moe 16y agoOn the other hand it doesn't seem that much when you consider todays storage density. You can fit around 0.5 PB into one rack nowadays. 200 racks then sounds a bit less impressive than 100 Petabytes. However, that ofcourse doesn't account for redundancy, nor for doing anything useful with such a pile of data. Both of which impose some interesting challenges at that scale.
- blasdel 16y agoYou think Google's mainline storage of their index is on disk? It is to laugh. I don't think that's been true since around the time they stopped building server shelves out of lego or cardboard. They've been keeping the full text of the web in RAM (for the snippets), with indexes, several times over. With independent live siblings in multiple datacenters.
- moe 16y agoNotice the last sentence? I merely tried to put the figure into perspective.
- houseabsolute 16y agoWell, the serving part of their index is in memory (or largely so) as has been publicly announced. But are you really suggesting they might do the same for indexing? Let's suppose for a second that they can get 8GB sticks of RAM for 1/3 the retail price of around $600, and for the sake of the back of the envelope calculation, let's round it up to ten GB. So let's call it $20/GB of RAM. Four gigabyte sticks might seem cheaper until you consider they'd have to double the number of machines to hold them. Now they mentioned that the size of the index is 100,000,000 GB, which means that they would have spent $2 billion on this RAM alone, not to mention all the other components that are required to house it. Especially considering how much their capital expenditure has fallen lately (http://www.datacenterknowledge.com/archives/2010/01/22/google-capex-edges-higher-in-q4/ http://www.datacenterknowledge.com/archives/2010/01/22/googl...), it doesn't seem very likely that they could afford to spend even $2 billion on this. And the assumptions I made about the price of RAM are pretty generous, considering they'd have had to have been acquiring this for some time, and therefore that the RAM would have been more expensive previously.
- swah 16y agoJava or C++
- coderdude 16y agoIf there could be an answer to what you said, it would probably be "both, and Python."
- shadowsun7 16y agoGoogle uses only four languages in their codebase: C++, Java, Python and Javascript. Yegge has covered this pretty extensively in the past. Go Google him.
- chintan 16y agohuh? 40 miles of iPods. +5 for an intuitive analogy. Priceless!
- iworkforthem 16y agoold index: problem: parts of the index were refreshed at a faster rate than others; the main layer would update every couple of weeks and to refresh a layer of the old index, Google would analyze the entire web first. impact: there was a significant delay between when Google found a page and made it available to its users. new index: solution: caffeine, Google's new index will analyze the web in small portions and update its search index on a continuous basis, globally. As Google find new pages, or new information on existing pages, Google can add these straight to the index. improvement: user can find fresher information than ever before—no matter when or where it was published. recent changes in Google search result page suggested that the searched results appears to be categorized further to the different structured data. This could be inline with the rolling out Caffeine.
- Corrado 16y agoHmmm... come to think about it I have been getting quite a lot of results from CodeWeblog.com in my search results. Unfortunately, that site seems to be a link farm and its so useless that I reported it to the Google spam page. Maybe its appearance in the top sites was a result of the Caffeine changes.
- biafra 16y agoAh! Its not only me who is annoyed by these codeweblog.com results. Is there a way to remove domains form google results as a preference? I know I can always add "-site:codeweblog.com" But can I save this preference for future search queries?
- joewein 16y agoI reported them as well, but like your report, it doesn't seem to have been acted on. See my blog post at: http://www.joewein.net/blog/2010/07/01/codeweblog-com-a-pile-of-garbage-indexed-by-google/ http://www.joewein.net/blog/2010/07/01/codeweblog-com-a-pile...
- samaparicio 16y agoWhy is every measure of storage always "demystified" with a paper stack analogy? I find it funny that when the paper stack would not provide the right scale, the used a stack of storage devices...
- jsiarto 16y agoI'm waiting for Google to open up an API or dashboard on top of Caffeine to compete with some of the monitoring tools like Radian6 and Jive. They could own that space and it's growing like crazy right now. Plus, we don't need any more companies coming out with "social web" search engines.
- grrr45 16y agoPosted by Carrie Grimes, Software Engineer Dr. Grimes currently leads a research & technical team in Search Infrastructure at Google and is actively trying to figure out what criteria make a search engine index "good," "fast," and "comprehensive." Yeah, right. "Software Engineer" ? bullshit.
- SkyMarshal 16y agoShe's more than a mere Software Engineer, she's a scientist. AB in Archeology/Anthropology from Harvard, then a PhD in Statistics from Stanford. Very interesting background to bring to this field. Call it real-time statistical anthropology.
- enneff 16y ago"Software Engineer" is a generic title at Google; the actual role can vary wildly. Carrie is an impressively talented engineer and statistician.
- deleted 16y ago[deleted]
- spuz 16y agoWhat I dislike about Google's current algorithm is that when I search for a common term such as "javadoc simpledateformat", the API documentation for Java 1.4.2 (as opposed to 1.6) comes up as apparently, that is what most documents on the web link to. I hope this new index will allow more recent and more up-to-date documents to get to the top of the results list.
- noss 16y agoI tend to use "java6 simpledateformat" as my query for that.
- Jermey128 16y agoIn Firefox I have the following quick-search to fix that (and make searching the API a little faster): http://www.google.com/search?q=java+6+%s&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a http://www.google.com/search?q=java+6+%s&ie=utf-8&oe... This should also work in Chrome.
- spuz 16y agoYep, I've done exactly that for the Java 6 docs in Chrome and FF. The general point of how do you keep the latest version of a set of documents the most relevant still applies however. I hope Google can fix this :)
- suraj 16y agoSimilarly searching for Apache directives (e.g. http://www.google.com/search?q=rewriterule http://www.google.com/search?q=rewriterule ) returns result for apache 1.3 docs. You have to add 2.2 while searching for apache info.
- cookiecaper 16y agoI had the same annoyance searching for PostgreSQL stuff -- the latest docs are almost never accessible from the Google page, just docs 2 years old or more.
- leej 16y agothey could be using in-memory compression. they could be storing hot parts of index in memory. they could be using both.
- dalore 16y agoAren't we all using Duck Duck Go anyway?
- c00p3r 16y agoSeems like the target group of this blog post are teenagers, not even students. Some street slang would look well near the size measurement in ipods. =)
- keegangrayson 16y agocode or it didn't happen...