7 ms·

Google should open source what actually matters: their search ranking algorithm

57 points by ivankirigin 17y ago

lacker 17y ago
(I used to work in the Search Quality division at Google.) Security by obscurity is a bad analogy here. The problem is that Google's algorithms are not mathematically perfect, in the sense of cryptography. Instead, they merely statistically tend to find good results. You can't find bugs just by looking at source code. You have to statistically analyze the effects of any proposed changes.
showerst 17y ago
The notion that being 'committed' to open source means that a company should give away everything of value is ludicrous. Google came up with great software that makes millions of peoples' lives easier, and they have every right to keep it secret and profit off of it. The zealous idea that open source and trade secrets cannot coexist is detrimental to the open source movement as a whole, in my opinion.
- rriepe 17y ago
  Well put. Almost any good idea taken to its extreme is a bad idea.
  - electromagnetic 17y ago
    High taxation eventually reaches communism, if there's benefits like healthcare 'refunded' to the citizen and if there isn't then that ends up being a kleptocracy. Low taxation eventually reaches anarchy as the government is no longer in control. Increasing birth rates enables economies to grow without becoming stagnant and old, placing undue demand due to pensions. An exceptionally high birth rate creates food production problems, housing problems, and generally the 'sardine can' effect. Low birthrates create a stagnant civilization that will eventually reach a tipping point when taxes get so high to pay for pensions that the younger generations will protest. Market freedom to the extreme leads to monopolies taking advantage, 'gentlemen's agreements' between companies to price fix and many other problems that rules and regulations try to control. Equally a controlled market develops problems in and of itself as it's unable to respond naturally to market fluctuations, and quickly becomes unruly and unstable, and as many of the countries who have held long-term controlled markets are testaments to, they develop massive black-market economies to deal with these. The zealous burning of fossil fuels, aside from global warming concerns, can expel vast quantities of radioactive materials, sulfurous emissions and a long list of things we've accepted in western society as being more harmful than smoking to the wide populations health, but rules and regulations on air quality increasingly get more lax (at least they have been where I live). With zero fossil fuel production, however, our civilization would collapse (at least with an immediate stop to production), and I'm not just talking the halting of transportation vehicles, I'm talking the stop in production of the advanced polymers used in virtually all aspects of our lives. The majority of people wouldn't have clothes on their back . . . my wardrobe would certainly be a lot cheaper if my clothes were all genuine cotton and wool. Oh and farewell cheap elasticated sweatpants. Every good idea taken to its extreme is a bad idea and will create monumental problems in its wake. Electric vehicles are amazing, however a pure-electric is a ridiculous prospect in winter when the capacity hits below 60% of room-temperature, suddenly your vehicle can't get you to work and back on a single charge. Zero integration of hybrids is, as everyone knows, damaging to the environment but also an extreme waste of entropy. A 1 tonne vehicle at 100km/h is a massive store of energy, and allowing that to be wasted is criminal, the amount of energy humanity has wasted on braking at highway speeds would likely run the entire human civilization for over a year.
- mattmanser 17y ago
  I think you're missing the point, the article is pretty much a direct response to: http://googleblog.blogspot.com/2009/12/meaning-of-open.html http://googleblog.blogspot.com/2009/12/meaning-of-open.html (submitted here: http://news.ycombinator.com/item?id=1008908 http://news.ycombinator.com/item?id=1008908) In it a Google open source zealot rants wildly about openess being ultimate, apart from search algorithms which should be closed. It also fails to mention all the times open has failed (communism being one of the more spectacular failures of open). A cynic might surmise what they actually mean is content should be open so they can profit from their closed search algorithm.
  - jedc 17y ago
    It was written by Jonathan Rosenberg, who's a Senior Vice President for Product Management, and on the top management committee at Google. That doesn't quite meet my personal description of an "open source zealot", though he's clearly quite passionate about it.
  - RyanMcGreal 17y ago
    > communism being one of the more spectacular failures of open Wait, what? Communism is a spectacular failure of centralized planning over the entire economy. It has nothing to do with open or closed in principle, and in every implementation it has been aggressively closed in terms of information flows and decision making.
    - jacquesm 17y ago
      > Communism is a spectacular failure of centralized planning over the entire economy. Wow, and here I was thinking it was a form of government. Communism in principle has absolutely nothing to do with the flow of information and/or decision making, and its implementations have only very little to do with its basic ideas. A true communistic government has never even been tried, we've only seen a 'new boss same as the old boss' approach to communism with one batch of fat cats replacing another.
      - mbreese 17y ago
- allenp 17y ago
  Additionally it may be the case that a large part of the value of the algorithm is created from its secrecy. If the algorithm was open it would be trivial to create sites with artificially high rankings which would reduce the value of the search results.
  - chollida1 17y ago
    > If the algorithm was open it would be trivial to create sites with artificially high rankings I don't necessarily think this is true. Sites that have high page rank, do so because other sites of high value link to them. Even if Google open sourced their search engine, spam sites would still have a hard time getting high quality inbound links. The reputable and high ranking sites wouldn't have any increased incentive to link to low quality spam sites.
    - DougWebb 17y ago
      It's not the ranking algorithm specifically that's a secret; it's the code that used to detect sites that work together to artificially raise page rank of another site by linking to it. The whole Page Rank algorithm is predicated on the idea that human website owners and contributors do a better job at assessing the value of a website and they express this by linking to the website. When links are created for other reasons they reduce the effectiveness of Page Rank, and when they are created specifically to bias Page Rank they do the most damage to the algorithm's results. Keeping the algorithm secret, and in particular the tweaks that eliminate links which aren't expressing "I judge this to be a good site", keep Page Rank useful.
    - EnderMB 17y ago
      Absolutely. I've worked in several contracted roles within online marketing and I've heard some awful advice from self-proclaimed SEO gurus about how to achieve a high rank for keywords on Google. A lot of what achiving high ranks is about is common sense and as Google improves their algorithm fewer tricks work. Anything past a simple competitive analysis, keyphrase analysis and a backlink analysis on you and your competitors is overkill. The quality of SEO analysts out there is so poor that I believe you could release the source code to Google's entire search engine and people would still get things wrong or believe in ways to get around what I assume is an extremely complex set of mathematically-oriented algorithms. It's a classic example of the blind following the blind, and those that typically have followers are those that are popular for performing SEO work on huge companies that would never actually need it (e.g. Coca Cola).
- kvs 17y ago
  Agreed. After reading the comments at cdixson.org by the author and reading openness article at Google, it seems like author is simply saying: "I agree they won't open it. But then they should stop talking sactimoniously about openness." which I agree with as well. Here is an interesting perspective: "And millions of websites will have to continue blindly relying on a small group of anonymous engineers in charge of the secret algorithm that determines their fate." Have you ever been delisted by Google? Is there a transparent process to get back on the list? Perhaps transparency is what Google needs the most at the moment.
  - jacquesm 17y ago
    > Perhaps transparency is what Google needs the most at the moment. Absolutely, and not only on the search engine part of it, Adwords/Adsense are extremely opaque and as a publisher continuously give you the feeling you're being screwed without any chance to prove that for lack of information.
    - mbreese 17y ago
      As a publisher, couldn't you do some sort of A/B testing with another ad provider? Something like 10% of page views using another set of ads? Or, maybe setup some kind of onclick tracking to verify ad clicks? I don't know what could be done as verification for the advertiser, but the publisher should be able to do something, right?
      - eli 17y ago
        No one else gets enough volume :/
    - zaphar 17y ago
      You say that now but if they provided that transparency then the amount of fraudulent activity would skyrocket. Knowing the details of the algorithms involved does nothing to help you as an advertiser or searcher. Googles extreme impartiality in search and ads is their greatest strength and open sourcing the algorithms involved would do nothing to increase that strength and contribute to a great deal less useful ads and search results on average.
- iamelgringo 17y ago
  The notion that being 'committed' to open source means that a company should give away everything of value is ludicrous. I disagree with the statement that Google would be giving away everything by opening up page rank. Google's "secret sauce" is hardly page rank alone. Google is really secretive about this stuff, but they arguably have the world's larges corpus of multilingual text in the form of their html index, coupled with the world's largest fault tolerant compute cluster, the largest repository of scanned text,the largest repository of digital video, and probably the largest repository of digital audio in the world. I'm sure that they also have the world's largest repository of people's surfing and searching habits, along with the one of the largest sets of analytics data. Even if page rank, map reduce, GFS, etc.. were open sourced I'm sure they'd still find a way to make billions off of the data that they have. But, I think the larger point that the author makes, is that unlike many giant technology companies (Microsoft, HP, Apple, etc..), Google has made billions off of using open source software. And, while they are a very large supporter of open source, what total percentage of their profits go back to OSS? What percentage of their code gets re-released as open source? These are good questions to ask, and I'd love to hear an open discussion about it.
buster 17y ago
opening this algorithm would lead to A LOT of new spammers and SEO-crap. Google, please keep it a secret, PLEASE!
mustpax 17y ago
The analogy to "security through obscurity" is flawed. Open security works when there's an obvious way a secure system should behave, say, make it infeasible to open a lock/decrypt data without the correct key. In Google’s, case the correct relevancy ordering of the web is a huge point of contention. So they cannot simply open it up hoping to close out all the holes in the open. It is also hard to distinguish acceptable behavior (linking) from unacceptable behavior (selling links) except in the aggregate through heuristics that assume ignorance on the part of the attacker. In this sense, Google’s system is a lot closer to an intrusion detection system than a lock or a cipher suite. You want to establish some ground rules but there’s no added security in declaring what you’re monitoring on your servers and how.
swombat 17y ago
That is a completely ridiculous point to make. Come on, be serious here. What would be the benefit to anyone except for spammers, if Google open-sourced their algorithms? It's a lose-lose option. Google loses some money, we lose a lot of search engine accuracy, and spammers gain new ways to screw around with search results. And does anyone gain from it? Not really. At this point, Google's supremacy is due just as much to their hardware know-how, to their market leader position, and to their excellent brand, as it is to their search algorithms. It won't help any competing search engines - just spammers.
- roc 17y ago
  The stated goal isn't to help other search engines; it's to get the community to help Google improve, for the good of all its users. The theory goes: if you let more people look at the algorithm, they can suggest better answers than Google alone, to thwart the spammers who are gaming the system either way. But I suspect the real goal has more to do with the deep suspicion that Google can or does do a significant bit of hand-tweaking. And these people want to force the algorithm into the open to either end or curtail that.
  - jacquesm 17y ago
    I'll believe that when I see open source achieve a conclusive victory against email spam. Spam:email = 90% or worse in search results it is closer to the opposite. I'm sure once the open source community actually manages to do this and prove the 'many eyes' doctrine applies to more than just security and bug fixes that many doors will open.
infynyxx 17y ago
Google's search ranking algorithm is their bread and butter so, I don't think they will ever open source it. But what always strikes me is that they have not yet open sourced their BigTable database systems even though, it's not their core revenue generating source. There are clones of BigTable in the form of Cassandra and HyperTable but it would be so much better if BigTable is open sourced.
bbsabelli 17y ago
tl;dr? Author confuses source code with revenue. Welcome to the less-than-free world.
jlees 17y ago
An interesting thing to think about even though the basic reaction is "wait, no, that's their industry secret/revenue basis/whatever". Google is so dominant right now that maybe it's going to get to the point where, in the interests of the open web, they get pressured and lobbied to release their secret sauce; after all, as swombat points out, it's not just the algorithm by a long chalk. It's especially interesting if you consider the origins of PageRank in academia, freely published and peer-reviewed, and still not that well understood by laypeople; would the open-sourced magic of Google even be understandable by that many folk? Those that do understand it could make a fortune as consultants; at the moment, shady SEO snake-oil salesmen are so prevalent and 99% of their advice is outdated or plain wrong, something that definitively calls them out on it could be helpful.
amix 17y ago
I would love if they open-sourced their infrastructure tools such as GFS, BigTable and Map-Reduce tools. Most of these tools are built using open-source technology such as Linux.
- stanleydrew 17y ago
  Apache's Hadoop (http://hadoop.apache.org/ http://hadoop.apache.org/) has open-sourced implementations of those tools.
  - iamelgringo 17y ago
    Yahoo was the company that opensourced hadoop, not google.
    - stanleydrew 17y ago
      Never said it was Google, just pointing out that open-source implementations exist.
  - mbreese 17y ago
    I'd be interested to see some benchmarks (unofficial-probably) between Hadoop and Google's C++ versions (at least, I assume Google's is C++). I've always wondered if there was any extra overhead in using Hadoop. I know Java is almost if not faster than C++ in some areas, and long running servers (like Hadoop) should benefit the most. But I wonder if there are some extra limitations Java places on things like BigTable.
stuff4ben 17y ago
I don't get it, is the author talking about PageRank? http://en.wikipedia.org/wiki/PageRank http://en.wikipedia.org/wiki/PageRank What more does he want?
- minsight 17y ago
  The algorithm used by Google today is light-years away from that described in the PageRank paper. It uses a form of PageRank, but it also uses many other parameters.
  - spf 17y ago
    If it's secret, then how can you be so sure? My impression is that PageRank, while initially extremely successful at filtering signal-from-noise, is very susceptible to manipulation. For example links farms which make unimportant pages seem important. As a result of such manipulation, Google started modifying their crawling, in order to reduce the amount of garbage going into PageRank (less garbage in, less garbage out). For example, they might look at the age and ownership in DNS records, and exclude domains that look fishy based on some heuristics. The spammers fought back with more tricks, and the resulting arms race has come to have an acronym: SEO. But the underlying idea of using an iterative approximation of the likelihood of encountering a page via a random walk of the web is preserved. (Some details here: http://www.miislita.com/information-retrieval-tutorial/matrix-tutorial-3-eigenvalues-eigenvectors.html http://www.miislita.com/information-retrieval-tutorial/matri...) If that's right (and I could well be wrong... would like to be convinced...), then I don't see it as a "light-years away" kind of issue, but rather a series of hacks to stay one step ahead of the spammers. I don't see any particular value in open sourcing those.
- mooism2 17y ago
  Google rankings are determined by much more than PageRank.
Mystalic 17y ago
Google is a business. Their job, legally defined, is to create value (aka profit) for its shareholders. A more open web has helped Google, but opening up its search algorithms, a cornerstone to Google's growth, is just ludicrous and bad business. Compared to a lot of other companies (Apple, Microsoft, nearly every other tech giant), they are an open company. It's already great that they realize there's value in an Open-source browser, and open-source OS, and multiple open platforms.
- tome 17y ago
  Their job, legally defined, is to create value (aka profit) for its shareholders. This is a meme I see a lot, but never with any evidence. That the board has a fiduciary duty to the shareholders I can understand. That it would have a legal obligation to create value, well, I think that would be hard to define, and I have never seen any evidence for the existence of this obligation.
  - deleted 17y ago
    [deleted]
  - mbreese 17y ago
    In the context of a publicly traded company this is the definition of fiduciary. The company has sold stock to investors with the sole purpose of increasing the value of the stock. Therefore, it is up to the board to make decisions that reflect this duty. You can argue about the timeline (short versus long-term growth), but you can't say that there isn't a legal obligation to attempt to increase value.
    - roc 17y ago
      > "The company has sold stock to investors with the sole purpose of increasing the value of the stock." I know they're not popular, or exciting like a stock value swing, but dividends still exist, right?
      - mbreese 17y ago
mlLK 17y ago
http://www.google.com/support/webmasters/ http://www.google.com/support/webmasters/ easily brings down this bogus claim. I honestly don't think much would change even if they did open-source their PageRank algorithm because the people willing and able enough to exploit it already do, and regardless of how much their PageRank increases Google will send your domain deep into an internet abyss and out of their index.
allenp 17y ago
I thought open source was about software, not about logic (business or otherwise) - am I missing something here?
hello0012356 17y ago
Welcome to: http://www.wowowbiz.com http://www.wowowbiz.com The website wholesale for many kinds of fashion shoes, like the nike,jordan,prada,adidas, also including the eans,shirts,bags,hat and the decorations. All the products are free shipping, and the the price is competitive, and also can accept online payment.,after the payment, can ship within short time. http://www.wowowbiz.com http://www.wowowbiz.com our price: jordan air max oakland raiders $34--39; Ed Hardy AF JUICY POLO Bikini $25; Christan Audigier BIKINI JACKET $25; gstar coogi evisu true jeans $35; coach chanel gucci LV handbags $36; coogi DG edhardy gucci t-shirts $18; CA edhardy vests.paul smith shoes $32; jordan dunk af1 max gucci shoes $37; EDhardy gucci ny New Era cap $16; coach okely Adidas CHANEL DG Sunglass $18; http://www.wowowbiz.com http://www.wowowbiz.com thank you for your visit Wish you a merry Christmas
hello12593 17y ago
Welcome to: http://www.wowowbiz.com http://www.wowowbiz.com The website wholesale for many kinds of fashion shoes, like the nike,jordan,prada,adidas, also including the eans,shirts,bags,hat and the decorations. All the products are free shipping, and the the price is competitive, and also can accept online payment.,after the payment, can ship within short time. http://www.wowowbiz.com http://www.wowowbiz.com our price: jordan air max oakland raiders $34--39; Ed Hardy AF JUICY POLO Bikini $25; Christan Audigier BIKINI JACKET $25; gstar coogi evisu true jeans $35; coach chanel gucci LV handbags $36; coogi DG edhardy gucci t-shirts $18; CA edhardy vests.paul smith shoes $32; jordan dunk af1 max gucci shoes $37; EDhardy gucci ny New Era cap $16; coach okely Adidas CHANEL DG Sunglass $18; http://www.wowowbiz.com http://www.wowowbiz.com thank you for your visit Wish you a merry Christmas
btilly 17y ago
I upvoted the story not because I agree with it, but because it explains a very successful open sourcing strategy that I wish more people understood. In general people are willing to spend a certain amount on a solution. Therefore if you can things that make the complement to what you make money on cheaper, that increases how much you can charge for your piece of the solution. Open sourcing is a way to make those pieces cheaper. Google does this. They give away tools and knowledge about how to make websites. They give away analytics and an A/B testing platform and educate people about how that makes it easier to produce good websites. This makes building good websites cheaper, and therefore increases how much people are willing to spend to advertise those websites. This isn't a new strategy. In the late 90s when Oracle and IBM got into open source, this was their strategy. Oracle wanted to get people to stop paying for Sun hardware so Oracle could negotiate higher prices. (I know more than a few people who were surprised to find that licenses for Oracle on Linux are higher than they are on Sun.) IBM was looking to reduce the price for complements to IBM websphere, and was looking for opportunities to avoid having customers pay the Microsoft tax. If you run a company and wish to strategically open source something, you should think in the same terms. What is the complement to what you are charging for, and how can you make that cheaper?
selven 17y ago
And what is wrong with commoditizing the complement? Google's working hard to make operating systems free and we'll probably see Microsoft working hard to open up search engines. End result: both operating systems and search engines will be open and the consumer benefits.
xcombinator 17y ago
Well, I can't understand this article, there is not "secret sauce". The google pagerank algorithm is open for all to see and understand. I myself made my own simple implementation. Everybody could do it. This man is not asking for code, is asking for DATA only google has. It's like an atheist saying to a catholic what he must do because he believes. When this man gives something of value to the world,like google did(I remember the Altavista-Ad-flickering in your eyes days) then he could demand to be given by others.
- eli 17y ago
  The documented PageRank algorithm doesn't have very much to do with how results are actually returned from Google today.
oujheush 17y ago
I am disturbed by the lack of faith in open source among the readers here. I mean, I understand some skepticism but this, to me, is pretty extreme IP-fetishism. It's perfectly understandable. Lots of money is made in the existing paradigm and so many of you have incentive to believe it's the most rational system. Besides, hard to imagine Google's doing something wrong, right? But the argument that "the only people this will help is spammers"? Really? Are the only people helped by open cryptographic protocols those who look to decipher traffic? "They have every right to keep it secret and profit off of it." Let's allow this for a moment. We've still got a false dichotomy. Making it public will not deprive them of their profit. First, incredible marketing coup. Does anyone else remember how much of Google's initial strength came from its rabid fans? This would do a lot of re-energize that base. Second, are there going to be new startups which can effectively compete with Google, even given its algorithm? How are they going to provide a better experience, or serve the volume of traffic, or index a larger amount of pages with the same algorithm with fewer resources? Third, who says they have to authorize competitors to use it? This is I think one of the more interesting points. They could release the algorithm but license it only for non-commercial use for anyone not themselves. Ridiculous? Because Microsoft would violate such a deal anyhow? Because the algorithm could simply be reimplemented? Perhaps. But it would be another roadblock. Fourth, why do we continue to believe that Google's strength comes from proprietary code? Why do we not recognize that its strength comes from mindshare, user experience, and quality of execution? None of which would be negatively affected by releasing the source. Perhaps Google would face heightened competition as a result of releasing its algorithm. Perhaps this would outweigh the benefits of doing so. But there would be significant good which would result, including public feedback which would make it significantly harder for spammers to be successful. But I'm just a foaming-at-the-mouth open source radical. And I'm sure history is on your collective side. If there's anything we've learnt from a hundred years or so of Computer Science (overestimate if you only count actual programming, underestimate if you allow Babbage as you should), it's surely that secrecy is what drives innovation, right?
ez77 17y ago
As a matter of fact, I think Google represents the ideal business model: one which should work even in a patents-free, copyright-free world.
- ez77 17y ago
  Let me elaborate. Not even Stallman would call for this... What can of totalitarian regime is desired, where businesses (even individuals?) are forced to publish all of their code? Wouldn't it be good enough to do away with patents and copyright? As long as I manage to keep my stuff private, let me (and Google) face the advantages and disadvantages. Some balance in the force, for Yoda's sake.
rpdillon 17y ago
This post manages to completely miss the point of the Google letter on openness. As some have incorrectly asserted, Google's latest post is not about "everything open", it is an internal memo to product managers that it occurred to them would be fit for public consumption. The central thesis of the letter is that "open" is hard to define (and defined differently by different people) and yet crucially important. Because it is hard to define, many people misuse it (OpenDNS comes to mind), and no one can call them on it because it is a vague term. Internally, Google employees argue about what open means, and the purpose of this letter was to clarify what open means to Google. They lay it out in extremely clear terms, and it is well structured. This response essay by Dixon misses all that completely - he unilaterally defines "open" to mean "open source", and then asks for Google to give away their software, claiming that if they don't, they're hypocrites. Going back to Google's definition of "open" (which is complex), one significant aspect they emphasize is the notion of "lock-in", i.e. locking in users, and locking out competitors. This is one area where Google puts their money where their mouth is; they let users leave. You can get at your email, export your Google Reader content, download all your Google Docs, archive your Picasa pictures, and export all your contacts. That means if a competitor comes along with something better, you can start using it. Try that with Facebook. Dixon acting like that is worthless because it isn't the "open-source" form of "open" is patently absurd. If this is confusing, I highly recommend reading Simon Phipps essays on open source to understand what kind of software makes a good candidate for a business to open source (operating systems, browsers, compilers...the basics that we all use to compete) and what does not (grid/cloud computing algorithms, specialized software, and other types of software that are differentiators). Essentially, the business model is to pool our collective resources in areas in which we don't want to compete (web server software, operating systems) and to spend time innovating in areas we do wish to compete (search algorithms, voice recognition, image analysis, etc.) Over time, as those specialized areas become more mainstream, they become better candidates for open-sourcing.
eli 17y ago
Yeah, and where's my pony?
vicaya 17y ago
I'm an open source advocate (even for search algorithms in general), but ranking algo/model is inherently adversarial and gameable until true search AI (always correctly identify good and original content without false positives) is feasible. I think it would be a good revenue strategy, if they charge $dollars to display top 10 positive features and top 10 negative features of any given query-document pair without any real weights/scores (maybe re-squashed with some logistic function). The $dollars amount is computed dynamically to offset the negative impact of the partial reveal of the ranking algo/model. This can be a tremendously useful tool for troubleshooting search results by site owners.
russell 17y ago
A more reasonable proposal would be for someone to pull a netflix; $1 million or even a $1o million prize for a significant advance in search relevancy. $10 million would be cheap for MS, Yahoo, ASK, a VC, or even DARPA.

57 points by ivankirigin 17y ago

(I used to work in the Search Quality division at Google.) Security by obscurity is a bad analogy here. The problem is that Google's algorithms are not mathematically perfect, in the sense of cryptography. Instead, they merely statistically tend to find good results. You can't find bugs just by looking at source code. You have to statistically analyze the effects of any proposed changes.

showerst 17y ago

The notion that being 'committed' to open source means that a company should give away everything of value is ludicrous. Google came up with great software that makes millions of peoples' lives easier, and they have every right to keep it secret and profit off of it. The zealous idea that open source and trade secrets cannot coexist is detrimental to the open source movement as a whole, in my opinion.

rriepe 17y ago
Well put. Almost any good idea taken to its extreme is a bad idea.
- electromagnetic 17y ago
  High taxation eventually reaches communism, if there's benefits like healthcare 'refunded' to the citizen and if there isn't then that ends up being a kleptocracy. Low taxation eventually reaches anarchy as the government is no longer in control. Increasing birth rates enables economies to grow without becoming stagnant and old, placing undue demand due to pensions. An exceptionally high birth rate creates food production problems, housing problems, and generally the 'sardine can' effect. Low birthrates create a stagnant civilization that will eventually reach a tipping point when taxes get so high to pay for pensions that the younger generations will protest. Market freedom to the extreme leads to monopolies taking advantage, 'gentlemen's agreements' between companies to price fix and many other problems that rules and regulations try to control. Equally a controlled market develops problems in and of itself as it's unable to respond naturally to market fluctuations, and quickly becomes unruly and unstable, and as many of the countries who have held long-term controlled markets are testaments to, they develop massive black-market economies to deal with these. The zealous burning of fossil fuels, aside from global warming concerns, can expel vast quantities of radioactive materials, sulfurous emissions and a long list of things we've accepted in western society as being more harmful than smoking to the wide populations health, but rules and regulations on air quality increasingly get more lax (at least they have been where I live). With zero fossil fuel production, however, our civilization would collapse (at least with an immediate stop to production), and I'm not just talking the halting of transportation vehicles, I'm talking the stop in production of the advanced polymers used in virtually all aspects of our lives. The majority of people wouldn't have clothes on their back . . . my wardrobe would certainly be a lot cheaper if my clothes were all genuine cotton and wool. Oh and farewell cheap elasticated sweatpants. Every good idea taken to its extreme is a bad idea and will create monumental problems in its wake. Electric vehicles are amazing, however a pure-electric is a ridiculous prospect in winter when the capacity hits below 60% of room-temperature, suddenly your vehicle can't get you to work and back on a single charge. Zero integration of hybrids is, as everyone knows, damaging to the environment but also an extreme waste of entropy. A 1 tonne vehicle at 100km/h is a massive store of energy, and allowing that to be wasted is criminal, the amount of energy humanity has wasted on braking at highway speeds would likely run the entire human civilization for over a year.
mattmanser 17y ago
I think you're missing the point, the article is pretty much a direct response to: http://googleblog.blogspot.com/2009/12/meaning-of-open.html http://googleblog.blogspot.com/2009/12/meaning-of-open.html (submitted here: http://news.ycombinator.com/item?id=1008908 http://news.ycombinator.com/item?id=1008908) In it a Google open source zealot rants wildly about openess being ultimate, apart from search algorithms which should be closed. It also fails to mention all the times open has failed (communism being one of the more spectacular failures of open). A cynic might surmise what they actually mean is content should be open so they can profit from their closed search algorithm.
- jedc 17y ago
  It was written by Jonathan Rosenberg, who's a Senior Vice President for Product Management, and on the top management committee at Google. That doesn't quite meet my personal description of an "open source zealot", though he's clearly quite passionate about it.
- RyanMcGreal 17y ago
  > communism being one of the more spectacular failures of open Wait, what? Communism is a spectacular failure of centralized planning over the entire economy. It has nothing to do with open or closed in principle, and in every implementation it has been aggressively closed in terms of information flows and decision making.
  - jacquesm 17y ago
    > Communism is a spectacular failure of centralized planning over the entire economy. Wow, and here I was thinking it was a form of government. Communism in principle has absolutely nothing to do with the flow of information and/or decision making, and its implementations have only very little to do with its basic ideas. A true communistic government has never even been tried, we've only seen a 'new boss same as the old boss' approach to communism with one batch of fat cats replacing another.
    - mbreese 17y ago
allenp 17y ago
Additionally it may be the case that a large part of the value of the algorithm is created from its secrecy. If the algorithm was open it would be trivial to create sites with artificially high rankings which would reduce the value of the search results.
- chollida1 17y ago
  > If the algorithm was open it would be trivial to create sites with artificially high rankings I don't necessarily think this is true. Sites that have high page rank, do so because other sites of high value link to them. Even if Google open sourced their search engine, spam sites would still have a hard time getting high quality inbound links. The reputable and high ranking sites wouldn't have any increased incentive to link to low quality spam sites.
  - DougWebb 17y ago
    It's not the ranking algorithm specifically that's a secret; it's the code that used to detect sites that work together to artificially raise page rank of another site by linking to it. The whole Page Rank algorithm is predicated on the idea that human website owners and contributors do a better job at assessing the value of a website and they express this by linking to the website. When links are created for other reasons they reduce the effectiveness of Page Rank, and when they are created specifically to bias Page Rank they do the most damage to the algorithm's results. Keeping the algorithm secret, and in particular the tweaks that eliminate links which aren't expressing "I judge this to be a good site", keep Page Rank useful.
  - EnderMB 17y ago
    Absolutely. I've worked in several contracted roles within online marketing and I've heard some awful advice from self-proclaimed SEO gurus about how to achieve a high rank for keywords on Google. A lot of what achiving high ranks is about is common sense and as Google improves their algorithm fewer tricks work. Anything past a simple competitive analysis, keyphrase analysis and a backlink analysis on you and your competitors is overkill. The quality of SEO analysts out there is so poor that I believe you could release the source code to Google's entire search engine and people would still get things wrong or believe in ways to get around what I assume is an extremely complex set of mathematically-oriented algorithms. It's a classic example of the blind following the blind, and those that typically have followers are those that are popular for performing SEO work on huge companies that would never actually need it (e.g. Coca Cola).
kvs 17y ago
Agreed. After reading the comments at cdixson.org by the author and reading openness article at Google, it seems like author is simply saying: "I agree they won't open it. But then they should stop talking sactimoniously about openness." which I agree with as well. Here is an interesting perspective: "And millions of websites will have to continue blindly relying on a small group of anonymous engineers in charge of the secret algorithm that determines their fate." Have you ever been delisted by Google? Is there a transparent process to get back on the list? Perhaps transparency is what Google needs the most at the moment.
- jacquesm 17y ago
  > Perhaps transparency is what Google needs the most at the moment. Absolutely, and not only on the search engine part of it, Adwords/Adsense are extremely opaque and as a publisher continuously give you the feeling you're being screwed without any chance to prove that for lack of information.
  - mbreese 17y ago
    As a publisher, couldn't you do some sort of A/B testing with another ad provider? Something like 10% of page views using another set of ads? Or, maybe setup some kind of onclick tracking to verify ad clicks? I don't know what could be done as verification for the advertiser, but the publisher should be able to do something, right?
    - eli 17y ago
      No one else gets enough volume :/
  - zaphar 17y ago
    You say that now but if they provided that transparency then the amount of fraudulent activity would skyrocket. Knowing the details of the algorithms involved does nothing to help you as an advertiser or searcher. Googles extreme impartiality in search and ads is their greatest strength and open sourcing the algorithms involved would do nothing to increase that strength and contribute to a great deal less useful ads and search results on average.
iamelgringo 17y ago
The notion that being 'committed' to open source means that a company should give away everything of value is ludicrous. I disagree with the statement that Google would be giving away everything by opening up page rank. Google's "secret sauce" is hardly page rank alone. Google is really secretive about this stuff, but they arguably have the world's larges corpus of multilingual text in the form of their html index, coupled with the world's largest fault tolerant compute cluster, the largest repository of scanned text,the largest repository of digital video, and probably the largest repository of digital audio in the world. I'm sure that they also have the world's largest repository of people's surfing and searching habits, along with the one of the largest sets of analytics data. Even if page rank, map reduce, GFS, etc.. were open sourced I'm sure they'd still find a way to make billions off of the data that they have. But, I think the larger point that the author makes, is that unlike many giant technology companies (Microsoft, HP, Apple, etc..), Google has made billions off of using open source software. And, while they are a very large supporter of open source, what total percentage of their profits go back to OSS? What percentage of their code gets re-released as open source? These are good questions to ask, and I'd love to hear an open discussion about it.

buster 17y ago

opening this algorithm would lead to A LOT of new spammers and SEO-crap. Google, please keep it a secret, PLEASE!

mustpax 17y ago

The analogy to "security through obscurity" is flawed. Open security works when there's an obvious way a secure system should behave, say, make it infeasible to open a lock/decrypt data without the correct key. In Google’s, case the correct relevancy ordering of the web is a huge point of contention. So they cannot simply open it up hoping to close out all the holes in the open. It is also hard to distinguish acceptable behavior (linking) from unacceptable behavior (selling links) except in the aggregate through heuristics that assume ignorance on the part of the attacker. In this sense, Google’s system is a lot closer to an intrusion detection system than a lock or a cipher suite. You want to establish some ground rules but there’s no added security in declaring what you’re monitoring on your servers and how.

swombat 17y ago

That is a completely ridiculous point to make. Come on, be serious here. What would be the benefit to anyone except for spammers, if Google open-sourced their algorithms? It's a lose-lose option. Google loses some money, we lose a lot of search engine accuracy, and spammers gain new ways to screw around with search results. And does anyone gain from it? Not really. At this point, Google's supremacy is due just as much to their hardware know-how, to their market leader position, and to their excellent brand, as it is to their search algorithms. It won't help any competing search engines - just spammers.

roc 17y ago
The stated goal isn't to help other search engines; it's to get the community to help Google improve, for the good of all its users. The theory goes: if you let more people look at the algorithm, they can suggest better answers than Google alone, to thwart the spammers who are gaming the system either way. But I suspect the real goal has more to do with the deep suspicion that Google can or does do a significant bit of hand-tweaking. And these people want to force the algorithm into the open to either end or curtail that.
- jacquesm 17y ago
  I'll believe that when I see open source achieve a conclusive victory against email spam. Spam:email = 90% or worse in search results it is closer to the opposite. I'm sure once the open source community actually manages to do this and prove the 'many eyes' doctrine applies to more than just security and bug fixes that many doors will open.

infynyxx 17y ago

Google's search ranking algorithm is their bread and butter so, I don't think they will ever open source it. But what always strikes me is that they have not yet open sourced their BigTable database systems even though, it's not their core revenue generating source. There are clones of BigTable in the form of Cassandra and HyperTable but it would be so much better if BigTable is open sourced.

bbsabelli 17y ago

tl;dr? Author confuses source code with revenue. Welcome to the less-than-free world.

jlees 17y ago

An interesting thing to think about even though the basic reaction is "wait, no, that's their industry secret/revenue basis/whatever". Google is so dominant right now that maybe it's going to get to the point where, in the interests of the open web, they get pressured and lobbied to release their secret sauce; after all, as swombat points out, it's not just the algorithm by a long chalk. It's especially interesting if you consider the origins of PageRank in academia, freely published and peer-reviewed, and still not that well understood by laypeople; would the open-sourced magic of Google even be understandable by that many folk? Those that do understand it could make a fortune as consultants; at the moment, shady SEO snake-oil salesmen are so prevalent and 99% of their advice is outdated or plain wrong, something that definitively calls them out on it could be helpful.

amix 17y ago

I would love if they open-sourced their infrastructure tools such as GFS, BigTable and Map-Reduce tools. Most of these tools are built using open-source technology such as Linux.

stanleydrew 17y ago
Apache's Hadoop (http://hadoop.apache.org/ http://hadoop.apache.org/) has open-sourced implementations of those tools.
- iamelgringo 17y ago
  Yahoo was the company that opensourced hadoop, not google.
  - stanleydrew 17y ago
    Never said it was Google, just pointing out that open-source implementations exist.
- mbreese 17y ago
  I'd be interested to see some benchmarks (unofficial-probably) between Hadoop and Google's C++ versions (at least, I assume Google's is C++). I've always wondered if there was any extra overhead in using Hadoop. I know Java is almost if not faster than C++ in some areas, and long running servers (like Hadoop) should benefit the most. But I wonder if there are some extra limitations Java places on things like BigTable.

stuff4ben 17y ago

I don't get it, is the author talking about PageRank? http://en.wikipedia.org/wiki/PageRank http://en.wikipedia.org/wiki/PageRank What more does he want?

minsight 17y ago
The algorithm used by Google today is light-years away from that described in the PageRank paper. It uses a form of PageRank, but it also uses many other parameters.
- spf 17y ago
  If it's secret, then how can you be so sure? My impression is that PageRank, while initially extremely successful at filtering signal-from-noise, is very susceptible to manipulation. For example links farms which make unimportant pages seem important. As a result of such manipulation, Google started modifying their crawling, in order to reduce the amount of garbage going into PageRank (less garbage in, less garbage out). For example, they might look at the age and ownership in DNS records, and exclude domains that look fishy based on some heuristics. The spammers fought back with more tricks, and the resulting arms race has come to have an acronym: SEO. But the underlying idea of using an iterative approximation of the likelihood of encountering a page via a random walk of the web is preserved. (Some details here: http://www.miislita.com/information-retrieval-tutorial/matrix-tutorial-3-eigenvalues-eigenvectors.html http://www.miislita.com/information-retrieval-tutorial/matri...) If that's right (and I could well be wrong... would like to be convinced...), then I don't see it as a "light-years away" kind of issue, but rather a series of hacks to stay one step ahead of the spammers. I don't see any particular value in open sourcing those.
mooism2 17y ago
Google rankings are determined by much more than PageRank.

Mystalic 17y ago

Google is a business. Their job, legally defined, is to create value (aka profit) for its shareholders. A more open web has helped Google, but opening up its search algorithms, a cornerstone to Google's growth, is just ludicrous and bad business. Compared to a lot of other companies (Apple, Microsoft, nearly every other tech giant), they are an open company. It's already great that they realize there's value in an Open-source browser, and open-source OS, and multiple open platforms.

tome 17y ago
Their job, legally defined, is to create value (aka profit) for its shareholders. This is a meme I see a lot, but never with any evidence. That the board has a fiduciary duty to the shareholders I can understand. That it would have a legal obligation to create value, well, I think that would be hard to define, and I have never seen any evidence for the existence of this obligation.
- deleted 17y ago
  [deleted]
- mbreese 17y ago
  In the context of a publicly traded company this is the definition of fiduciary. The company has sold stock to investors with the sole purpose of increasing the value of the stock. Therefore, it is up to the board to make decisions that reflect this duty. You can argue about the timeline (short versus long-term growth), but you can't say that there isn't a legal obligation to attempt to increase value.
  - roc 17y ago
    > "The company has sold stock to investors with the sole purpose of increasing the value of the stock." I know they're not popular, or exciting like a stock value swing, but dividends still exist, right?
    - mbreese 17y ago

mlLK 17y ago

http://www.google.com/support/webmasters/ http://www.google.com/support/webmasters/ easily brings down this bogus claim. I honestly don't think much would change even if they did open-source their PageRank algorithm because the people willing and able enough to exploit it already do, and regardless of how much their PageRank increases Google will send your domain deep into an internet abyss and out of their index.

allenp 17y ago

I thought open source was about software, not about logic (business or otherwise) - am I missing something here?

hello0012356 17y ago

Welcome to: http://www.wowowbiz.com http://www.wowowbiz.com The website wholesale for many kinds of fashion shoes, like the nike,jordan,prada,adidas, also including the eans,shirts,bags,hat and the decorations. All the products are free shipping, and the the price is competitive, and also can accept online payment.,after the payment, can ship within short time. http://www.wowowbiz.com http://www.wowowbiz.com our price: jordan air max oakland raiders $34--39; Ed Hardy AF JUICY POLO Bikini $25; Christan Audigier BIKINI JACKET $25; gstar coogi evisu true jeans $35; coach chanel gucci LV handbags $36; coogi DG edhardy gucci t-shirts $18; CA edhardy vests.paul smith shoes $32; jordan dunk af1 max gucci shoes $37; EDhardy gucci ny New Era cap $16; coach okely Adidas CHANEL DG Sunglass $18; http://www.wowowbiz.com http://www.wowowbiz.com thank you for your visit Wish you a merry Christmas

hello12593 17y ago

btilly 17y ago

I upvoted the story not because I agree with it, but because it explains a very successful open sourcing strategy that I wish more people understood. In general people are willing to spend a certain amount on a solution. Therefore if you can things that make the complement to what you make money on cheaper, that increases how much you can charge for your piece of the solution. Open sourcing is a way to make those pieces cheaper. Google does this. They give away tools and knowledge about how to make websites. They give away analytics and an A/B testing platform and educate people about how that makes it easier to produce good websites. This makes building good websites cheaper, and therefore increases how much people are willing to spend to advertise those websites. This isn't a new strategy. In the late 90s when Oracle and IBM got into open source, this was their strategy. Oracle wanted to get people to stop paying for Sun hardware so Oracle could negotiate higher prices. (I know more than a few people who were surprised to find that licenses for Oracle on Linux are higher than they are on Sun.) IBM was looking to reduce the price for complements to IBM websphere, and was looking for opportunities to avoid having customers pay the Microsoft tax. If you run a company and wish to strategically open source something, you should think in the same terms. What is the complement to what you are charging for, and how can you make that cheaper?

selven 17y ago

And what is wrong with commoditizing the complement? Google's working hard to make operating systems free and we'll probably see Microsoft working hard to open up search engines. End result: both operating systems and search engines will be open and the consumer benefits.

xcombinator 17y ago

Well, I can't understand this article, there is not "secret sauce". The google pagerank algorithm is open for all to see and understand. I myself made my own simple implementation. Everybody could do it. This man is not asking for code, is asking for DATA only google has. It's like an atheist saying to a catholic what he must do because he believes. When this man gives something of value to the world,like google did(I remember the Altavista-Ad-flickering in your eyes days) then he could demand to be given by others.

eli 17y ago
The documented PageRank algorithm doesn't have very much to do with how results are actually returned from Google today.

oujheush 17y ago

I am disturbed by the lack of faith in open source among the readers here. I mean, I understand some skepticism but this, to me, is pretty extreme IP-fetishism. It's perfectly understandable. Lots of money is made in the existing paradigm and so many of you have incentive to believe it's the most rational system. Besides, hard to imagine Google's doing something wrong, right? But the argument that "the only people this will help is spammers"? Really? Are the only people helped by open cryptographic protocols those who look to decipher traffic? "They have every right to keep it secret and profit off of it." Let's allow this for a moment. We've still got a false dichotomy. Making it public will not deprive them of their profit. First, incredible marketing coup. Does anyone else remember how much of Google's initial strength came from its rabid fans? This would do a lot of re-energize that base. Second, are there going to be new startups which can effectively compete with Google, even given its algorithm? How are they going to provide a better experience, or serve the volume of traffic, or index a larger amount of pages with the same algorithm with fewer resources? Third, who says they have to authorize competitors to use it? This is I think one of the more interesting points. They could release the algorithm but license it only for non-commercial use for anyone not themselves. Ridiculous? Because Microsoft would violate such a deal anyhow? Because the algorithm could simply be reimplemented? Perhaps. But it would be another roadblock. Fourth, why do we continue to believe that Google's strength comes from proprietary code? Why do we not recognize that its strength comes from mindshare, user experience, and quality of execution? None of which would be negatively affected by releasing the source. Perhaps Google would face heightened competition as a result of releasing its algorithm. Perhaps this would outweigh the benefits of doing so. But there would be significant good which would result, including public feedback which would make it significantly harder for spammers to be successful. But I'm just a foaming-at-the-mouth open source radical. And I'm sure history is on your collective side. If there's anything we've learnt from a hundred years or so of Computer Science (overestimate if you only count actual programming, underestimate if you allow Babbage as you should), it's surely that secrecy is what drives innovation, right?

ez77 17y ago

As a matter of fact, I think Google represents the ideal business model: one which should work even in a patents-free, copyright-free world.

ez77 17y ago
Let me elaborate. Not even Stallman would call for this... What can of totalitarian regime is desired, where businesses (even individuals?) are forced to publish all of their code? Wouldn't it be good enough to do away with patents and copyright? As long as I manage to keep my stuff private, let me (and Google) face the advantages and disadvantages. Some balance in the force, for Yoda's sake.

rpdillon 17y ago

This post manages to completely miss the point of the Google letter on openness. As some have incorrectly asserted, Google's latest post is not about "everything open", it is an internal memo to product managers that it occurred to them would be fit for public consumption. The central thesis of the letter is that "open" is hard to define (and defined differently by different people) and yet crucially important. Because it is hard to define, many people misuse it (OpenDNS comes to mind), and no one can call them on it because it is a vague term. Internally, Google employees argue about what open means, and the purpose of this letter was to clarify what open means to Google. They lay it out in extremely clear terms, and it is well structured. This response essay by Dixon misses all that completely - he unilaterally defines "open" to mean "open source", and then asks for Google to give away their software, claiming that if they don't, they're hypocrites. Going back to Google's definition of "open" (which is complex), one significant aspect they emphasize is the notion of "lock-in", i.e. locking in users, and locking out competitors. This is one area where Google puts their money where their mouth is; they let users leave. You can get at your email, export your Google Reader content, download all your Google Docs, archive your Picasa pictures, and export all your contacts. That means if a competitor comes along with something better, you can start using it. Try that with Facebook. Dixon acting like that is worthless because it isn't the "open-source" form of "open" is patently absurd. If this is confusing, I highly recommend reading Simon Phipps essays on open source to understand what kind of software makes a good candidate for a business to open source (operating systems, browsers, compilers...the basics that we all use to compete) and what does not (grid/cloud computing algorithms, specialized software, and other types of software that are differentiators). Essentially, the business model is to pool our collective resources in areas in which we don't want to compete (web server software, operating systems) and to spend time innovating in areas we do wish to compete (search algorithms, voice recognition, image analysis, etc.) Over time, as those specialized areas become more mainstream, they become better candidates for open-sourcing.

eli 17y ago

Yeah, and where's my pony?

vicaya 17y ago

I'm an open source advocate (even for search algorithms in general), but ranking algo/model is inherently adversarial and gameable until true search AI (always correctly identify good and original content without false positives) is feasible. I think it would be a good revenue strategy, if they charge $dollars to display top 10 positive features and top 10 negative features of any given query-document pair without any real weights/scores (maybe re-squashed with some logistic function). The $dollars amount is computed dynamically to offset the negative impact of the partial reveal of the ranking algo/model. This can be a tremendously useful tool for troubleshooting search results by site owners.

russell 17y ago

A more reasonable proposal would be for someone to pull a netflix; $1 million or even a $1o million prize for a significant advance in search relevancy. $10 million would be cheap for MS, Yahoo, ASK, a VC, or even DARPA.