6 ms·

Summary of the approach (p10): "1) Public curation mechanisms for quality; 2) Transparency, telling users exactly how the information originated; 3) Open dat

by abalone 11y ago

Summary of the approach (p10): "1) Public curation mechanisms for quality; 2) Transparency, telling users exactly how the information originated; 3) Open data access to metadata, giving users the exact date source of the information; 4) Protected user privacy, with their searching protected by strict privacy controls; 5) No advertising, which assures the free flow of information and a complete separation from commercial interests; 6) Internalization, which emphasizes community building and the sharing of information instead of a top-down approach." My first thought: How will transparency impact SEO? Will spammers be able to better game the algorithm when they know its internals? However I am excited at the prospect of a wikipedia-like public curation system for the entire web. I admit I'm flabbergasted that the whole thing ever worked, but it does.

jondubois 11y ago
I'm pretty sure that the spammer argument is just an excuse used by Google to allow them to keep their business practices out of public scrutiny. Google search results are biased in favour of content produced by those who have money and power. Google ranks everything based on popularity - Not based on quality. Popularity and quality are two independent concepts and not necessarily related. That's something which Wikimedia understands but which Google doesn't.
- bitmapbrother 11y ago
  I'd ask you to cite your claims, but we both know you can't. It's a pity your issues with Google cause you to pollute discussions with BS.
  - userbinator 11y ago
    Do a Google search for anything even slightly obscure, and you're likely to find the first page or so of results filled with highly-SEO'd sites that offer little in the way of deep, detailed content. The smaller sites which do have that content, but just haven't been SEO'd much, have been eclipsed. They're still there, but rendered nearly inaccessible. Interesting discussion on a search engine that does sort of the opposite of Google: https://news.ycombinator.com/item?id=3910304 https://news.ycombinator.com/item?id=3910304
    - nl 11y ago
      I hear this a lot, but I'd love to see an example (especially including the sites that should rank).
      - luxpir 11y ago
        This may or may not be what you're looking for, but my freelance site just can't hit the front page in my industry when dozens of highly funded agencies can dominate it. I've published a 50k word industry-specific book on my site, have SEO'd as much as possible and have an older domain than those better funded. Won't link to it here, but it seems to be a real issue to me. If a search engine let you differentiate and sort between content match vs pagerank match vs Adwords spend we might be able to mitigate the issue somewhat.
        cpeterso 11y ago
        btw, your site's redirect to the HTTPS version doesn't seem to work correctly in Firefox, Safari, or IE. After reading your comment, I was curious to learn more. When I typed in just your domain name plus CMD+ENTER (which adds "www." and ".com" to the address bar text in Firefox), I got a 404 page, not the 301 redirect to the HTTPS site. When I add "http://" http://", the redirect seems to work.
        nl 11y ago
        This. I don't work in this area, but I'd say that 99% of the time I hear someone complaining about how Google is favoring sites that pay for advertising over them I find that they are making these incredibly basic errors. For me, http://www.linguaquote.com/ http://www.linguaquote.com/ gives a 404. It's only when I go to https://www.linguaquote.com/ https://www.linguaquote.com/ that it works.
        luxpir 11y ago
        That's all well and good, and thanks for checking, but I wasn't referring to that site in the parent comment. The other site does have SSL enabled, but only recently and via Let's Encrypt. The issue is much longer standing than this. So I'm afraid it's not quite as simple as you make out in this case. In other news, I've just pinpointed the missing line in the recently changed nginx config for Linguaquote; the http://www http://www block had it's redirect commented out. Still, for this site in particular Google Webmaster tools is set up for the https version where no errors have been reported and SSL Labs gave an A+ for the stapling, PFS, heartbleed etc. etc. efforts I went to. I don't think this redirect was having an adverse effect on ranking, but I don't expect this site to hit the front page just yet - much more content to add before aiming for that.
  - kevingadd 11y ago
    If you've ever read about the original PageRank algorithm, the parent post is a pretty reasonable way to describe it. I have no idea what the current algorithm looks like but I'd be shocked if it somehow switched to evaluating the 'quality' of content, however one might do that with an algorithm.
    - justincormack 11y ago
      Well they do have some quality metrics, like duplication with other content, words used and so on. I suspect more, eg writing style measurements, correlated with other things that are found useful. There is a lot that could be done without actually understanding the content, although if course it can be gamed.
  - jondubois 11y ago
    The primary measurement used for Google's PageRank algorithm is the number and "value" of backlinks that a page has. The "value" of a backlink is determined by the cumulative number of descendent sub-backlinks that it has. This is common knowledge among SEO professionals. Basically, when judging quality, Google is making assumptions like: "This page has a lot of backlinks, and those backlinks themselves have a lot of backlinks... Therefore this page is of high quality." This approach puts all the power in the hands of content providers (bloggers) who are funded by big companies (or well-funded startups) and who serve the interests of those companies. Google wrongly assumes that content-providers serve the interest of consumers and that they can be trusted - Which is not the case.
- jonathankoren 11y ago
  Google does take quality into account. That's the whole point behind static ranking algorithms. However, quality isn't some universal concept. I'd say the inevitable paper on gravity wave detection is highest quality on it, but it certainly isn't popular because it's impenetrable to the lay masses, unlike say a Wikipedia article, which falls into look-at-me-i'm-oh-so-smart territory when it gets into double gradients and other math formulas with more letters than digits.
- nl 11y ago
  If you think "spam" isn't the defining problem of web search then you've never tried building a search engine. It's 90% of the problem. Google does take plenty of quality features into account[1]. PageRank is one of course, but that isn't some corporate conspiracy, it's that it is a good feature. [1] https://moz.com/search-ranking-factors/correlations https://moz.com/search-ranking-factors/correlations
PythonForGirls 11y ago
"Public Curation" doesn't make quality. It makes a mob-rule system where only the most popular ideas flourish.
- nickpsecurity 11y ago
  Nonsense. It means content will be filtered through the lens of one or more individuals. The results vary dramatically. Mob-rule is one possibility. Yahoo Directory used to be a great example of mid-level of quality where it gave nice start and obscure stuff people overlooked. On high end, the link below shows Stanford Encyclopedia of Philosophy set a pretty awesome precedent for high-quality curation: https://news.ycombinator.com/item?id=10266103 https://news.ycombinator.com/item?id=10266103
contingencies 11y ago
1) Public curation mechanisms for quality; The Mozilla / open directory project tried this. Curation doesn't scale and often assumes a single unifying ontology. This is particularly problematic in a cross-cultural context. Besides, 'quality' is not a unidimensional metric in a result set: consider timeliness, authority, notability, uniqueness, comprehensibility, etc. 2) Transparency, telling users exactly how the information originated; Most search engines include a URL, I can see a [crawldate] button like the [cache] or [translate] buttons on each hit adding some information, but this will be of dubious additional utility for most searches. 3) Open data access to metadata, giving users the exact date source of the information; As above. 4) Protected user privacy, with their searching protected by strict privacy controls; We have duckduckgo already, friends are welcome but it's hardly a unique offering nor a trustworthy one given Snowden's revelations regarding the scale of systematic 5 eyes traffic monitoring/recording. 5) No advertising, which assures the free flow of information and a complete separation from commercial interests; DDG or Google or Bing with plugins can supply this. Not ground breaking. 6) Internalization, which emphasizes community building and the sharing of information instead of a top-down approach. This is so amorphous as to be a non-point. So out of six points, 2 things (33%) are only useful in edge cases, 1 thing (16%) is too vague to be useful, and the other 3 things (50%) are currently implemented by others and have been tried before. I would like to see the input of the former Blekko guys on this, https://news.ycombinator.com/user?id=ChuckMcM https://news.ycombinator.com/user?id=ChuckMcM + https://news.ycombinator.com/user?id=greglindahl https://news.ycombinator.com/user?id=greglindahl
- bobajeff 11y ago
  Duckduckgo is ad free? I never knew this. How do they make money?
  - contingencies 11y ago
    My mistake. You are right, they do have ads, I just always use a blocker. I've updated the text.
  - stephenr 11y ago
    Ddg has an option to disable ads, they just ask that you help promote them.
deleted 11y ago
[deleted]
rmc 11y ago
> 4) Protected user privacy, with their searching protected by strict privacy controls; They'll have to keep the servers outside the USA then. It's illegal for European organisations to transfer personal data to the USA now that Safe Harbour is invalid.

by abalone 11y ago

jondubois 11y ago

I'm pretty sure that the spammer argument is just an excuse used by Google to allow them to keep their business practices out of public scrutiny. Google search results are biased in favour of content produced by those who have money and power. Google ranks everything based on popularity - Not based on quality. Popularity and quality are two independent concepts and not necessarily related. That's something which Wikimedia understands but which Google doesn't.

bitmapbrother 11y ago
I'd ask you to cite your claims, but we both know you can't. It's a pity your issues with Google cause you to pollute discussions with BS.
- userbinator 11y ago
  Do a Google search for anything even slightly obscure, and you're likely to find the first page or so of results filled with highly-SEO'd sites that offer little in the way of deep, detailed content. The smaller sites which do have that content, but just haven't been SEO'd much, have been eclipsed. They're still there, but rendered nearly inaccessible. Interesting discussion on a search engine that does sort of the opposite of Google: https://news.ycombinator.com/item?id=3910304 https://news.ycombinator.com/item?id=3910304
  - nl 11y ago
    I hear this a lot, but I'd love to see an example (especially including the sites that should rank).
    - luxpir 11y ago
      This may or may not be what you're looking for, but my freelance site just can't hit the front page in my industry when dozens of highly funded agencies can dominate it. I've published a 50k word industry-specific book on my site, have SEO'd as much as possible and have an older domain than those better funded. Won't link to it here, but it seems to be a real issue to me. If a search engine let you differentiate and sort between content match vs pagerank match vs Adwords spend we might be able to mitigate the issue somewhat.
      - cpeterso 11y ago
        btw, your site's redirect to the HTTPS version doesn't seem to work correctly in Firefox, Safari, or IE. After reading your comment, I was curious to learn more. When I typed in just your domain name plus CMD+ENTER (which adds "www." and ".com" to the address bar text in Firefox), I got a 404 page, not the 301 redirect to the HTTPS site. When I add "http://" http://", the redirect seems to work.
        nl 11y ago
        This. I don't work in this area, but I'd say that 99% of the time I hear someone complaining about how Google is favoring sites that pay for advertising over them I find that they are making these incredibly basic errors. For me, http://www.linguaquote.com/ http://www.linguaquote.com/ gives a 404. It's only when I go to https://www.linguaquote.com/ https://www.linguaquote.com/ that it works.
        luxpir 11y ago
        That's all well and good, and thanks for checking, but I wasn't referring to that site in the parent comment. The other site does have SSL enabled, but only recently and via Let's Encrypt. The issue is much longer standing than this. So I'm afraid it's not quite as simple as you make out in this case. In other news, I've just pinpointed the missing line in the recently changed nginx config for Linguaquote; the http://www http://www block had it's redirect commented out. Still, for this site in particular Google Webmaster tools is set up for the https version where no errors have been reported and SSL Labs gave an A+ for the stapling, PFS, heartbleed etc. etc. efforts I went to. I don't think this redirect was having an adverse effect on ranking, but I don't expect this site to hit the front page just yet - much more content to add before aiming for that.
- kevingadd 11y ago
  If you've ever read about the original PageRank algorithm, the parent post is a pretty reasonable way to describe it. I have no idea what the current algorithm looks like but I'd be shocked if it somehow switched to evaluating the 'quality' of content, however one might do that with an algorithm.
  - justincormack 11y ago
    Well they do have some quality metrics, like duplication with other content, words used and so on. I suspect more, eg writing style measurements, correlated with other things that are found useful. There is a lot that could be done without actually understanding the content, although if course it can be gamed.
- jondubois 11y ago
  The primary measurement used for Google's PageRank algorithm is the number and "value" of backlinks that a page has. The "value" of a backlink is determined by the cumulative number of descendent sub-backlinks that it has. This is common knowledge among SEO professionals. Basically, when judging quality, Google is making assumptions like: "This page has a lot of backlinks, and those backlinks themselves have a lot of backlinks... Therefore this page is of high quality." This approach puts all the power in the hands of content providers (bloggers) who are funded by big companies (or well-funded startups) and who serve the interests of those companies. Google wrongly assumes that content-providers serve the interest of consumers and that they can be trusted - Which is not the case.
jonathankoren 11y ago
Google does take quality into account. That's the whole point behind static ranking algorithms. However, quality isn't some universal concept. I'd say the inevitable paper on gravity wave detection is highest quality on it, but it certainly isn't popular because it's impenetrable to the lay masses, unlike say a Wikipedia article, which falls into look-at-me-i'm-oh-so-smart territory when it gets into double gradients and other math formulas with more letters than digits.
nl 11y ago
If you think "spam" isn't the defining problem of web search then you've never tried building a search engine. It's 90% of the problem. Google does take plenty of quality features into account[1]. PageRank is one of course, but that isn't some corporate conspiracy, it's that it is a good feature. [1] https://moz.com/search-ranking-factors/correlations https://moz.com/search-ranking-factors/correlations

PythonForGirls 11y ago

"Public Curation" doesn't make quality. It makes a mob-rule system where only the most popular ideas flourish.

nickpsecurity 11y ago
Nonsense. It means content will be filtered through the lens of one or more individuals. The results vary dramatically. Mob-rule is one possibility. Yahoo Directory used to be a great example of mid-level of quality where it gave nice start and obscure stuff people overlooked. On high end, the link below shows Stanford Encyclopedia of Philosophy set a pretty awesome precedent for high-quality curation: https://news.ycombinator.com/item?id=10266103 https://news.ycombinator.com/item?id=10266103

contingencies 11y ago

1) Public curation mechanisms for quality; The Mozilla / open directory project tried this. Curation doesn't scale and often assumes a single unifying ontology. This is particularly problematic in a cross-cultural context. Besides, 'quality' is not a unidimensional metric in a result set: consider timeliness, authority, notability, uniqueness, comprehensibility, etc. 2) Transparency, telling users exactly how the information originated; Most search engines include a URL, I can see a [crawldate] button like the [cache] or [translate] buttons on each hit adding some information, but this will be of dubious additional utility for most searches. 3) Open data access to metadata, giving users the exact date source of the information; As above. 4) Protected user privacy, with their searching protected by strict privacy controls; We have duckduckgo already, friends are welcome but it's hardly a unique offering nor a trustworthy one given Snowden's revelations regarding the scale of systematic 5 eyes traffic monitoring/recording. 5) No advertising, which assures the free flow of information and a complete separation from commercial interests; DDG or Google or Bing with plugins can supply this. Not ground breaking. 6) Internalization, which emphasizes community building and the sharing of information instead of a top-down approach. This is so amorphous as to be a non-point. So out of six points, 2 things (33%) are only useful in edge cases, 1 thing (16%) is too vague to be useful, and the other 3 things (50%) are currently implemented by others and have been tried before. I would like to see the input of the former Blekko guys on this, https://news.ycombinator.com/user?id=ChuckMcM https://news.ycombinator.com/user?id=ChuckMcM + https://news.ycombinator.com/user?id=greglindahl https://news.ycombinator.com/user?id=greglindahl

bobajeff 11y ago
Duckduckgo is ad free? I never knew this. How do they make money?
- contingencies 11y ago
  My mistake. You are right, they do have ads, I just always use a blocker. I've updated the text.
- stephenr 11y ago
  Ddg has an option to disable ads, they just ask that you help promote them.

deleted 11y ago

[deleted]

rmc 11y ago

> 4) Protected user privacy, with their searching protected by strict privacy controls; They'll have to keep the servers outside the USA then. It's illegal for European organisations to transfer personal data to the USA now that Safe Harbour is invalid.