4 ms·

Google's entire (initial) claim-to-fame was "PageRank", referring both to the ranking of pages and co-founder Larry Page, which strongly prioritised a relevance

Google's entire (initial) claim-to-fame was "PageRank", referring both to the ranking of pages and co-founder Larry Page, which strongly prioritised a relevance attribute over raw keyword findings (which then-popular alternatives such as Alta Vista, Yahoo, AskJeeves, Lycos, Infoseek, HotBot, etc., relied on, or the rather more notorious paid-rankings schemes in which SERP order was effectively sold). When it was first introduced, Google Web Search was absolutely worlds ahead of any competition. I remember this well having used them previously and adopted Google quite early (1998/99). Even with PageRank result prioritisation is highly subject to gaming. Raw keyword search is far more so (keyword stuffing and other shenanigans), moreso as any given search engine begins to become popular and catch the attention of publishers. Google now applies other additional ordering factors as well. And of course has come to dominate SERP results with paid, advertised, listings, which are all but impossible to discern from "organic" search results. (I've not used Google Web Search as my primary tool for well over a decade, and probably only run a few searches per month. DDG is my primary, though I'll look at a few others including Kagi and Marginalia, though those rarely.) <https://en.wikipedia.org/wiki/PageRank https://en.wikipedia.org/wiki/PageRank> "The anatomy of a large-scale hypertextual Web search engine" (1998) <http://infolab.stanford.edu/pub/papers/google.pdf http://infolab.stanford.edu/pub/papers/google.pdf> (PDF) Early (1990s) search engines: <https://en.wikipedia.org/wiki/Search_engine#1990s:_Birth_of_search_engines https://en.wikipedia.org/wiki/Search_engine#1990s:_Birth_of_...>.

saltysalt 9mo ago
PageRank was an innovative idea in the early days of the Internet when trust was high, but yes it's absolutely gamed now and I would be surprised if Google still relies on it. Fair play to them though, it enabled them to build a massive business.
- marginalia_nu 9mo ago
  Anchor text information is arguably a better source for relevance ranking in my experience. I publish exports of the ones Marginalia is aware of[1] if you want to play with integrating them. [1] https://downloads.marginalia.nu/exports/ https://downloads.marginalia.nu/exports/ grab 'atags-25-04-20.parquet'
  - saltysalt 9mo ago
    Very interesting, and it is very kind of you to share your data like that. Will review!
  - dredmorbius 9mo ago
    Though I'd think that you'd want to weight unaffiliated sites' anchor text to a given URL much higher than an affiliated site. "Affiliation" is a tricky term itself. Content farms were popular in the aughts (though they seem to have largely subsided), firms such as Claria and Gator. There are chumboxes (Outbrain, Taboola), and of course affiliate links (e.g., to Amazon or other shopping sites). SEO manipulation is its own whole universe. (I'm sure you know far more about this than I do, I'm mostly talking at other readers, and maybe hoping to glean some more wisdom from you ;-)
    - marginalia_nu 9mo ago
      Oh yeah, there's definitely room for improvement in that general direction. Indexing anchor texts is much better than page rank, but in isolation, it's not sufficient. I've also seen some benefit fingerpinting the network traffic the websites make using a headless browser, to identify which ad networks they load. Very few spam sites have no ads, since there wouldn't be any economy in that. e.g. https://marginalia-search.com/site/www.salon.com?view=traffic https://marginalia-search.com/site/www.salon.com?view=traffi... The full data set of DOM samples + recorded network traffic are in an enormous sqlite file (400GB+), and I haven't yet worked out any way of distributing the data yet. Though it's in the back of my mind as something I'd like to solve.
      - dredmorbius 9mo ago
        Oh, that is clever! I'd also suspect that there are networks / links which are more likely signs of low-value content than others. Off the top of my head, crypto, MLM, known scam/fraud sites, and perhaps share links to certain social networks might be negative indicators.
        9mo ago
- snowwrestler 9mo ago
  Google’s biggest search signal now is aggregate behavioral data reported from Chrome. That pervasive behavioral surveillance is the main reason Apple has never allowed a native Chrome app on iOS. It’s also why it is so hard to compete with Google. You guys are talking about techniques for analyzing the corpus of the search index. Google does that and has a direct view into how millions of people interact with it.
  - saltysalt 9mo ago
    Yes indeed, they have an impossibly deep moat and deeper pockets. I'm certainly not trying to compete with them with my little side project, it's just for fun!
  - xnx 9mo ago
    > That pervasive behavioral surveillance is the main reason Apple has never allowed a native Chrome app on iOS The Chrome iOS app still knows every url visited, duration, scroll depth, etc.
  - danans 9mo ago
    > That pervasive behavioral surveillance is the main reason Apple has never allowed a native Chrome app on iOS. There is a native Chrome app on iOS. It gets all the same url visit data as Chrome on other platforms. Apple blocks 3rd party renderers and JS engines on iOS to protect its App Store from competition that might deliver software and content through other channels that they don't take a cut of.

saltysalt 9mo ago

PageRank was an innovative idea in the early days of the Internet when trust was high, but yes it's absolutely gamed now and I would be surprised if Google still relies on it. Fair play to them though, it enabled them to build a massive business.

marginalia_nu 9mo ago
Anchor text information is arguably a better source for relevance ranking in my experience. I publish exports of the ones Marginalia is aware of[1] if you want to play with integrating them. [1] https://downloads.marginalia.nu/exports/ https://downloads.marginalia.nu/exports/ grab 'atags-25-04-20.parquet'
- saltysalt 9mo ago
  Very interesting, and it is very kind of you to share your data like that. Will review!
- dredmorbius 9mo ago
  Though I'd think that you'd want to weight unaffiliated sites' anchor text to a given URL much higher than an affiliated site. "Affiliation" is a tricky term itself. Content farms were popular in the aughts (though they seem to have largely subsided), firms such as Claria and Gator. There are chumboxes (Outbrain, Taboola), and of course affiliate links (e.g., to Amazon or other shopping sites). SEO manipulation is its own whole universe. (I'm sure you know far more about this than I do, I'm mostly talking at other readers, and maybe hoping to glean some more wisdom from you ;-)
  - marginalia_nu 9mo ago
    Oh yeah, there's definitely room for improvement in that general direction. Indexing anchor texts is much better than page rank, but in isolation, it's not sufficient. I've also seen some benefit fingerpinting the network traffic the websites make using a headless browser, to identify which ad networks they load. Very few spam sites have no ads, since there wouldn't be any economy in that. e.g. https://marginalia-search.com/site/www.salon.com?view=traffic https://marginalia-search.com/site/www.salon.com?view=traffi... The full data set of DOM samples + recorded network traffic are in an enormous sqlite file (400GB+), and I haven't yet worked out any way of distributing the data yet. Though it's in the back of my mind as something I'd like to solve.
    - dredmorbius 9mo ago
      Oh, that is clever! I'd also suspect that there are networks / links which are more likely signs of low-value content than others. Off the top of my head, crypto, MLM, known scam/fraud sites, and perhaps share links to certain social networks might be negative indicators.
      - 9mo ago
snowwrestler 9mo ago
Google’s biggest search signal now is aggregate behavioral data reported from Chrome. That pervasive behavioral surveillance is the main reason Apple has never allowed a native Chrome app on iOS. It’s also why it is so hard to compete with Google. You guys are talking about techniques for analyzing the corpus of the search index. Google does that and has a direct view into how millions of people interact with it.
- saltysalt 9mo ago
  Yes indeed, they have an impossibly deep moat and deeper pockets. I'm certainly not trying to compete with them with my little side project, it's just for fun!
- xnx 9mo ago
  > That pervasive behavioral surveillance is the main reason Apple has never allowed a native Chrome app on iOS The Chrome iOS app still knows every url visited, duration, scroll depth, etc.
- danans 9mo ago
  > That pervasive behavioral surveillance is the main reason Apple has never allowed a native Chrome app on iOS. There is a native Chrome app on iOS. It gets all the same url visit data as Chrome on other platforms. Apple blocks 3rd party renderers and JS engines on iOS to protect its App Store from competition that might deliver software and content through other channels that they don't take a cut of.