5 ms·

> A new breakthrough heuristic today will look something totally different, just as meritocratic and possibly resistant to gaming. I wonder how much of this co

> A new breakthrough heuristic today will look something totally different, just as meritocratic and possibly resistant to gaming. I wonder how much of this could be obtained back by penalizing: 1. The number of javascript dependencies 2. The number of ads on the page, or the depth of the ad network This might start a virtuous circle, but in the end, this is just a game of cat-and-mouse, and website might optimize for this as well. What we might need to break this is a variety of search engines that uses different criteria to rank pages. I suspect it would be pretty hard, if not impossible, to optimize for all of them. And in any case, frequently change the ranking algorithms to combat over-optimization by the websites (as that's classically done against ossification for protocols, or any overfitting to outside forces in a competitive system).

derefr 7y ago
You could even have all this under one roof: one common search spider that feeds this ensemble of different ranking algorithms to produce a set of indices, and then a search engine front end that round-robins queries out between the different indices. (Don’t like your query? Spin the algorithm wheel! “I’m Feeling Lucky” indeed.)
- zozbot234 7y ago
  The Common Crawl is a thing already. Unfortunately, a "full" text crawl of the internets is a YUUUGE amount of data to manage, and I can't think of anything that could change that in the foreseeable future. That's why I think providing a federated Web directory standard, ala ODP/DMOZ except not limited to a single source, would be a far more impactful development.
  - reaperducer 7y ago
    Unfortunately, a "full" text crawl of the internets is a YUUUGE amount of data to manage Maybe instead of a problem, there is an opportunity here. Back before Google ate the intarwebs, there used to be niche search engines. Perhaps that is an idea whose time has come again. For example, if I want information from a government source, I use a search engine that specializes in crawling only government web sites. If I want information about Berlin, I use a search engine that only crawls web sites with information about Berlin, or that are located in Berlin. If I want information about health, I use a search engine that only crawls medical web sites. Each topic is still a wealth of information, but siloed enough that the amount of data could be manageable to a small or medium-sized company. And the market would keep the niches from getting so small that they become useful. A search engine dedicated to Hello Kitty lanyards isn't going to monetize.
    - zozbot234 7y ago
      That's the problem that web directories solve. It's not that you're wrong, it's just largely orthogonal to the problem that you'd need a large crawl of the internets for, i.e. spotting sites about X niche that you wouldn't find even from other directly-related sites, and that are too obscure, new, etc. to be linked in any web directory.
      - reaperducer 7y ago
        That's the problem that web directories solve Not really. A web directory is a directory of web sites. I can't search a web directory for content within the web sites, which is what a niche search engine would do.
        Beldin 7y ago
        On the other hand, the niche search engine depends upon having such a web directory (the list of sites to index).
        teddyh 7y ago
        Like WAIS? https://en.wikipedia.org/wiki/Wide_area_information_server https://en.wikipedia.org/wiki/Wide_area_information_server
    - HomeDeLaPot 7y ago
      Don't forget the search engine search engine!
    - LargoLasskhyfv 7y ago
      I´d be happy with something like Searx [1,2,3] [1] https://en.wikipedia.org/wiki/Searx https://en.wikipedia.org/wiki/Searx [2] https://asciimoo.github.io/searx/ https://asciimoo.github.io/searx/ [3] https://stats.searx.xyz/ https://stats.searx.xyz/ featuring the semantic map of [4] https://swisscows.ch/ https://swisscows.ch/ incorporating [5] https://curlie.org/ https://curlie.org/ and Wikipedia and something like Yelp/YellowPages embedded in Open Streetmaps for businesses and points of interest, with a no frills interface showing the history (via timeslide?) of edits. Bang! Done!
  - anoncake 7y ago
    Javascript, which Google coincidentally pushed and still pushes for, doesn't exactly make the web easier to crawl either.
  - chongli 7y ago
    You don’t really need to store a full text crawl if you’re going to be penalizing or blacklisting all of the ad-filled SEO junk sites. If your algorithm scores the site below a certain threshold then flag it as junk and store only a hash of the page. Another potentially useful approach is to construct a graph database of all these sites, with links as edges. If one page gets flagged as junk then you can lower the scores of all other pages within its clique [1]. This could potentially cause a cascade of junk-flagging, cleaning large swathes of these undesirable sites from the index. [1] https://en.wikipedia.org/wiki/Clique_(graph_theory) https://en.wikipedia.org/wiki/Clique_(graph_theory)
- numpad0 7y ago
  What if: SEO consultants aren’t gaming system, but the search and web is being optimized for “measurable immediate economic impact” that is ad revenue at this moment — due to web itself being in-monetizable and unable to generate value. I don’t like the whole concept of SEO, I don’t like the way the web is today, but I think we should stop and think before resorting to “immoral few is destroying things, we unfuck it reclaim what we deserve” type simplification.
  - majewsky 7y ago
    > the search and web is being optimized for “measurable immediate economic impact” that is ad revenue at this moment So much is obvious. The discussion is about whether there is a less shitty metric.
aaron-santos 7y ago
Merging js deps into one big resource isn't difficult. The number of ads point is interesting though. How would one determine what is an ad and what is an image? I have my ideas, but optimizing on this boundary sounds like it would lead to weird outcomes.
- zozbot234 7y ago
  Adblockers have to solve that problem already. And it's actually really easy because "ads" aren't just ads unfortunately, they're also third-party code that's trying to track you as you browse the site. So it's reasonably easy to spot them and filter them out.
  - thrwaway69 7y ago
    Though, advertisers are already using first party redirection. Future of adblockers is bleak. https://github.com/uBlockOrigin/uBlock-issues/issues/780/ https://github.com/uBlockOrigin/uBlock-issues/issues/780/
    - dredmorbius 7y ago
      Back in the early days of banner ads, a CSS-based approach to blocking was to target images by size. Since advertising revolved around specific standards of advertising "units" (effectively: sizes of images), those could be identified and blocked. That worked well, for a time. This is ultimately whack-a-mole. For the past decade or so, point-of-origin based blockers have worked effectively, because that's how advertising networks have operated. If the ad targets start getting unified, we may have to switch to other signatures: - Again, sizes of images or DOM elements. - Content matching known hash signatures, or constant across multiple requests to a site (other than known branding elements / graphics). - "Things that behave like ads behave" as defined by AI encoded into ad blockers. - CSS / page elements. Perhaps applying whitelist rather than blacklist policies. - User-defined element subtraction. There's little in the history of online advertising that suggests users will simply give up.
      - 7y ago
mathattack 7y ago
Given that Google makes money off the ads, that would be hard. DuckDuckGo could pull it off. You need another revenue stream though.
- MayeulC 7y ago
  Google has taken on so many markets that I don't think they can do anything reasonably well (or disruptive) without conflicting interests. A breakup is overdue: if they didn't control both search and ads, the web would be a lot better nowadays. If they didn't control web browsers as well, standards would be much more important.
Razengan 7y ago
> What we might need to break this is ... Create a core protocol at the same level as DNS etc., that web servers can use to offer an index of everything they serve/relay. A multitude of user-side apps may then query that protocol, with each app using different algorithms, heuristics and offering different options.
- zozbot234 7y ago
  > Create a core protocol at the same level as DNS etc., that web servers can use to offer an index of everything they serve/relay. So, back to gopher? That might actually work!
- dredmorbius 7y ago
  I've been thinking along similar lines for a year or so now. There are several puzzling omissions from Web standards, particularly given that keyword-based search was part of the original CERN WWW discussion: http://info.cern.ch/hypertext/WWW/Addressing/Search.html http://info.cern.ch/hypertext/WWW/Addressing/Search.html IF we had a distributable search protocol, index, and infrastructure ... the entire online landscape might look rather different. Note that you'd likely need some level of client support for this. And the world's leading client developer has a strongly-motivated incentive to NOT provide this functionality integrally. A distributed self-provided search would also have numerous issues -- false or misleading results (keyword stuffing, etc.) would be harder to vet than the present situation. Which suggests that some form of vetting / verifying provided indices would be required. Even a provided-index model would still require a reputational (ranking) mechanism. Arguably, Google's biggest innovation wasn't spidering, but ranking. The problem now is that Google's ranking ... both doesn't work, and incentivises behaviours strongly opposed to user interests. Penalising abusive practices has to be built into the system, with those penalties being rapid, effective, and for repeat offenders, highly durable. The problem of potential for third-party malfeasance -- e.g., engaging in behaviours appearing to favour one site, but performed to harm that site's reputation through black-hat SEO penalties, also has to be considered. As a user, the one thing I'd most like to be able to do is specify blacklists of sites / domains I never want to have appear in my search results. Without having to log in to a search provider and leave a "personalised" record of what those sites are. (Some form of truly anonymised aggregation of such blocklists would, of course, be of some use, and facilitating this is an interesting challenge.)
hombre_fatal 7y ago
The problem with this kind of unrelated disciplinary action, like punishing sites for having too much javascript, is that you're losing sight of the goal of a search engine: helping people find the content that they are looking for. I'll take the website that has exactly what I'm looking for but loads in 30 seconds over an irrelevant website that loads instantly. I suppose these factors can help when you need to tie-break information super-commodities like a search for "dogs" where all other potential tie-breakers have already been observed with no affect, but nothing more.
biggestdecision 7y ago
Google already penalizes based off payload size/download speed.
frandroid 7y ago
> 1. The number of javascript dependencies How about we don't start looking at the /how/ a site is made, when it's already difficult to sort out the /what/ it is.

derefr 7y ago

You could even have all this under one roof: one common search spider that feeds this ensemble of different ranking algorithms to produce a set of indices, and then a search engine front end that round-robins queries out between the different indices. (Don’t like your query? Spin the algorithm wheel! “I’m Feeling Lucky” indeed.)

zozbot234 7y ago
The Common Crawl is a thing already. Unfortunately, a "full" text crawl of the internets is a YUUUGE amount of data to manage, and I can't think of anything that could change that in the foreseeable future. That's why I think providing a federated Web directory standard, ala ODP/DMOZ except not limited to a single source, would be a far more impactful development.
- reaperducer 7y ago
  Unfortunately, a "full" text crawl of the internets is a YUUUGE amount of data to manage Maybe instead of a problem, there is an opportunity here. Back before Google ate the intarwebs, there used to be niche search engines. Perhaps that is an idea whose time has come again. For example, if I want information from a government source, I use a search engine that specializes in crawling only government web sites. If I want information about Berlin, I use a search engine that only crawls web sites with information about Berlin, or that are located in Berlin. If I want information about health, I use a search engine that only crawls medical web sites. Each topic is still a wealth of information, but siloed enough that the amount of data could be manageable to a small or medium-sized company. And the market would keep the niches from getting so small that they become useful. A search engine dedicated to Hello Kitty lanyards isn't going to monetize.
  - zozbot234 7y ago
    That's the problem that web directories solve. It's not that you're wrong, it's just largely orthogonal to the problem that you'd need a large crawl of the internets for, i.e. spotting sites about X niche that you wouldn't find even from other directly-related sites, and that are too obscure, new, etc. to be linked in any web directory.
    - reaperducer 7y ago
      That's the problem that web directories solve Not really. A web directory is a directory of web sites. I can't search a web directory for content within the web sites, which is what a niche search engine would do.
      - Beldin 7y ago
        On the other hand, the niche search engine depends upon having such a web directory (the list of sites to index).
      - teddyh 7y ago
        Like WAIS? https://en.wikipedia.org/wiki/Wide_area_information_server https://en.wikipedia.org/wiki/Wide_area_information_server
  - HomeDeLaPot 7y ago
    Don't forget the search engine search engine!
  - LargoLasskhyfv 7y ago
    I´d be happy with something like Searx [1,2,3] [1] https://en.wikipedia.org/wiki/Searx https://en.wikipedia.org/wiki/Searx [2] https://asciimoo.github.io/searx/ https://asciimoo.github.io/searx/ [3] https://stats.searx.xyz/ https://stats.searx.xyz/ featuring the semantic map of [4] https://swisscows.ch/ https://swisscows.ch/ incorporating [5] https://curlie.org/ https://curlie.org/ and Wikipedia and something like Yelp/YellowPages embedded in Open Streetmaps for businesses and points of interest, with a no frills interface showing the history (via timeslide?) of edits. Bang! Done!
- anoncake 7y ago
  Javascript, which Google coincidentally pushed and still pushes for, doesn't exactly make the web easier to crawl either.
- chongli 7y ago
  You don’t really need to store a full text crawl if you’re going to be penalizing or blacklisting all of the ad-filled SEO junk sites. If your algorithm scores the site below a certain threshold then flag it as junk and store only a hash of the page. Another potentially useful approach is to construct a graph database of all these sites, with links as edges. If one page gets flagged as junk then you can lower the scores of all other pages within its clique [1]. This could potentially cause a cascade of junk-flagging, cleaning large swathes of these undesirable sites from the index. [1] https://en.wikipedia.org/wiki/Clique_(graph_theory) https://en.wikipedia.org/wiki/Clique_(graph_theory)
numpad0 7y ago
What if: SEO consultants aren’t gaming system, but the search and web is being optimized for “measurable immediate economic impact” that is ad revenue at this moment — due to web itself being in-monetizable and unable to generate value. I don’t like the whole concept of SEO, I don’t like the way the web is today, but I think we should stop and think before resorting to “immoral few is destroying things, we unfuck it reclaim what we deserve” type simplification.
- majewsky 7y ago
  > the search and web is being optimized for “measurable immediate economic impact” that is ad revenue at this moment So much is obvious. The discussion is about whether there is a less shitty metric.

aaron-santos 7y ago

Merging js deps into one big resource isn't difficult. The number of ads point is interesting though. How would one determine what is an ad and what is an image? I have my ideas, but optimizing on this boundary sounds like it would lead to weird outcomes.

zozbot234 7y ago
Adblockers have to solve that problem already. And it's actually really easy because "ads" aren't just ads unfortunately, they're also third-party code that's trying to track you as you browse the site. So it's reasonably easy to spot them and filter them out.
- thrwaway69 7y ago
  Though, advertisers are already using first party redirection. Future of adblockers is bleak. https://github.com/uBlockOrigin/uBlock-issues/issues/780/ https://github.com/uBlockOrigin/uBlock-issues/issues/780/
  - dredmorbius 7y ago
    Back in the early days of banner ads, a CSS-based approach to blocking was to target images by size. Since advertising revolved around specific standards of advertising "units" (effectively: sizes of images), those could be identified and blocked. That worked well, for a time. This is ultimately whack-a-mole. For the past decade or so, point-of-origin based blockers have worked effectively, because that's how advertising networks have operated. If the ad targets start getting unified, we may have to switch to other signatures: - Again, sizes of images or DOM elements. - Content matching known hash signatures, or constant across multiple requests to a site (other than known branding elements / graphics). - "Things that behave like ads behave" as defined by AI encoded into ad blockers. - CSS / page elements. Perhaps applying whitelist rather than blacklist policies. - User-defined element subtraction. There's little in the history of online advertising that suggests users will simply give up.
    - 7y ago

mathattack 7y ago

Given that Google makes money off the ads, that would be hard. DuckDuckGo could pull it off. You need another revenue stream though.

MayeulC 7y ago
Google has taken on so many markets that I don't think they can do anything reasonably well (or disruptive) without conflicting interests. A breakup is overdue: if they didn't control both search and ads, the web would be a lot better nowadays. If they didn't control web browsers as well, standards would be much more important.

Razengan 7y ago

> What we might need to break this is ... Create a core protocol at the same level as DNS etc., that web servers can use to offer an index of everything they serve/relay. A multitude of user-side apps may then query that protocol, with each app using different algorithms, heuristics and offering different options.

zozbot234 7y ago
> Create a core protocol at the same level as DNS etc., that web servers can use to offer an index of everything they serve/relay. So, back to gopher? That might actually work!
dredmorbius 7y ago
I've been thinking along similar lines for a year or so now. There are several puzzling omissions from Web standards, particularly given that keyword-based search was part of the original CERN WWW discussion: http://info.cern.ch/hypertext/WWW/Addressing/Search.html http://info.cern.ch/hypertext/WWW/Addressing/Search.html IF we had a distributable search protocol, index, and infrastructure ... the entire online landscape might look rather different. Note that you'd likely need some level of client support for this. And the world's leading client developer has a strongly-motivated incentive to NOT provide this functionality integrally. A distributed self-provided search would also have numerous issues -- false or misleading results (keyword stuffing, etc.) would be harder to vet than the present situation. Which suggests that some form of vetting / verifying provided indices would be required. Even a provided-index model would still require a reputational (ranking) mechanism. Arguably, Google's biggest innovation wasn't spidering, but ranking. The problem now is that Google's ranking ... both doesn't work, and incentivises behaviours strongly opposed to user interests. Penalising abusive practices has to be built into the system, with those penalties being rapid, effective, and for repeat offenders, highly durable. The problem of potential for third-party malfeasance -- e.g., engaging in behaviours appearing to favour one site, but performed to harm that site's reputation through black-hat SEO penalties, also has to be considered. As a user, the one thing I'd most like to be able to do is specify blacklists of sites / domains I never want to have appear in my search results. Without having to log in to a search provider and leave a "personalised" record of what those sites are. (Some form of truly anonymised aggregation of such blocklists would, of course, be of some use, and facilitating this is an interesting challenge.)

hombre_fatal 7y ago

The problem with this kind of unrelated disciplinary action, like punishing sites for having too much javascript, is that you're losing sight of the goal of a search engine: helping people find the content that they are looking for. I'll take the website that has exactly what I'm looking for but loads in 30 seconds over an irrelevant website that loads instantly. I suppose these factors can help when you need to tie-break information super-commodities like a search for "dogs" where all other potential tie-breakers have already been observed with no affect, but nothing more.

biggestdecision 7y ago

Google already penalizes based off payload size/download speed.

frandroid 7y ago

> 1. The number of javascript dependencies How about we don't start looking at the /how/ a site is made, when it's already difficult to sort out the /what/ it is.