6 ms·

The real game-changer in search would be if companies would agree to publish indexes of their own sites in an open standard to a place that everyone could acces

The real game-changer in search would be if companies would agree to publish indexes of their own sites in an open standard to a place that everyone could access. This would undercut the monopoly power that large search engines have and allow everyone to focus on innovating the best way to search that content vs. having to spend so much time and money to crawl and index it.

TheFerridge 4y ago
I believe that is precisely what the project is aiming to do, and to turn it into a public resource.
wizofaus 4y ago
I suspect you underestimate how much of the power of search engines is being able to interpret search queries and figure out what a user is really looking for. Even if there were a public, standardised up-to-date high performance full-text index of the entire web freely available I'm willing to bet Google search would be a useful value-add in its ability to answer natural language queries.
- rgrieselhuber 4y ago
  I run an SEO platform SaaS, so I'm familiar. :)
- mellavora 4y ago
  > you underestimate how much of the power of search engines is being able to interpret search queries and figure out what a user is really looking for. So you mean that a search engine is supposed to ignore what you are asking for and instead give you what it thinks you really meant?
  - wizofaus 4y ago
    If you search for [what year was queen Elizabeth born], Google doesn't return pages that have that phrase in it, rather it returns pages that have an answer to that question. You could call that "ignoring" if you like, but it's what 99% of users expect.
- denton-scratch 4y ago
  I resort to "natural language queries" only in desperation, when queries that are lists of search-terms have failed. Actually, they aren't really natural language queries. They are just ordered lists of search-terms. Goo provides no mechanism for saying "This is an english-language question". And even if Goo could parse my natural language, and rephrase it as something like "Are you looking for a list of books published by Douglas Hofstadter?", when you turn that into a query on the index, it stops having anything to do with natural language.
  - wizofaus 4y ago
    A "dumb" search that just took a search phrase like "books published by Douglas Hofstadter" is going to return pages that have that phrase in it, or something close to that phrase. Google will prioritise results that actually contain such lists, regardless of whether the page contains a phrase like that (e.g. the word "published" is basically ignored by Google). That's all I meant.
bobajeff 4y ago
One problem with that is now you have to trust the websites to give an accurate index of their content.
- rgrieselhuber 4y ago
  Indeed
- jeffbee 4y ago
  Anyone who thinks this will work has never tried to index a site. A huge amount of effort is spent trying to figure out if the site is serving different content to users vs crawlers, or if the site is coded to appear visually different to humans vs machines. If you ask sites to index themselves you will get lies only.
  - rgrieselhuber 4y ago
    I index sites all the time and I think it could work. There will be other problems, of course, but we already are partly there with XML sitemaps. Relying on the large search engines to enforce “honesty” from websites puts them into a mediator role that has a number of negative effects both for search in general and, increasingly, society at large.
    - kittiepryde 4y ago
      Relying on sites to be honest about themselves, is even less likely. There are monetary incentives for many of them not to do that. Many sites host dishonest and clickbait content with extreme levels of SEO already. The cost of dishonesty decreases if you can directly modify the index.
      - rgrieselhuber 4y ago
        I think that is primarily a symptom of the fact that we have a bottleneck on search interface providers. If it were easier / cheaper for new search engines / rankers to exist in the market, they could fairly easily filter out unscrupulous domains.
rrwo 4y ago
There are already sitemaps, and pages used structured data like HTML5/ARIA roles, RDF or JSON+LD to provide some semantic annotations. I'd rather that web robots use this information to build useful indexes than to have to worry about generating yet another feed in the hopes that it helps people find my content in a search engine. Besides, a web robot can determine how much other sites link to my content and help determine its overall ranking in results. Adding another type of index file to my site will do nothing to determine how it relates to other sites.
- rgrieselhuber 4y ago
  The structured data on sites, unfortunately, still requires a crawler to index that content, which serves as a barrier for search engine startups. At a minimum, adding some metadata content to XML sitemaps would go a long way to solving some of this problem (title, meta description, content summary, even structured data to the sitemaps).
  - Eduard 4y ago
    What's the problem of using any of the many free webcrawler (libraries) available to crawl a website (even if solely based on the pages advertised by sitemap.xml / robots.txt-announced sitemaps), then extract structured data from these pages? I don't see this as a barrier unique to startups.
    - rgrieselhuber 4y ago
      It's easy to do for small sets of sites, but try doing this at web-scale and you quickly run into a large financial barrier. It's not about technical feasibility as much as it is cost.
  - DocTomoe 4y ago
    Deep down in my soul, the long-locked-away SEO of my money-hustling youth just grinned in anticipation. We have had embedded metadata in websites for decades. In the beginning, Search Engines did even use them. Until someone started stuffing unrelated keywords in it to rank higher.
arjenpdevries 4y ago
We will explore that idea in the project, I also think it may help (but vulnerable for Web index spam by adversary parties).
- rgrieselhuber 4y ago
  That is indeed the biggest problem but maybe something that can be more effectively dealt with downstream by the content rankers and potentially even the user base / custom search algorithm builders. Brave's Goggles project is a good early prototype of this concept.
boyter 4y ago
I’d rather see them publish a federated search of their own content.
- rgrieselhuber 4y ago
  Your comment prompted me to check out Searchcode, looks very interesting. How would the federated search model work in this example? Instead of you having to index the various code repositories, they would index themselves and make their search of those indexes available via a federated API?
_Algernon_ 4y ago
People would abuse that for SEO purposes within seconds.
- rgrieselhuber 4y ago
  The market need would then be shifted to the best search interfaces instead of who has the most money to build the biggest index. A much better focus, IMO.
freediver 4y ago
Standard for this already exists [1] but it does not solve the problems of 1. Implementation (sites do not need to have a sitemap; or those that have it, may not have an accurate one) 2. Discoverability (finding sites in the first place, you'll need a centralised directory of all sites; or resort back to crawling in which case sitemaps are not needed) 3. Ranking (biggest problem in creating a search engine) [1] https://www.sitemaps.org/protocol.html https://www.sitemaps.org/protocol.html
- rgrieselhuber 4y ago
  The sitemaps standard (if this is the basis) would need to be expanded to support additional metadata / structured data to support this idea. 1. This would be up to sites, to your point, major question would be best way to create incentives. 2. This is solvable via a number of approaches, but the search engines themselves would be mostly responsible for finding the right approach for their business. I know how I would do it. 3. Indeed, which would be the main point of this decentralization, to let search engines focus on their hardest problem. Edit: would Kagi not benefit from having to worry about crawling / indexing sites?
  - deleted 4y ago
    [deleted]
  - freediver 4y ago
    > would Kagi not benefit from having to worry about crawling / indexing sites? It would, but sitemaps do not provide that function as we discussed above. However if EU Open Web Search succeeded, that is something we could probably use to some extent.
    - lexx 4y ago
      Or use to extend
spookthesunset 4y ago
I'm pretty sure we tried that way back in the day with <meta name="keywords" content="spam spam spam spam">. People would stuff that with every word in the english language. Older search engines that used those keywords returned some pretty awful results. You simply can't trust sites, who have a strong incentive to get to the top of SEO rankings, to not lie. In fact, given at least one of your competitors will stuff their keywords to get to the top you'll have to do it too. It would become an arms race for who can stuff the most garbage into their indexes to "win". It just doesn't work. All search engines that attempt to be useful will have to filter out the junk. You just have to trust that the search engine you are using isn't withholding results from you that it considers "bad" (eg: "misinformation" (i.e. stuff somebody disagrees with)). And to me, that is the crux of the debate really. Nobody wants spam for search results--everybody agrees with that and there is no real debate about filtering that crap out. The argument really is should a very large company that has a huge market share get to decide what constitutes "fact" and what is "misinformation". Based on 2.5 years of experience so far, what was once deemed "misinformation" has a sneaky way of becoming "factual information". Labeling and hiding "misinformation" because it goes against some narrative pushed by incredibly powerful entities is very scary and there was a hell of a lot of exactly that going on during this covid crap. I used to fall on the side of "private companies can do whatever they want" but now I'm not so sure. Companies like FB, Twitter or Google play a huge role in shaping politics and society. I'm no longer convinced it is okay to let them play the role of "fact checker" or anything like that. Filtering spam is one thing, but hiding "misinformation" is entirely different.
- rgrieselhuber 4y ago
  Your last point is also the one (aside from the economics) I am the most interested in. I think we live in a world now where we are so used to a few tech giants mediating everything for us that we can't even imagine other solutions to this problem, but it's also how we got to this point in the first place.
- closedloop129 4y ago
  >You simply can't trust sites, who have a strong incentive to get to the top of SEO rankings Why is it not enough to punish sites that abuse the keywords?

TheFerridge 4y ago

I believe that is precisely what the project is aiming to do, and to turn it into a public resource.

wizofaus 4y ago

I suspect you underestimate how much of the power of search engines is being able to interpret search queries and figure out what a user is really looking for. Even if there were a public, standardised up-to-date high performance full-text index of the entire web freely available I'm willing to bet Google search would be a useful value-add in its ability to answer natural language queries.

rgrieselhuber 4y ago
I run an SEO platform SaaS, so I'm familiar. :)
mellavora 4y ago
> you underestimate how much of the power of search engines is being able to interpret search queries and figure out what a user is really looking for. So you mean that a search engine is supposed to ignore what you are asking for and instead give you what it thinks you really meant?
- wizofaus 4y ago
  If you search for [what year was queen Elizabeth born], Google doesn't return pages that have that phrase in it, rather it returns pages that have an answer to that question. You could call that "ignoring" if you like, but it's what 99% of users expect.
denton-scratch 4y ago
I resort to "natural language queries" only in desperation, when queries that are lists of search-terms have failed. Actually, they aren't really natural language queries. They are just ordered lists of search-terms. Goo provides no mechanism for saying "This is an english-language question". And even if Goo could parse my natural language, and rephrase it as something like "Are you looking for a list of books published by Douglas Hofstadter?", when you turn that into a query on the index, it stops having anything to do with natural language.
- wizofaus 4y ago
  A "dumb" search that just took a search phrase like "books published by Douglas Hofstadter" is going to return pages that have that phrase in it, or something close to that phrase. Google will prioritise results that actually contain such lists, regardless of whether the page contains a phrase like that (e.g. the word "published" is basically ignored by Google). That's all I meant.

bobajeff 4y ago

One problem with that is now you have to trust the websites to give an accurate index of their content.

rgrieselhuber 4y ago
Indeed
jeffbee 4y ago
Anyone who thinks this will work has never tried to index a site. A huge amount of effort is spent trying to figure out if the site is serving different content to users vs crawlers, or if the site is coded to appear visually different to humans vs machines. If you ask sites to index themselves you will get lies only.
- rgrieselhuber 4y ago
  I index sites all the time and I think it could work. There will be other problems, of course, but we already are partly there with XML sitemaps. Relying on the large search engines to enforce “honesty” from websites puts them into a mediator role that has a number of negative effects both for search in general and, increasingly, society at large.
  - kittiepryde 4y ago
    Relying on sites to be honest about themselves, is even less likely. There are monetary incentives for many of them not to do that. Many sites host dishonest and clickbait content with extreme levels of SEO already. The cost of dishonesty decreases if you can directly modify the index.
    - rgrieselhuber 4y ago
      I think that is primarily a symptom of the fact that we have a bottleneck on search interface providers. If it were easier / cheaper for new search engines / rankers to exist in the market, they could fairly easily filter out unscrupulous domains.

rrwo 4y ago

There are already sitemaps, and pages used structured data like HTML5/ARIA roles, RDF or JSON+LD to provide some semantic annotations. I'd rather that web robots use this information to build useful indexes than to have to worry about generating yet another feed in the hopes that it helps people find my content in a search engine. Besides, a web robot can determine how much other sites link to my content and help determine its overall ranking in results. Adding another type of index file to my site will do nothing to determine how it relates to other sites.

rgrieselhuber 4y ago
The structured data on sites, unfortunately, still requires a crawler to index that content, which serves as a barrier for search engine startups. At a minimum, adding some metadata content to XML sitemaps would go a long way to solving some of this problem (title, meta description, content summary, even structured data to the sitemaps).
- Eduard 4y ago
  What's the problem of using any of the many free webcrawler (libraries) available to crawl a website (even if solely based on the pages advertised by sitemap.xml / robots.txt-announced sitemaps), then extract structured data from these pages? I don't see this as a barrier unique to startups.
  - rgrieselhuber 4y ago
    It's easy to do for small sets of sites, but try doing this at web-scale and you quickly run into a large financial barrier. It's not about technical feasibility as much as it is cost.
- DocTomoe 4y ago
  Deep down in my soul, the long-locked-away SEO of my money-hustling youth just grinned in anticipation. We have had embedded metadata in websites for decades. In the beginning, Search Engines did even use them. Until someone started stuffing unrelated keywords in it to rank higher.

arjenpdevries 4y ago

We will explore that idea in the project, I also think it may help (but vulnerable for Web index spam by adversary parties).

rgrieselhuber 4y ago
That is indeed the biggest problem but maybe something that can be more effectively dealt with downstream by the content rankers and potentially even the user base / custom search algorithm builders. Brave's Goggles project is a good early prototype of this concept.

boyter 4y ago

I’d rather see them publish a federated search of their own content.

rgrieselhuber 4y ago
Your comment prompted me to check out Searchcode, looks very interesting. How would the federated search model work in this example? Instead of you having to index the various code repositories, they would index themselves and make their search of those indexes available via a federated API?

_Algernon_ 4y ago

People would abuse that for SEO purposes within seconds.

rgrieselhuber 4y ago
The market need would then be shifted to the best search interfaces instead of who has the most money to build the biggest index. A much better focus, IMO.

freediver 4y ago

Standard for this already exists [1] but it does not solve the problems of 1. Implementation (sites do not need to have a sitemap; or those that have it, may not have an accurate one) 2. Discoverability (finding sites in the first place, you'll need a centralised directory of all sites; or resort back to crawling in which case sitemaps are not needed) 3. Ranking (biggest problem in creating a search engine) [1] https://www.sitemaps.org/protocol.html https://www.sitemaps.org/protocol.html

rgrieselhuber 4y ago
The sitemaps standard (if this is the basis) would need to be expanded to support additional metadata / structured data to support this idea. 1. This would be up to sites, to your point, major question would be best way to create incentives. 2. This is solvable via a number of approaches, but the search engines themselves would be mostly responsible for finding the right approach for their business. I know how I would do it. 3. Indeed, which would be the main point of this decentralization, to let search engines focus on their hardest problem. Edit: would Kagi not benefit from having to worry about crawling / indexing sites?
- deleted 4y ago
  [deleted]
- freediver 4y ago
  > would Kagi not benefit from having to worry about crawling / indexing sites? It would, but sitemaps do not provide that function as we discussed above. However if EU Open Web Search succeeded, that is something we could probably use to some extent.
  - lexx 4y ago
    Or use to extend

spookthesunset 4y ago

I'm pretty sure we tried that way back in the day with <meta name="keywords" content="spam spam spam spam">. People would stuff that with every word in the english language. Older search engines that used those keywords returned some pretty awful results. You simply can't trust sites, who have a strong incentive to get to the top of SEO rankings, to not lie. In fact, given at least one of your competitors will stuff their keywords to get to the top you'll have to do it too. It would become an arms race for who can stuff the most garbage into their indexes to "win". It just doesn't work. All search engines that attempt to be useful will have to filter out the junk. You just have to trust that the search engine you are using isn't withholding results from you that it considers "bad" (eg: "misinformation" (i.e. stuff somebody disagrees with)). And to me, that is the crux of the debate really. Nobody wants spam for search results--everybody agrees with that and there is no real debate about filtering that crap out. The argument really is should a very large company that has a huge market share get to decide what constitutes "fact" and what is "misinformation". Based on 2.5 years of experience so far, what was once deemed "misinformation" has a sneaky way of becoming "factual information". Labeling and hiding "misinformation" because it goes against some narrative pushed by incredibly powerful entities is very scary and there was a hell of a lot of exactly that going on during this covid crap. I used to fall on the side of "private companies can do whatever they want" but now I'm not so sure. Companies like FB, Twitter or Google play a huge role in shaping politics and society. I'm no longer convinced it is okay to let them play the role of "fact checker" or anything like that. Filtering spam is one thing, but hiding "misinformation" is entirely different.

rgrieselhuber 4y ago
Your last point is also the one (aside from the economics) I am the most interested in. I think we live in a world now where we are so used to a few tech giants mediating everything for us that we can't even imagine other solutions to this problem, but it's also how we got to this point in the first place.
closedloop129 4y ago
>You simply can't trust sites, who have a strong incentive to get to the top of SEO rankings Why is it not enough to punish sites that abuse the keywords?