3 ms·

> In today's internet this makes so much more sense than everything being run by the same company I have expressed similar ideas in other discussions here on H

> In today's internet this makes so much more sense than everything being run by the same company I have expressed similar ideas in other discussions here on HN, so I agree with you that this would be a really neat to have an ecosystem like this. But the question is how do you make money? What is your incentive to run a crawler? What is your incentive to run an indexer? People are used to the idea that web search is free, so you search sites cannot charge people money. It also means that they cannot buy indexer services.

iovrthoughtthis 7y ago
What about open, distributed indexes instead of companies? Sites that wished to appear in searches would have an incentive to get added to an index. That could create a market for compute to calculate and update indexes.
dcolkitt 7y ago
I think it could work with micro transactions on the backend. In B2C Internet consumers won't pay for content. But in B2B its the norm. Paying for an indexer or crawler is really no different than the umpteen million AWS transactions that occur everyday. Most consumers probably won't want to pay for search. At least initially until the culture changes. But that's fine, they don't have to access the crawler or indexers directly. Instead they can point their browser at the frontend URL of their choice. And that frontend can monetize by injecting ads or whatever other business model they want. As long as they earn more in ads than they pay on the open market for backend crawlers, indexers, etc., then they're a viable business. Given how ridiculously profitable Google is, this shouldn't be a difficult hurdle to clear.
- MadWombat 7y ago
  The requirements in infrastructure are not symmetrical here. An indexer requires a significantly bigger investment in both storage and computational power than either a crawler or an end-user site. Seems like everyone would want to run a crawler, but nobody would want to run an indexer. Also, it is unclear how to start something like this. A sort of a chicken and egg problem where you cannot start running any of these components because there is nobody running the other parts.
  - drmabus 7y ago
    the egg came first. on a serious note, yacy is good enough to crawl a bunch of sites you like and get much superior search results be it limited to your own interest. google is complete garbage. specially nowadays google just returns unrelated cnn pages. try search for corona virus map, I get tons of news articles talking about the same page but the page is not in the results. whatever you add to the query it keeps returning the same news articles from the same sources. if you populate yacy with pages about the topic you will yield real results.
    - drivebycomment 7y ago
      I just tried "coronavirus map" and it showed maps or an article with map from WP, CNN, NYT. And one from CDC. That looks reasonable to me. Edit: checking the result again, the first page also has a europe.eu page with a map, and another from arcgis.com with a map. So most of the first page results seem highly relevant and good quality.
    - MadWombat 7y ago
      So far, Google search results are far better than everything else I tried. Granted, I haven't tried yacy, but I doubt it would have the type of data aggregation that makes Google results what they are.
- EGreg 7y ago
  Something like this maybe? https://qbix.com/token https://qbix.com/token
notduncansmith 7y ago
It’s also difficult to trust decentralized crawling infrastructure. How do you know that crawlers are reporting honestly?
- MadWombat 7y ago
  Hopefully this could be market regulated, if a particular crawler gets caught making things up or doing otherwise a bad job crawling, indexers could simply stop using it in favor of a different crawler.

iovrthoughtthis 7y ago

What about open, distributed indexes instead of companies? Sites that wished to appear in searches would have an incentive to get added to an index. That could create a market for compute to calculate and update indexes.

dcolkitt 7y ago

I think it could work with micro transactions on the backend. In B2C Internet consumers won't pay for content. But in B2B its the norm. Paying for an indexer or crawler is really no different than the umpteen million AWS transactions that occur everyday. Most consumers probably won't want to pay for search. At least initially until the culture changes. But that's fine, they don't have to access the crawler or indexers directly. Instead they can point their browser at the frontend URL of their choice. And that frontend can monetize by injecting ads or whatever other business model they want. As long as they earn more in ads than they pay on the open market for backend crawlers, indexers, etc., then they're a viable business. Given how ridiculously profitable Google is, this shouldn't be a difficult hurdle to clear.

MadWombat 7y ago
The requirements in infrastructure are not symmetrical here. An indexer requires a significantly bigger investment in both storage and computational power than either a crawler or an end-user site. Seems like everyone would want to run a crawler, but nobody would want to run an indexer. Also, it is unclear how to start something like this. A sort of a chicken and egg problem where you cannot start running any of these components because there is nobody running the other parts.
- drmabus 7y ago
  the egg came first. on a serious note, yacy is good enough to crawl a bunch of sites you like and get much superior search results be it limited to your own interest. google is complete garbage. specially nowadays google just returns unrelated cnn pages. try search for corona virus map, I get tons of news articles talking about the same page but the page is not in the results. whatever you add to the query it keeps returning the same news articles from the same sources. if you populate yacy with pages about the topic you will yield real results.
  - drivebycomment 7y ago
    I just tried "coronavirus map" and it showed maps or an article with map from WP, CNN, NYT. And one from CDC. That looks reasonable to me. Edit: checking the result again, the first page also has a europe.eu page with a map, and another from arcgis.com with a map. So most of the first page results seem highly relevant and good quality.
  - MadWombat 7y ago
    So far, Google search results are far better than everything else I tried. Granted, I haven't tried yacy, but I doubt it would have the type of data aggregation that makes Google results what they are.
EGreg 7y ago
Something like this maybe? https://qbix.com/token https://qbix.com/token

notduncansmith 7y ago

It’s also difficult to trust decentralized crawling infrastructure. How do you know that crawlers are reporting honestly?

MadWombat 7y ago
Hopefully this could be market regulated, if a particular crawler gets caught making things up or doing otherwise a bad job crawling, indexers could simply stop using it in favor of a different crawler.