6 ms·

One reason search is hard is people are very motivated to game your system. That makes being transparent about how it works a fool's errand. It also isn't clear

by wbl 4y ago

One reason search is hard is people are very motivated to game your system. That makes being transparent about how it works a fool's errand. It also isn't clear how the economic structure works: improving search relevance by x% is tremendously socially valuable, but probably makes Google's bottom line go up by a thousandth of x, and they have a very direct understanding of the connection. Without that money, how will this get the resources to succeed given the adversaries are learning from the people with a lot more?

systemvoltage 4y ago
Exactly, I think the failure of such project is embedded in the incentives. OpenMaps are good but you’re not going to convince Google to make them default on Android. The correct strategy isn’t to hand this over to a giant bureaucracy, but to create an atmosphere where we can have a dozen alternatives to Google.
- cloutchaser 4y ago
  The only way you can beat google is to create a search engine in a small niche where they can’t compete, then eventually over the years expand into general search once they can’t catch up (innovators dilemma basically). The only search product that comes close to this is Amazon search or booking.com (or maybe YouTube). All multi billion dollar businesses. And when smaller ones emerge like the flight scanner one, google buys them. I don’t see google going anywhere soon. They have a lot of faults but they’re still too good.
  - supermatt 4y ago
    > The only search product that comes close to this is Amazon search or booking.com (or maybe YouTube). What is so great about those search examples?
    - aydyn 4y ago
      Absolutely nothing, Amazon search is horrendous. Search by price is completely broken (and has been for years). Filter by exact phrase is non-existent. No way to search products by size, and on and on.
  - systemvoltage 4y ago
    This is HN and we're not even pretending to be optimistic about beating incumbents, but it is much deeper than that - there is also a profound sense of defeatism directed towards European entrepeurs as well as the world at large. It is an indictment of the current nihilistic zeitgeist.
    - gnz11 4y ago
      Well, it is a forum run by a Silicon Valley VC firm … so of course there will be a bit of “only SV can do anything interesting in tech” gatekeeping going on.
      - systemvoltage 4y ago
        Idk, even people here in SV has given up.
  - newaccount74 4y ago
    In Austria there is "geizhals.at" [1], which is a price comparison website for electronics, and they have the best parametric search engine I have ever seen. It's 1000x better than Amazon search. However, there's no way Geizhals will ever expand into general search. The reason why they are so much better than the competition is that they are focussing on a small, profitable niche and presumably use manual data entry to ensure they have the best data. [1]: there's a UK version too "skinflint.co.uk"
    - Lio 4y ago
      Just opened skinflint.co.uk to have a look. I was greeted by "Do Not Track detected only storing necessary information". That's wonderful! I don't think I've seen a website that does that before and I think it's worth praising.
    - Akronymus 4y ago
      Geizhals is my go to for anything computer related. Somewhat amusing that such a good site comes out from this little country.
- tyiz 4y ago
  Exactly, neither Google nor Apple will open the gates to their fenced garden. It will always be more convenient for users to use their solution.
- zaarn 4y ago
  This does seem to be the goal of the project. They mention that they want to make a centralized control panel and crawler, where you can control how your website is crawled and when But at the end of the day the resulting dataset would be sold to third parties so they can rank the results appropriately. Which to me seems to be the only sane way forward. Only a government could run something the scale of the Google crawler and succeed at doing so. And then everyone can build search engines on top of that.
  - denton-scratch 4y ago
    > And then everyone can build search engines on top of that. That bit troubles me. If the index is maintained by a government agency, and every search engine is using the same index, then that's a massive censorship avenue. I wonder how "open" Open Web Index is going to be.
    - zaarn 4y ago
      If the index is censored, nothing stops you from adding to the index yourself. The vast majority of the net is, after all, now indexed, so you can run your own indexer to cover whatever they didn't cover.
marginalia_nu 4y ago
This sentiment is common, but is largely based on Google (and similar commercial actors). I'm not convinced it generalizes as much as it seems. It is uniquely profitable to manipulate Google because Google's outlook of the world informs not only its search engine, but its advertisement business. > improving search relevance by x% is tremendously socially valuable, but probably makes Google's bottom line go up by a thousandth of x, and they have a very direct understanding of the connection. I'm not sure this is correct. In a vacuum of real competition, the most profitable ought to be how it is right now, when search results are kind of ambiguously bad, so you need to click and skim through a few results to maybe find what you want, multiplying the number of ad impressions.
- wbl 4y ago
  Being result number 1 not result number 5 matters a lot to companies that are selling things to people who are searching. The value of the ads comes from that property, not the other way around.
  - boltzmann-brain 4y ago
    Indeed, and that's why search relevance is important. We can talk about the Importance Of This until we die of old age. Google failed us, Google is Borg, Yandex is FSB, Bing is ... Bing, etc. However, the fact that there is a problem to be solved and It Is Important doesn't mean that the EU will solve it. If anything, it's just the set-up for another political play that will have damaging consequences to the internet as a whole, just like GDPR, EU's poster-child "internet project". They made GDPR just strong enough legislatively to be annoying, but not strong enough to actually change anything. Companies can still store EU citizens' data anywhere they want and do whatever they want. There's no insight into this. It is an unenforceable law and the only artifact of it existing is that I have to have "I don't care about cookies" installed, so that Avast Antivirus can eventually decide to silently collect and sell on my data. For all intents and purposes OpenWebSearch is most likely not meant to succeed at anything either, and is just going to be a political stepping stone towards legislature that will be awkward and make the internet worse for all users. EU has a long history of creating or hanging onto laws that betray a misunderstanding of how the digital age works, how the internet works, how data can be copied or moved around for free. Here's an example. Every country in Europe has some sort of legal construct in place that will prevent you from secretly recording a conversation you're having and uploading it to the internet. So for example, take Austria. They accomplish it by preventing you from publishing it on the internet. However, they don't prevent you from recording it. There are laws against secretly recording a conversation you are not a part of, but there's no such law for the situation when you are part of the conversation. So you can record and upload, just not publish. However, you can get on a train and go to a different country which has laws that prevent you from recording, but has no laws against publishing. Then you're in the clear. Or you can just use a VPN so that it looks like you uploaded and published it from the other country. Or you can just upload to YouTube, which will not show where the thing was published from - and claim that you did so during your tourist visit to Vietnam or whatever. And if someone brings a civil lawsuit? Good luck trawling the Vietnamese legal system for clues about that specific issue. Hope you know this rare language with virtually no legal experts who at the same time speak Vietnamese and your particular local European language. The costs would be on the order of tens of thousands at least, which is out of reach for anyone but the wealthiest EU citizens. Want more? Egregious copyright related laws are known by everyone. So are dns blocks of shunned sites. Or the recent Austrian project to block cloudflare IPs which literally broke the internet. The fact such laws are still in place from before the internet - or are even still being put out - and are effectively unenforceable while making the internet worse for everyone - makes it clear that the governments in place simply have a misunderstanding of how the internet works. None of that will stop politicians from coming up with BS excuses breaking the world with fever-dream laws in order to push their latest political agenda. The real question is: if such clearly unfit legislature is being put in place for political folly in industries we understand - how much of that is happening in industries we don't understand? Health Care, Food, Agriculture, Civil Engineering, Patent Law, and so on. My guess? You can tell what my guess is. As for Open Web Search, what everyone should really be asking themselves is: ok, so what's the scam that's going to be pulled here?
tremon 4y ago
"One reason democracy is hard is that people are very motivated to game your system. That makes being transparent about how it works a fool's errand." That's a major fallacy you're opening with. If something is important to us (as a society), we will find ways to make it serve us well even when it's under attack. The rest of your post even makes a similar point: the social value of well-working search is greater than the economic value for even the biggest search monopolist on the planet. So why not socialize it?
- xani_ 4y ago
  Well, on one side we have zero examples of open engines actually working well, on other we have long nuanced history of SEO vs search engine fighting unending battle, with SEO side trying to circumvent every way the search engine is trying to stop them from poisoning the search results. And barely edging out on that. Being idealistic about it won't change the outcome.
  - account42 4y ago
    Do we have any examples of search engines not run by ad companies having problems fighting SEO?
    - rjmunro 4y ago
      Altavista was initially a demo of the capabilities of DEC Alpha CPUs. It was initially amazing. It got taken over by SEO rubbish, and beaten by Google who invented PageRank to get good results again. PageRank was initially much less prone to SEO shenanigans because it relied on signals from other sites (incoming links) to decide how important a result was. Of course, as Google became more popular, people started sharing links on other pages and so on to cheat the PageRank algorithm. And Google have been caught in a fight with SEO ever since.
  - simiones 4y ago
    Do we have any example of an open engine working badly? ' As far as I know, this hasn't really been tried, definitely not with the kinds of resources this project will have available.

by wbl 4y ago

systemvoltage 4y ago

Exactly, I think the failure of such project is embedded in the incentives. OpenMaps are good but you’re not going to convince Google to make them default on Android. The correct strategy isn’t to hand this over to a giant bureaucracy, but to create an atmosphere where we can have a dozen alternatives to Google.

cloutchaser 4y ago
The only way you can beat google is to create a search engine in a small niche where they can’t compete, then eventually over the years expand into general search once they can’t catch up (innovators dilemma basically). The only search product that comes close to this is Amazon search or booking.com (or maybe YouTube). All multi billion dollar businesses. And when smaller ones emerge like the flight scanner one, google buys them. I don’t see google going anywhere soon. They have a lot of faults but they’re still too good.
- supermatt 4y ago
  > The only search product that comes close to this is Amazon search or booking.com (or maybe YouTube). What is so great about those search examples?
  - aydyn 4y ago
    Absolutely nothing, Amazon search is horrendous. Search by price is completely broken (and has been for years). Filter by exact phrase is non-existent. No way to search products by size, and on and on.
- systemvoltage 4y ago
  This is HN and we're not even pretending to be optimistic about beating incumbents, but it is much deeper than that - there is also a profound sense of defeatism directed towards European entrepeurs as well as the world at large. It is an indictment of the current nihilistic zeitgeist.
  - gnz11 4y ago
    Well, it is a forum run by a Silicon Valley VC firm … so of course there will be a bit of “only SV can do anything interesting in tech” gatekeeping going on.
    - systemvoltage 4y ago
      Idk, even people here in SV has given up.
- newaccount74 4y ago
  In Austria there is "geizhals.at" [1], which is a price comparison website for electronics, and they have the best parametric search engine I have ever seen. It's 1000x better than Amazon search. However, there's no way Geizhals will ever expand into general search. The reason why they are so much better than the competition is that they are focussing on a small, profitable niche and presumably use manual data entry to ensure they have the best data. [1]: there's a UK version too "skinflint.co.uk"
  - Lio 4y ago
    Just opened skinflint.co.uk to have a look. I was greeted by "Do Not Track detected only storing necessary information". That's wonderful! I don't think I've seen a website that does that before and I think it's worth praising.
  - Akronymus 4y ago
    Geizhals is my go to for anything computer related. Somewhat amusing that such a good site comes out from this little country.
tyiz 4y ago
Exactly, neither Google nor Apple will open the gates to their fenced garden. It will always be more convenient for users to use their solution.
zaarn 4y ago
This does seem to be the goal of the project. They mention that they want to make a centralized control panel and crawler, where you can control how your website is crawled and when But at the end of the day the resulting dataset would be sold to third parties so they can rank the results appropriately. Which to me seems to be the only sane way forward. Only a government could run something the scale of the Google crawler and succeed at doing so. And then everyone can build search engines on top of that.
- denton-scratch 4y ago
  > And then everyone can build search engines on top of that. That bit troubles me. If the index is maintained by a government agency, and every search engine is using the same index, then that's a massive censorship avenue. I wonder how "open" Open Web Index is going to be.
  - zaarn 4y ago
    If the index is censored, nothing stops you from adding to the index yourself. The vast majority of the net is, after all, now indexed, so you can run your own indexer to cover whatever they didn't cover.

marginalia_nu 4y ago

This sentiment is common, but is largely based on Google (and similar commercial actors). I'm not convinced it generalizes as much as it seems. It is uniquely profitable to manipulate Google because Google's outlook of the world informs not only its search engine, but its advertisement business. > improving search relevance by x% is tremendously socially valuable, but probably makes Google's bottom line go up by a thousandth of x, and they have a very direct understanding of the connection. I'm not sure this is correct. In a vacuum of real competition, the most profitable ought to be how it is right now, when search results are kind of ambiguously bad, so you need to click and skim through a few results to maybe find what you want, multiplying the number of ad impressions.

wbl 4y ago
Being result number 1 not result number 5 matters a lot to companies that are selling things to people who are searching. The value of the ads comes from that property, not the other way around.
- boltzmann-brain 4y ago
  Indeed, and that's why search relevance is important. We can talk about the Importance Of This until we die of old age. Google failed us, Google is Borg, Yandex is FSB, Bing is ... Bing, etc. However, the fact that there is a problem to be solved and It Is Important doesn't mean that the EU will solve it. If anything, it's just the set-up for another political play that will have damaging consequences to the internet as a whole, just like GDPR, EU's poster-child "internet project". They made GDPR just strong enough legislatively to be annoying, but not strong enough to actually change anything. Companies can still store EU citizens' data anywhere they want and do whatever they want. There's no insight into this. It is an unenforceable law and the only artifact of it existing is that I have to have "I don't care about cookies" installed, so that Avast Antivirus can eventually decide to silently collect and sell on my data. For all intents and purposes OpenWebSearch is most likely not meant to succeed at anything either, and is just going to be a political stepping stone towards legislature that will be awkward and make the internet worse for all users. EU has a long history of creating or hanging onto laws that betray a misunderstanding of how the digital age works, how the internet works, how data can be copied or moved around for free. Here's an example. Every country in Europe has some sort of legal construct in place that will prevent you from secretly recording a conversation you're having and uploading it to the internet. So for example, take Austria. They accomplish it by preventing you from publishing it on the internet. However, they don't prevent you from recording it. There are laws against secretly recording a conversation you are not a part of, but there's no such law for the situation when you are part of the conversation. So you can record and upload, just not publish. However, you can get on a train and go to a different country which has laws that prevent you from recording, but has no laws against publishing. Then you're in the clear. Or you can just use a VPN so that it looks like you uploaded and published it from the other country. Or you can just upload to YouTube, which will not show where the thing was published from - and claim that you did so during your tourist visit to Vietnam or whatever. And if someone brings a civil lawsuit? Good luck trawling the Vietnamese legal system for clues about that specific issue. Hope you know this rare language with virtually no legal experts who at the same time speak Vietnamese and your particular local European language. The costs would be on the order of tens of thousands at least, which is out of reach for anyone but the wealthiest EU citizens. Want more? Egregious copyright related laws are known by everyone. So are dns blocks of shunned sites. Or the recent Austrian project to block cloudflare IPs which literally broke the internet. The fact such laws are still in place from before the internet - or are even still being put out - and are effectively unenforceable while making the internet worse for everyone - makes it clear that the governments in place simply have a misunderstanding of how the internet works. None of that will stop politicians from coming up with BS excuses breaking the world with fever-dream laws in order to push their latest political agenda. The real question is: if such clearly unfit legislature is being put in place for political folly in industries we understand - how much of that is happening in industries we don't understand? Health Care, Food, Agriculture, Civil Engineering, Patent Law, and so on. My guess? You can tell what my guess is. As for Open Web Search, what everyone should really be asking themselves is: ok, so what's the scam that's going to be pulled here?

tremon 4y ago

"One reason democracy is hard is that people are very motivated to game your system. That makes being transparent about how it works a fool's errand." That's a major fallacy you're opening with. If something is important to us (as a society), we will find ways to make it serve us well even when it's under attack. The rest of your post even makes a similar point: the social value of well-working search is greater than the economic value for even the biggest search monopolist on the planet. So why not socialize it?

xani_ 4y ago
Well, on one side we have zero examples of open engines actually working well, on other we have long nuanced history of SEO vs search engine fighting unending battle, with SEO side trying to circumvent every way the search engine is trying to stop them from poisoning the search results. And barely edging out on that. Being idealistic about it won't change the outcome.
- account42 4y ago
  Do we have any examples of search engines not run by ad companies having problems fighting SEO?
  - rjmunro 4y ago
    Altavista was initially a demo of the capabilities of DEC Alpha CPUs. It was initially amazing. It got taken over by SEO rubbish, and beaten by Google who invented PageRank to get good results again. PageRank was initially much less prone to SEO shenanigans because it relied on signals from other sites (incoming links) to decide how important a result was. Of course, as Google became more popular, people started sharing links on other pages and so on to cheat the PageRank algorithm. And Google have been caught in a fight with SEO ever since.
- simiones 4y ago
  Do we have any example of an open engine working badly? ' As far as I know, this hasn't really been tried, definitely not with the kinds of resources this project will have available.