25 ms·

EU Open Web Search project kicked off

404 points by ZacnyLos 4y ago

rurban 4y ago
openwebsearch without open code and open issues, really? A tiny image with a few nodes and stakeholders, including third party indices and monetary services ("Third Party Enrichment Services"). In which decade are they living? https://openwebsearch.eu/the-project/ https://openwebsearch.eu/the-project/
jacooper 4y ago
> A new EU project OpenWebSearch.eu … [in which] … the key idea is to separate index construction from the search engines themselves, where the most expensive step to create index shards can be carried out on large clusters while the search engine itself can be operated locally. …[including] an Open-Web-Search Engine Hub, [where anyone can] share their specifications of search engines and pre-computed, regularly updated search indices. … that would enable a new future of human-centric search without privacy concerns. So.. Who's going to create the index? Indexing the web is expensive, and its offset by the ads the indexer runs on their search website, such as Google, bing, brave and others.
- moffkalast 4y ago
  Someone who's snagging an EU grant, that's who.
  - ur-whale 4y ago
    > Someone who's snagging an EU grant, that's who. Bullseye.
- amelius 4y ago
  I wonder how privacy will be ensured when your query hits the map-reduce infrastructure running on these clusters. Regarding privacy the bar is significantly higher than what Google has to deal with. This will come at some cost in quality and/or speed.
- caust1c 4y ago
  Every individual website has an incentive to create indices of their own content, and hosting providers could provide it as a service. Not hard to envision. Search Engines could download these indices periodically to build the meta-search.
  - wizofaus 4y ago
    Also not hard to envision websites being incentivised to lie in their indexes.
    - caust1c 4y ago
      Lie how? Meta indexes can be picky about the sites they list, simple as that. One could imagine personal meta search engines which only list the sites you care about. Bad sites simply don't get listed.
      - wizofaus 4y ago
        The point is that if a popular search engine returned pages primarily based on whether particular search terms appeared in their self-published indexes (rather than using a crawler that parsed the page content to build its own index), web sites could easily publish indexes that just listed popular terms users are known to search for regardless of whether the site had anything to do with those search terms. Pretty much what uses to happen 10-15 years ago when everyone believed meta tags were the way to achieve SEO.
- arvonle 4y ago
  Well easy, Europeans are going to fund it with their taxes, public search labs are going to build it, private companies squeeze the concept for cash for as long as possible while taking EU taxpayer money as "research grants" and claiming operational costs as "research tax credits" which will set them up as special parners for the next EU cashgrab (I mean quadriannual plan) since they have successfully run a public-private partnership (into the ground) t. was involved in one of those plans as part of the research team in a public lab
nathan_phoenix 4y ago
This is just a short reply to a blog which mentions that the project started... The actual website of the project (with some concrete info) can be found here: https://openwebsearch.eu/ https://openwebsearch.eu/
- dang 4y ago
  Changed now. Thanks!
Proven 4y ago
[dead]
bslqn 4y ago
tricky777 4y ago
seems like a very interesting idea. So many times I wanted some kind of advanced gogle-query-language. (i know about allinurl and such, but thats not enough. google is tuned for average user, which is good for google, but not for any non average query)
rgrieselhuber 4y ago
The real game-changer in search would be if companies would agree to publish indexes of their own sites in an open standard to a place that everyone could access. This would undercut the monopoly power that large search engines have and allow everyone to focus on innovating the best way to search that content vs. having to spend so much time and money to crawl and index it.
- TheFerridge 4y ago
  I believe that is precisely what the project is aiming to do, and to turn it into a public resource.
- wizofaus 4y ago
  I suspect you underestimate how much of the power of search engines is being able to interpret search queries and figure out what a user is really looking for. Even if there were a public, standardised up-to-date high performance full-text index of the entire web freely available I'm willing to bet Google search would be a useful value-add in its ability to answer natural language queries.
  - rgrieselhuber 4y ago
    I run an SEO platform SaaS, so I'm familiar. :)
  - mellavora 4y ago
    > you underestimate how much of the power of search engines is being able to interpret search queries and figure out what a user is really looking for. So you mean that a search engine is supposed to ignore what you are asking for and instead give you what it thinks you really meant?
    - wizofaus 4y ago
      If you search for [what year was queen Elizabeth born], Google doesn't return pages that have that phrase in it, rather it returns pages that have an answer to that question. You could call that "ignoring" if you like, but it's what 99% of users expect.
rrwo 4y ago
It will be interesting to see what the index contains, and how it is structured. What made Google such a game changer was that they based their index not just on the contents, but on how pages linked to each other.
- arjenpdevries 4y ago
  That's the marketing story. I think it's because they didn't clutter their homepage like AltaVista did.
  - margarina72 4y ago
    I was a heavy AltaVista user back in the time, I can tell you Google results were really higher quality, it wasn't only their homepage (even if that was also quite refreshing).
  - rrwo 4y ago
    Nope. It's actually how it worked. They patented it and published papers about it.
andrewmcwatters 4y ago
I suspect search engines are an outdated concept for at least the largest of sites, who will generally, but not always, have better ways to directly search their own content. The remainder of the search problem seems to just be collecting relevant trafficked sites for listing in results. Today Google et al seem to be doing this BY HAND. And it's not even obfuscated. Recently, for the first time in my life, the wizard behind the curtain seems to have been exposed. I feel strongly that one could probably start a small index that catered to a fairly large audience. And honestly, for other queries, just tell the user to search that site directly. I think you could even market it to users as not a technical limitation, but behavior that should be considered fuddy-duddy. Like, really, you're going to search me? You know they have their own search right? Even Yellow Pages faded into obscurity eventually.
- ur-whale 4y ago
  > the largest of sites, who will generally, but not always, have better ways to directly search their own content. I have the exact opposite experience. To wit: searching HN via the algolia link at the bottom is way worse than searching on Google with a site:ycombinator.com restrict. Same thing for YouTube, where the search engine is tuned for maximizing watch time and strictly not to return what you're looking for.
  - not2b 4y ago
    I would agree; I've almost never seen a site that provides a better way to directly search its own content than the major search engines provide. Wikipedia might come closest because their search compares the user input to article titles, and that's often helpful (mainly because the article titles are chosen to describe the article content, and aren't clickbait). But pretty much anything else uses a search approach that is far inferior to what Google or Bing provide, and it won't find what you want.
hrbf 4y ago
I’ve already caught their crawler ignoring robots.txt directives on one of my sites, aggressively indexing explicitly excluded information.
- lizardactivist 4y ago
  Out of curiosity, what's the url for your website, and from what IP or host do their crawlers connect?
  - hrbf 4y ago
    The main connecting IP was 195.113.175.41.
- arjenpdevries 4y ago
  That cannot be true, as the project has yet to start. But anyone can start a crawler, so you may have encountered other people's software. We wouldn't be so unknowledgeable to ignore robots.txt ;-)
  - hrbf 4y ago
    It was a crawler with the user agent "hgf AlphaXCrawl/0.1 (+https://www.fim.uni-passau.de/data-science/forschung/open-search https://www.fim.uni-passau.de/data-science/forschung/open-se...)", operated by the University Passau and Open Search Foundation named on your landing page. It would be a mighty big coincidence if this wasn't a project connected to this endeavor, especially when it confirms being an experimental crawler of said project at the UA URL.
- jaimex2 4y ago
  Wouldn't it be impossible to know if it ignored robots.txt? Just because it crawled it doesn't mean it stored it.
  - hrbf 4y ago
    Storage or not is entirely irrelevant to robots.txt directives. It guides automated access. It must be parsed first and excluded URLs must not be accessed at all.
thepangolino 4y ago
dang 4y ago
Url changed from https://www.zylstra.org/blog/2022/09/eu-open-web-search-project-started/ https://www.zylstra.org/blog/2022/09/eu-open-web-search-proj..., which points to https://djoerdhiemstra.com/2022/open-web-search-project-kicked-off/ https://djoerdhiemstra.com/2022/open-web-search-project-kick..., which points to this.
- lucideer 4y ago
  Which now shows: > Resource Limit Is Reached > The website is temporarily unable to service your request as it exceeded resource limit. Please try again later. Original URL might be more resilient...
  - dang 4y ago
    Hmm. I can access the page without that message. In any case the Internet Archive seems to have it: https://web.archive.org/web/20220920183027/https://openwebsearch.eu/ https://web.archive.org/web/20220920183027/https://openwebse...
DOsinga 4y ago
With a budget of 8.5M Eur/Usd. Alphabet spends 200B per year. If 40% of that is spend on search, their budget is 10 thousand times larger.
- lucideer 4y ago
  It's definitely a comparative underdog regardless, but if you think Alphabet spends anywhere near 40% on search you're out of your mind. I'd be shocked if their spend is double-digits. I'd be unsurprised if it's <1%.
- o_m 4y ago
  I doubt 40% is spent on search. Seeing how bad Google has gotten, it seems more likely there is just a skeleton crew keeping the lights on
- mkl95 4y ago
  I would be shocked if Alphabet spent >5% on search. But even 1% would dwarf this project.
- mellavora 4y ago
  The https://memex.marginalia.nu/ https://memex.marginalia.nu/ search engine runs basically on one desktop computer (if I recall, the creator made a post on this a week or so ago). How much of what google spends on "search" is strictly for search, vs business goals related to search? How much of that google spend is salary? What are those salaries? How do they compare with EU post-doc salaries?
beardedman 4y ago
Oh cool, but do you mean the "EU Open Web Search Data Collection Program"?
notright 4y ago
the EU loves taxing productive companies and wasting said money in stillborn projects that nevertheless promise a kind of bright socialist federalist Europe in their bureaucratic minds
- arjenpdevries 4y ago
  Isn't it lovely?!
  - notright 4y ago
    I am fine as long as they pay for these self-centered utopias with their own money
    - 6510 4y ago
      I'm fine with paying 2 cent for this.
  - htkibar 4y ago
    On a side note - how does one get involved with the project should they wish to do so?
- Comevius 4y ago
  At least we have some of the most livable countries on Earth to show for it. I take taxes over any trickle-down economics, and don't let me stop you looking up the definition of socialist, because you are using it wrong. Besides it's a 8.5 million EUR project, it's literally nothing, it's payroll for a few people. The money is being invested into people who then spend most of it, so it's a triple investment.
- qualudeheart 4y ago
  Who said anything about socialism? It’s a geopolitical tool to weaken American and Russian influence from Google and Yandex.
ur-whale 4y ago
Looks like it's Northern EU only. No research institutes from {France, Italy, Spain, Greece, Portugal, etc ...} involved.
- arjenpdevries 4y ago
  Slovenia, Czech Republic. But yes, I think there was a competing proposal from Italy/Spain. Not enough budget for two projects in this area, unfortunately, as they were good too.
dataking 4y ago
I don't see any mention of Quaero, the EU search engine that was supposed to compete with Google [0, 1]. How is this time different? [0] https://en.wikipedia.org/wiki/Quaero https://en.wikipedia.org/wiki/Quaero [1] https://www.dw.com/en/germany-pulls-away-from-quaero-search-engine-project/a-2287489 https://www.dw.com/en/germany-pulls-away-from-quaero-search-...
- notright 4y ago
  This was the past legislature project. The new legislature brings CHANGE. They are not the same..
- arjenpdevries 4y ago
  For starters: the objective is to create the index not the engine, that's quite a different ambition. We are very aware of the Quaero/Theseus history :-)
  - notright 4y ago
  - marginalia_nu 4y ago
    What is the difference?
    - freediver 4y ago
      Supposedely the project is about just building the platform/infrastructure (which is what the index is) upon which search engines can be built. These search engines will then have the freedom to define their own search product experience, business model, even ranking of results.
      - jonas21 4y ago
        So something even more vaguely defined and detached from real use cases than last time? Great.
boyter 4y ago
I have written this before but I’ll put it here again. What I would like to see is a federated search engine. Based on activitypub that works like mastodon. Don’t like the results from one source? Just remove them from your sources, or lower their ranking. Similar to yacy but you can work with the protocol to connect or build whatever type of index you want using whatever technology you like, and communicate over an existing standard. Want to build the worlds best index of Pokémon sites, then go do it. Want to build a search engine using idris or ats? Sure! I did note the professors are on mastodon so perhaps this may actually happen. One of these days I’ll actually implement the above assuming nobody else does. I figured if I can at least get the basics done and a reference implementation that’s easy to run it could prove the concept. If anyone is interested in this do email my in my bio. What I worry about for this project is that it becomes another island which prohibits remixing of results like google and bing, and its own index and ranking algorithms become gamed. I wish the creators best of luck though. I am also hoping for some more blogs and papers about the internals of he engine. So little information is published in the space that anything is welcome, especially if it’s deeply technical.
- fabrice_d 4y ago
  At least one of the partners (https://openwebsearch.eu/partners/radboud-university/ https://openwebsearch.eu/partners/radboud-university/) does research on "federated search systems", so there's hope!
- arjenpdevries 4y ago
  We like Federated search, we like decentralized search, and even P2P search; we are trying to find a good mix, and decided to get started rather than wait! Exciting times.
  - hkt 4y ago
    I would love to be able to run a node that mirrors part or all of an index like this, and to let people query it - a bit like https://torrents-csv.ml/#/ https://torrents-csv.ml/#/ Good luck! I'll be watching your progress and cheering you all on!
  - 4y ago
ur-whale 4y ago
Search is way more than just indexing. I'd really like to see them match the 20+ years of search quality fine-tuning that Google built into their search engine. Not that Google is as good as it used to, but still, catching up with them is way more complicated than just building a big crawl + index piece of infrastructure. And all of that on a government-funded shoestring budget. Mmmh. Good luck to them, but I'm not holding my breath.
- Kamq 4y ago
  I'd like to see them match the 10 years of fine tuning from 2012 google. My understanding is google used to keep a lot of data around for "long tail" queries, and stopped doing that at some point. This seems to be the issue around search declining outside of "food near me" type queries (which they absolutely excel at).
merb 4y ago
so it began, that sern starts to gather market share. -- I doubt this will take off. I mean they investend more in funding and marketing instead of starting to built something. they should've started with code (agpl3 of course) and invited more and more people. at the moment this is more buzzword bingo bullshit than anything else. it's basically always the same problem, instead of focusing on the product, they fous more on the message.
Animats 4y ago
Early version: https://www.chatnoir.eu/ https://www.chatnoir.eu/
imhoguy 4y ago
"unbiased...based on European values" - will it fly?
- topspin 4y ago
  European values are inherently unbiased. What's the problem? o.O
  - not2b 4y ago
    There's no such thing as an unbiased yet useful search engine. The very act of choosing which article to put at the top is a value judgment, therefore a bias. The only way to make things "unbiased" would be to rank the matches randomly, and that would be useless.
    - xigoi 4y ago
      This is just an index on which different search engines with different priorities can be built.
    - denton-scratch 4y ago
      > The only way to make things "unbiased" would be to rank the matches randomly Well, I would rather like to have visibility into the ranking algorithm, and to be able to control the resultset ordering.
      - not2b 4y ago
        So would the SEO people. But even disregarding that problem, to have a halfway decent search ranking you're going to need deep learning, and you'll get a model that is largely opaque.
        denton-scratch 4y ago
        Heh! You've mistaken me for a marketer, I think. When I said I wanted control over the ranking, I meant as a user. Tell me what parameters I can rank on; give me a UI that makes it reasonably easy to express a ranking. That's it. I hope UI design and stuff is orthogonal to the construction of an open index. I think this could be very interesting.
logicalmonster 4y ago
What does "based on European values and jurisdiction" refer to? I'd love to be pleasantly surprise, but this sounds like it's ripe for centralized censorship.
- notright 4y ago
- InTheArena 4y ago
  Given the history of the 20th century, this kind of comment promoting European values and jurisdiction seems..... dicey. Companies ethical records, as shitty as they are have nothing on the mass destruction, genocide and stupidity of governments.
- wumpus 4y ago
  "Right to be forgotten" is an example of a European value.
  - denton-scratch 4y ago
    Um, bad example. It was a misconceived bit of legislation. There's no logic behind that claimed right. What happened to my right to remember what I choose? Why shouldn't I publish my memories on a website? Anyway, it's not a "European value", it's a piece of legislation, so it falls under the "European jurisdiction" category, not the "European values" category. That complying with "European values" is part of the project goals is unfortunate. Who decides what are European values? There's no Declaration of European Values. We don't all have the same values. There are just European laws, and then a whole bunch of opinions that not all Europeans agree about.
    - permo-w 4y ago
      people really, truly, go out of their way to criticise the EU. there are things about the EU that deserve criticism, but this is reaching to the point of deliberate misunderstanding. no one is saying that a person who wants to be forgotten cannot be talked about on your website, or that you have to somehow deliberately delete your memories of that person. you are not a corporation. the right to be forgotten is about the right for you to be forgotten by a corporation. the right for you to not be in their database anymore
marginalia_nu 4y ago
I'm a bit skeptical EU-funding a bunch of professors is the way a search engine will be built. The primary goal for academics is to publish new findings, while what you need to build a search engine is rock solid CS and information retrieval basics. Academically, it's not very exciting. Most of it was hashed out in the 1980s or earlier.
- margarina72 4y ago
  One could have been skeptical about US-funding a bunch of university students to build a search engine a few years ago.
  - harha 4y ago
    Search was quite broken back then. It got reasonably good at some point, now I’d argue it’s the content as a whole that’s gotten worse. I’m just skeptical if the EU bureaucrats will put the money in the right place, and if this is even the right approach.
    - margarina72 4y ago
      The project can fail, you are correct, but it does not take anything from any other projects, it is just an other initiative trying to contribute in the space. Parent commenter own search project Marginalia Search [1] could even benefit from it, or even maybe collaborate with it. It is not a winner-take-all situation, and we need various open initiative in this space to get out of the current conundrum we are in with Google stronghold on search. 1: https://search.marginalia.nu https://search.marginalia.nu
      - harha 4y ago
        Taxes are taking from funds for personal projects, and universities already get a lot of funding in Europe to do research, maybe they should focus on creating a better environment for students to become researchers or entrepreneurs instead. Overall I think there are better ways to improve search from an EU perspective by doing what they are supposed to be doing: - create a fair environment for companies to compete in, e.g., take a look how Google, Apple or Meta's assets are set up to make it harder for competitors, break that up - improve standards in eduction – it doesn't really make sense for all member countries to think of and maintain a good CS curriculum and they all seem to be pretty bad at it - make it easier to build something and get funded, and reward creating prosperity, don't tax it to death Just tested marginalia's random mode btw. Pretty cool, reminds me of the internet when I was a kid (edit: formatting)
antics9 4y ago
We need to develop a social aspect to search where results are also moderated and curated by humans in some kind of way.
- topspin 4y ago
  And when that curation produces results you find abhorrent? What then? Because I guarantee it would; a metaphysical certitude.
  - antics9 4y ago
    Do you hang out with abhorrent people?
- wumpus 4y ago
  Blekko (2007-2015) did that -- apparently it wasn't enough for commercial success.
Extropy_ 4y ago
On first glance, I see the word "unbiased" immediately followed by "based on European values". Now, I'm no expert, but to me, that seems pretty biased.
- radiojasper 4y ago
  biased on European values
- casualrandomcom 4y ago
  "being unbiased" can be fairly honestly called a "value". Europeans apparently have chosen it as one of their own values. If you are arguing that being truly unbiased cannot ever be realized, I counter that neither truth nor justice can ever be truly realized, by this should not stop anyone from taking them as their values. signed by: an expert
  - denton-scratch 4y ago
    It's not that there's anything wrong with aspiring to be unbiased; the problem arises when someone really thinks they are unbiased, or claims to be unbiased. You get this a lot in the MSM; the Guardian constantly exhorts it's readers to subscribe, to support unbiased journalism. I prefer reporting that wears its biases on its sleeve. Regrettably The Guardian squeezed out most of its most interesting writers, apparently by repeatedly spiking their stories. Yeah, Truth and Justice are platonic abstracts; nobody thinks they exist in the real world. But people do believe that unbiased reporting is possible. It isn't.
deleted 4y ago
[deleted]
s-xyz 4y ago
Correct me if I am wrong, but so the purpose is to create an index database, upon which custom search engines can be attached upon? Ie, the EU will crawl all pages on the web?
- murphyslab 4y ago
  The index is just the first step according to news articles: > Once the index has been created, the next step is to develop search applications. > The team at TU Graz will be particularly active here in the CoDiS Lab and will work on the conception and user-centric aspects of the search applications. This includes, for example, research into new search paradigms that enable searchers to have a say in how the search takes place. The idea is that there are different search algorithms or that you can influence the behavior of the search algorithms. For example, you could search specifically for scientific documents or for documents with arguments, include search terms that have already been used, or include documents from the intranet in the search. https://www.krone.at/2791083 https://www.krone.at/2791083
margarina72 4y ago
Interesting to see the amount of negative comments. Most of the negativity seems to come from the following points: - EU-funded project cannot succeed in tech because previous EU-funded projects have failed in tech in the past (and generally government-funded project in tech are suspicious) - Search is hard, and therefore it will fail - The project is underfunded Even if the points above can all be valid (though the obvious US-funded startup launched by a bunch of uni students seemed to have fared pretty well) it seems we are missing the point of this project. The proposal is to contribute to the creation of open building blocks necessary to enable others (including private US companies) to make better search products. Better search product are needed. Shall we remember HN of some of the intense conversations that happened here about Google failing us: - Google Search is Dying [1] - Every Google result now looks like an ad [2] - Google no longer producing high quality search results in significant categories [3] So while, yes search is a hard topic, we should welcome initiatives aiming at improving the ground infrastructure needed to lower the barrier to entry on this subject and hope it will allow many companies to build better search products (or inspire other initiatives to contribute in similar and even more successful way) 1: https://news.ycombinator.com/item?id=30347719 https://news.ycombinator.com/item?id=30347719 2: https://news.ycombinator.com/item?id=22107823 https://news.ycombinator.com/item?id=22107823 3: https://news.ycombinator.com/item?id=29772136 https://news.ycombinator.com/item?id=29772136
- Proven 4y ago
  [dead]
- ffhhj 4y ago
  If these thinkers can find a way to remove the incentives of spammers, misinformators, and stakeholders to pollute results, that would be a great achievement. It could be seen as a big economic game, simulating these actors might allow comming up with rules to balance this game and minimize pollution.
  - petre 4y ago
    It might be possible using moderation and reporting abuse or registering with the search engine to aid moderation and banning abusers. This will probably also take down right wing websites, porn and online gambling.
hexage1814 4y ago
Will it censor the so called "hate speech"?
- speedgoose 4y ago
  I have never seen “hate speech” that wasn’t hate speech.
luis8 4y ago
Would it be hard to expand it as global endeavour?
- solardev 4y ago
  Governments would never agree on what to show or hide.
fspeech 4y ago
A crawler should see the web pages as a human would and read the pages as a human would and extract keywords based on the real displayed content. After that the search could be as simple as looking up a book in your library catalog. Think liberally about the search key, e.g. "<40% screen is display ads". If the search engine gets popular, you can then worry about scaling up to serve the results faster. Oh PageRank (R) can be done using real semantic links (actual human comments) rather than easily spammed hyperlinks. And you can group all similar pages and serve those as a single expandable/refinable entry.
- alkonaut 4y ago
  Yes, telling spam (spam blogs, copy-pasted versions of Stackoverflow, etc) from real organic content is easy for a human, so I can only assume it would be pretty easy for a computer too. At least until this becomes a practice that spam site makers have to consider and adapt. For example, given 10 similar articles where one has a well-known domain but few links and the others are all cross-linked only among a known spam-net, then just show the well known page (stackoverflow) and not the useless copycats. It seems doable for a computer to also navigate the page like a human and try to negotiate various cookiewalls and similar. If a page requires jumping through hoops to enter, rank it poorly. There should be a single decline/reject if there is a button at all, and the content ranked should be the content shown after rejecting. If it shows two buttons "Accept" and "Options" then just don't index that page.
  - denton-scratch 4y ago
    > If a page requires jumping through hoops to enter, rank it poorly. Oh, how I wish that search engines would simply give up on encountering a paywall/cookiewall, and refuse to index the page.
psuresh 4y ago
Good to see more efforts from EU to negate big tech domination. Federation is the way forward.
tommica 4y ago
I really hope this project is successful - the heart seems to be in the right place!
tyiz 4y ago
Galileo, anyone? The EU is the trash bin of politicians who failed in their respective country. Not elected and saturated with piles of money. It is under control of the American Empire, thus designed to fail with a push of a button if necessary (€ is the key). Wake up!
qualudeheart 4y ago
This is an excellent idea to disintermediate Google. A search tool for the commons without the spyware and opportunities for search manipulation is very important, and the more actors work on it, the better. Perhaps cooperations with other tech savvy democracies could occur. India is full of skilled programmers. Unlike Russia or China it is easy to see Europe cooperate with it.
- denton-scratch 4y ago
  Hmm. India also has a government that is borderline fascist, promotes ethnic and religious hatred, and doesn't seem to believe in internet freedom (they keep shutting it down). And it's not the lack of skilled programmers that makes Western governments wary of cooperating with China and Russia. China and Russia have totalitarian governments that are terrified of the internet, and would like it to go away. I wouldn't want to include them in a project to make the internet work better.
jhoelzel 4y ago
I agree that we are sadly ending the chapter of free search but im quite sure this is just another roll call for some professors to make a quick buck like it always is. Search needs help its true, but 8 million are not even gonna move the needle and that other engines have that much money invested, just to maintain their network switches.
stiray 4y ago
What personally worries me the most is "Meet the partners" page. I didnt look for all but I did look who is partner from Slovenia. The most privacy invading ISP/Mobile company in our country selling statistics about their users (although anonymized, but walking on a thin line). I just hope it wont go down that drain. Regarding all the negativity about the government project. If we can run CERN we can surely do a web search project.
berry_sortoro 4y ago
Roark66 4y ago
Seriously... I really wanted to like this project, but it seems everything EU touches as of recently gets worse. From the webpafe that half of the time shows "Resource Limit exceeded" to a technology stack diagram on the bottom of this page https://openwebsearch.eu/the-project/ https://openwebsearch.eu/the-project/ being completely unreadable due to bad scaling. It is very disappointing really. Another example from the top of my head. Here in Poland we have ID cards(as every other EU country) . Those ID cards have to be renewed every now and then (10~15years). In last years an online system for government services was implemented including for renewal of those cards. One could take a photo with a mobile phone, submit an application and pick up a card from a gov office in few weeks time. Unfortunately, EU made a law that ID cards applications have to be acommpanied by biometrics (fingerprints) so this system has been thrown away. One has to physically go to the gov office, scan their fingerprints, apply for a new id card and then go again to pick it up... Ok, so what happens in 10 years time? They should have the fingerprints already, right? No. They take the fingerprints, they store them only until one picks up the id card and then they are deleted. There are no fingerprint database, they are not stored anywhere. The fingerprints are used only to ensure the same person that submitted the application picks up the document. It makes zero sense, other than to break the previous online system. Thanks EU.
- ssmiler 4y ago
  And why the hell the government will keep the fingerprints? So that once in 10 years I save 1h. The benefit is minor compared to bad things that can be done with a fingerprints DB (mass surveillance..)
  - jld89 4y ago
    > And why the hell the government will keep the fingerprints? The government already does... just saying.
    - simiones 4y ago
      They just explained that it doesn't (assuming we trust them, of course).
hericium 4y ago
Is there a privacy policy in English? The only one I could find[1] is in German. [1] https://openwebsearch.eu/privacy-policy/ https://openwebsearch.eu/privacy-policy/
sazz 4y ago
Before Corona, I would have really welcomed this announcement. But to be honest, when someone says that the project will be "based on European values," I think it's stillborn. Who is defining those "European values"? Is it the European Commission? As a liberal person who still dreams of mature citizens who form their own opinions well-informed from a rich debate, I now see the "European values" quite critically. In Germany - if you criticize the Corona measures you are called a Nazi, if you criticize the Ukraine war you are a Russian troll. Are those then the "European values"? The European Commission has already established some projects to weight the information according to its will (SocialTruth, PROVENANCE, EUNOMIA, etc.). When a government agency talks about truth, then all the hairs on the back of my neck stand up. When governments speak of disinformation, it is usually only in the sense that the information does not correspond to their interests. I had to learn that the so-called fact checkers don't check facts, they just sell a counter opinion as a "fact". So for a European search engine, a filter is then placed upstream that filters out all disinformation? In the past, that was called censorship. Today, it's more like citizen service.
- johnywalks 4y ago
  > If you criticize the Corona measure you are a Nazi, if you criticize the Ukraine war you are a Russian troll? These notions are propagated by the US to the rest of the world. While EU politicians seem to go along, europeans do not tolerate it much and that much is reflected in society. If the EU or more gov sponsored search engines pop up, i have no doubt they will want to control them in their own way - i'm okay with that. Right now our only option is US controlled entities that answer to US govs and rules. At least here we can point fingers and hold politicians accountable if they try to influence things the wrong way. We can't do that with US companies.
  - sazz 4y ago
    > At least here we can point fingers and hold politicians accountable if they try to influence things the wrong way. We can't do that with US companies. Germany is currently governed by a chancellor who has in fact been convicted of lying, has selective memory lapses as a co-responsible party in one of the biggest banking scandals, and was also instrumental in the political decisions to push ahead with dependence on Russian gas. So sorry if I doubt that politicians can be made accountable for anything.
reacharavindh 4y ago
I wish a project doesn't try to moonshot a thing that we already had.. I want the good old Google search where you type something, and all relevant web result pop out (without creepy ads, trackers, personalisation, shopping links to get you to spend). I understand it is very expensive to crawl the web, that's where an org like the EU can come in. I wish they bootstrapped a simple, open index that any startup can use to provide clean search fronts. Crawling the web, and indexing content can be quite expensive, if EU bootstrapped that infrastructure, and then say augmented it with federated crawling and indexing (sort of like the distributed computing MIT condor project), where people, Universities etc in the EU could contribute their spare computer time to crawling the web and indexing it for the EU index, making search better for everyone. Heck, I'd even put solar panels in my bike shed and hook it up to a PC to crawl the web while the sun is shining. I'm dreaming too much, I need coffee..
- deleted 4y ago
  [deleted]
- mjburgess 4y ago
  But the web is different; google much less so. There is no "old google search", because almost everything to search through is noise and spam. PageRank made user preferences as signalled by hyperlinks the key signal of quality. Who really hyperlinks any more? And how many, by percentage, are non-automated etc.? The web this works for is one of forums & personal websites. It's hard to say that today there are any properties of websites that are a reliable signal of quality. Hence the proliferation of voting sites (such as HN, etc.) which are little more than search engines augmented with reliable signals of user preferences.
  - permo-w 4y ago
    >But the web is different; google much less so. There is no "old google search", because almost everything to search through is noise and spam the commenter is not referring to the search results, they're referring to the interface not having all the wank that google packs in to optimise for profit >Who really hyperlinks any more? who doesn't?
paul-j_ 4y ago
I hope it will have proper regex capability for narrowing the search results to more relevant content.
SirOibaf 4y ago
GAIA-X, is that you?
nickdothutton 4y ago
Some of you might be interested in the privacy focussed "French Google" Qwant. https://en.wikipedia.org/wiki/Qwant https://en.wikipedia.org/wiki/Qwant Plot twist: It's funded by Huawei.
sylware 4y ago
??? Where is the EU own and operated 2nm foundry?
denton-scratch 4y ago
I couldn't work out what "building blocks" they're going to produce. Are they going to produce a public index, that search engines can use? A set of APIs? > the researchers will develop the core of a European Open Web Index (OWI) as a basis for a new Internet Search in Europe. That sounds as if building an index is at least within scope. But isn't the design of the index one of the key differentiators between different search engines? Perhaps the OWI will just produce tools for building transparent, privacy-preserving search engines, rather than crawling the web itself. It's not clear (to me) from reading the site.
- arjenpdevries 4y ago
  The project is starting so not all your questions are answerable today, but, we definitly will produce an open web index, already by the end of the first year, with improvements for years two and three. We further deliver components to make search engines on top of this index. The project vision is that there will be many different search engines, not just 4 worldwide. Hoping to lead the way!
zerof1l 4y ago
It doesn't sound open and unbiased to me, because it will allow the government to censor on an index level.
frozencell 4y ago
> Unbiased One of the tech lead feed [0] looks quite biased towards Ukraine though. I hope it doesn't interfere with the search engine. [0] https://twitter.com/mgrani https://twitter.com/mgrani
- tristanb 4y ago
  There is no bias. Russia is a terrorist state that has brutally and illegally murdered civilians in a country they chose to invade. Hopefully it will be their continued downfall.
  - frozencell 4y ago
    All countries are terrorist states if we look at it this way.
XCSme 4y ago
> Free, open and unbiased access to information How can something be unbiased yet somehow stop showing illegal search results?

404 points by ZacnyLos 4y ago

openwebsearch without open code and open issues, really? A tiny image with a few nodes and stakeholders, including third party indices and monetary services ("Third Party Enrichment Services"). In which decade are they living? https://openwebsearch.eu/the-project/ https://openwebsearch.eu/the-project/

jacooper 4y ago

> A new EU project OpenWebSearch.eu … [in which] … the key idea is to separate index construction from the search engines themselves, where the most expensive step to create index shards can be carried out on large clusters while the search engine itself can be operated locally. …[including] an Open-Web-Search Engine Hub, [where anyone can] share their specifications of search engines and pre-computed, regularly updated search indices. … that would enable a new future of human-centric search without privacy concerns. So.. Who's going to create the index? Indexing the web is expensive, and its offset by the ads the indexer runs on their search website, such as Google, bing, brave and others.

moffkalast 4y ago
Someone who's snagging an EU grant, that's who.
- ur-whale 4y ago
  > Someone who's snagging an EU grant, that's who. Bullseye.
amelius 4y ago
I wonder how privacy will be ensured when your query hits the map-reduce infrastructure running on these clusters. Regarding privacy the bar is significantly higher than what Google has to deal with. This will come at some cost in quality and/or speed.
caust1c 4y ago
Every individual website has an incentive to create indices of their own content, and hosting providers could provide it as a service. Not hard to envision. Search Engines could download these indices periodically to build the meta-search.
- wizofaus 4y ago
  Also not hard to envision websites being incentivised to lie in their indexes.
  - caust1c 4y ago
    Lie how? Meta indexes can be picky about the sites they list, simple as that. One could imagine personal meta search engines which only list the sites you care about. Bad sites simply don't get listed.
    - wizofaus 4y ago
      The point is that if a popular search engine returned pages primarily based on whether particular search terms appeared in their self-published indexes (rather than using a crawler that parsed the page content to build its own index), web sites could easily publish indexes that just listed popular terms users are known to search for regardless of whether the site had anything to do with those search terms. Pretty much what uses to happen 10-15 years ago when everyone believed meta tags were the way to achieve SEO.
arvonle 4y ago
Well easy, Europeans are going to fund it with their taxes, public search labs are going to build it, private companies squeeze the concept for cash for as long as possible while taking EU taxpayer money as "research grants" and claiming operational costs as "research tax credits" which will set them up as special parners for the next EU cashgrab (I mean quadriannual plan) since they have successfully run a public-private partnership (into the ground) t. was involved in one of those plans as part of the research team in a public lab

nathan_phoenix 4y ago

This is just a short reply to a blog which mentions that the project started... The actual website of the project (with some concrete info) can be found here: https://openwebsearch.eu/ https://openwebsearch.eu/

dang 4y ago
Changed now. Thanks!

Proven 4y ago

[dead]

bslqn 4y ago

tricky777 4y ago

seems like a very interesting idea. So many times I wanted some kind of advanced gogle-query-language. (i know about allinurl and such, but thats not enough. google is tuned for average user, which is good for google, but not for any non average query)

rgrieselhuber 4y ago

The real game-changer in search would be if companies would agree to publish indexes of their own sites in an open standard to a place that everyone could access. This would undercut the monopoly power that large search engines have and allow everyone to focus on innovating the best way to search that content vs. having to spend so much time and money to crawl and index it.

TheFerridge 4y ago
I believe that is precisely what the project is aiming to do, and to turn it into a public resource.
wizofaus 4y ago
I suspect you underestimate how much of the power of search engines is being able to interpret search queries and figure out what a user is really looking for. Even if there were a public, standardised up-to-date high performance full-text index of the entire web freely available I'm willing to bet Google search would be a useful value-add in its ability to answer natural language queries.
- rgrieselhuber 4y ago
  I run an SEO platform SaaS, so I'm familiar. :)
- mellavora 4y ago
  > you underestimate how much of the power of search engines is being able to interpret search queries and figure out what a user is really looking for. So you mean that a search engine is supposed to ignore what you are asking for and instead give you what it thinks you really meant?
  - wizofaus 4y ago
    If you search for [what year was queen Elizabeth born], Google doesn't return pages that have that phrase in it, rather it returns pages that have an answer to that question. You could call that "ignoring" if you like, but it's what 99% of users expect.

rrwo 4y ago

It will be interesting to see what the index contains, and how it is structured. What made Google such a game changer was that they based their index not just on the contents, but on how pages linked to each other.

arjenpdevries 4y ago
That's the marketing story. I think it's because they didn't clutter their homepage like AltaVista did.
- margarina72 4y ago
  I was a heavy AltaVista user back in the time, I can tell you Google results were really higher quality, it wasn't only their homepage (even if that was also quite refreshing).
- rrwo 4y ago
  Nope. It's actually how it worked. They patented it and published papers about it.

andrewmcwatters 4y ago

I suspect search engines are an outdated concept for at least the largest of sites, who will generally, but not always, have better ways to directly search their own content. The remainder of the search problem seems to just be collecting relevant trafficked sites for listing in results. Today Google et al seem to be doing this BY HAND. And it's not even obfuscated. Recently, for the first time in my life, the wizard behind the curtain seems to have been exposed. I feel strongly that one could probably start a small index that catered to a fairly large audience. And honestly, for other queries, just tell the user to search that site directly. I think you could even market it to users as not a technical limitation, but behavior that should be considered fuddy-duddy. Like, really, you're going to search me? You know they have their own search right? Even Yellow Pages faded into obscurity eventually.

ur-whale 4y ago
> the largest of sites, who will generally, but not always, have better ways to directly search their own content. I have the exact opposite experience. To wit: searching HN via the algolia link at the bottom is way worse than searching on Google with a site:ycombinator.com restrict. Same thing for YouTube, where the search engine is tuned for maximizing watch time and strictly not to return what you're looking for.
- not2b 4y ago
  I would agree; I've almost never seen a site that provides a better way to directly search its own content than the major search engines provide. Wikipedia might come closest because their search compares the user input to article titles, and that's often helpful (mainly because the article titles are chosen to describe the article content, and aren't clickbait). But pretty much anything else uses a search approach that is far inferior to what Google or Bing provide, and it won't find what you want.

hrbf 4y ago

I’ve already caught their crawler ignoring robots.txt directives on one of my sites, aggressively indexing explicitly excluded information.

lizardactivist 4y ago
Out of curiosity, what's the url for your website, and from what IP or host do their crawlers connect?
- hrbf 4y ago
  The main connecting IP was 195.113.175.41.
arjenpdevries 4y ago
That cannot be true, as the project has yet to start. But anyone can start a crawler, so you may have encountered other people's software. We wouldn't be so unknowledgeable to ignore robots.txt ;-)
- hrbf 4y ago
  It was a crawler with the user agent "hgf AlphaXCrawl/0.1 (+https://www.fim.uni-passau.de/data-science/forschung/open-search https://www.fim.uni-passau.de/data-science/forschung/open-se...)", operated by the University Passau and Open Search Foundation named on your landing page. It would be a mighty big coincidence if this wasn't a project connected to this endeavor, especially when it confirms being an experimental crawler of said project at the UA URL.
jaimex2 4y ago
Wouldn't it be impossible to know if it ignored robots.txt? Just because it crawled it doesn't mean it stored it.
- hrbf 4y ago
  Storage or not is entirely irrelevant to robots.txt directives. It guides automated access. It must be parsed first and excluded URLs must not be accessed at all.

thepangolino 4y ago

dang 4y ago

Url changed from https://www.zylstra.org/blog/2022/09/eu-open-web-search-project-started/ https://www.zylstra.org/blog/2022/09/eu-open-web-search-proj..., which points to https://djoerdhiemstra.com/2022/open-web-search-project-kicked-off/ https://djoerdhiemstra.com/2022/open-web-search-project-kick..., which points to this.

lucideer 4y ago
Which now shows: > Resource Limit Is Reached > The website is temporarily unable to service your request as it exceeded resource limit. Please try again later. Original URL might be more resilient...
- dang 4y ago
  Hmm. I can access the page without that message. In any case the Internet Archive seems to have it: https://web.archive.org/web/20220920183027/https://openwebsearch.eu/ https://web.archive.org/web/20220920183027/https://openwebse...

DOsinga 4y ago

With a budget of 8.5M Eur/Usd. Alphabet spends 200B per year. If 40% of that is spend on search, their budget is 10 thousand times larger.

lucideer 4y ago
It's definitely a comparative underdog regardless, but if you think Alphabet spends anywhere near 40% on search you're out of your mind. I'd be shocked if their spend is double-digits. I'd be unsurprised if it's <1%.
o_m 4y ago
I doubt 40% is spent on search. Seeing how bad Google has gotten, it seems more likely there is just a skeleton crew keeping the lights on
mkl95 4y ago
I would be shocked if Alphabet spent >5% on search. But even 1% would dwarf this project.
mellavora 4y ago
The https://memex.marginalia.nu/ https://memex.marginalia.nu/ search engine runs basically on one desktop computer (if I recall, the creator made a post on this a week or so ago). How much of what google spends on "search" is strictly for search, vs business goals related to search? How much of that google spend is salary? What are those salaries? How do they compare with EU post-doc salaries?

beardedman 4y ago

Oh cool, but do you mean the "EU Open Web Search Data Collection Program"?

notright 4y ago

the EU loves taxing productive companies and wasting said money in stillborn projects that nevertheless promise a kind of bright socialist federalist Europe in their bureaucratic minds

arjenpdevries 4y ago
Isn't it lovely?!
- notright 4y ago
  I am fine as long as they pay for these self-centered utopias with their own money
  - 6510 4y ago
    I'm fine with paying 2 cent for this.
- htkibar 4y ago
  On a side note - how does one get involved with the project should they wish to do so?
Comevius 4y ago
At least we have some of the most livable countries on Earth to show for it. I take taxes over any trickle-down economics, and don't let me stop you looking up the definition of socialist, because you are using it wrong. Besides it's a 8.5 million EUR project, it's literally nothing, it's payroll for a few people. The money is being invested into people who then spend most of it, so it's a triple investment.
qualudeheart 4y ago
Who said anything about socialism? It’s a geopolitical tool to weaken American and Russian influence from Google and Yandex.

ur-whale 4y ago

Looks like it's Northern EU only. No research institutes from {France, Italy, Spain, Greece, Portugal, etc ...} involved.

arjenpdevries 4y ago
Slovenia, Czech Republic. But yes, I think there was a competing proposal from Italy/Spain. Not enough budget for two projects in this area, unfortunately, as they were good too.

dataking 4y ago

I don't see any mention of Quaero, the EU search engine that was supposed to compete with Google [0, 1]. How is this time different? [0] https://en.wikipedia.org/wiki/Quaero https://en.wikipedia.org/wiki/Quaero [1] https://www.dw.com/en/germany-pulls-away-from-quaero-search-engine-project/a-2287489 https://www.dw.com/en/germany-pulls-away-from-quaero-search-...

notright 4y ago
This was the past legislature project. The new legislature brings CHANGE. They are not the same..
arjenpdevries 4y ago
For starters: the objective is to create the index not the engine, that's quite a different ambition. We are very aware of the Quaero/Theseus history :-)
- notright 4y ago
- marginalia_nu 4y ago
  What is the difference?
  - freediver 4y ago
    Supposedely the project is about just building the platform/infrastructure (which is what the index is) upon which search engines can be built. These search engines will then have the freedom to define their own search product experience, business model, even ranking of results.
    - jonas21 4y ago
      So something even more vaguely defined and detached from real use cases than last time? Great.

boyter 4y ago

I have written this before but I’ll put it here again. What I would like to see is a federated search engine. Based on activitypub that works like mastodon. Don’t like the results from one source? Just remove them from your sources, or lower their ranking. Similar to yacy but you can work with the protocol to connect or build whatever type of index you want using whatever technology you like, and communicate over an existing standard. Want to build the worlds best index of Pokémon sites, then go do it. Want to build a search engine using idris or ats? Sure! I did note the professors are on mastodon so perhaps this may actually happen. One of these days I’ll actually implement the above assuming nobody else does. I figured if I can at least get the basics done and a reference implementation that’s easy to run it could prove the concept. If anyone is interested in this do email my in my bio. What I worry about for this project is that it becomes another island which prohibits remixing of results like google and bing, and its own index and ranking algorithms become gamed. I wish the creators best of luck though. I am also hoping for some more blogs and papers about the internals of he engine. So little information is published in the space that anything is welcome, especially if it’s deeply technical.

fabrice_d 4y ago
At least one of the partners (https://openwebsearch.eu/partners/radboud-university/ https://openwebsearch.eu/partners/radboud-university/) does research on "federated search systems", so there's hope!
arjenpdevries 4y ago
We like Federated search, we like decentralized search, and even P2P search; we are trying to find a good mix, and decided to get started rather than wait! Exciting times.
- hkt 4y ago
  I would love to be able to run a node that mirrors part or all of an index like this, and to let people query it - a bit like https://torrents-csv.ml/#/ https://torrents-csv.ml/#/ Good luck! I'll be watching your progress and cheering you all on!
- 4y ago

ur-whale 4y ago

Search is way more than just indexing. I'd really like to see them match the 20+ years of search quality fine-tuning that Google built into their search engine. Not that Google is as good as it used to, but still, catching up with them is way more complicated than just building a big crawl + index piece of infrastructure. And all of that on a government-funded shoestring budget. Mmmh. Good luck to them, but I'm not holding my breath.

Kamq 4y ago
I'd like to see them match the 10 years of fine tuning from 2012 google. My understanding is google used to keep a lot of data around for "long tail" queries, and stopped doing that at some point. This seems to be the issue around search declining outside of "food near me" type queries (which they absolutely excel at).

merb 4y ago

so it began, that sern starts to gather market share. -- I doubt this will take off. I mean they investend more in funding and marketing instead of starting to built something. they should've started with code (agpl3 of course) and invited more and more people. at the moment this is more buzzword bingo bullshit than anything else. it's basically always the same problem, instead of focusing on the product, they fous more on the message.

Animats 4y ago

Early version: https://www.chatnoir.eu/ https://www.chatnoir.eu/

imhoguy 4y ago

"unbiased...based on European values" - will it fly?

topspin 4y ago
European values are inherently unbiased. What's the problem? o.O
- not2b 4y ago
  There's no such thing as an unbiased yet useful search engine. The very act of choosing which article to put at the top is a value judgment, therefore a bias. The only way to make things "unbiased" would be to rank the matches randomly, and that would be useless.
  - xigoi 4y ago
    This is just an index on which different search engines with different priorities can be built.
  - denton-scratch 4y ago
    > The only way to make things "unbiased" would be to rank the matches randomly Well, I would rather like to have visibility into the ranking algorithm, and to be able to control the resultset ordering.
    - not2b 4y ago
      So would the SEO people. But even disregarding that problem, to have a halfway decent search ranking you're going to need deep learning, and you'll get a model that is largely opaque.
      - denton-scratch 4y ago
        Heh! You've mistaken me for a marketer, I think. When I said I wanted control over the ranking, I meant as a user. Tell me what parameters I can rank on; give me a UI that makes it reasonably easy to express a ranking. That's it. I hope UI design and stuff is orthogonal to the construction of an open index. I think this could be very interesting.

logicalmonster 4y ago

What does "based on European values and jurisdiction" refer to? I'd love to be pleasantly surprise, but this sounds like it's ripe for centralized censorship.

notright 4y ago
InTheArena 4y ago
Given the history of the 20th century, this kind of comment promoting European values and jurisdiction seems..... dicey. Companies ethical records, as shitty as they are have nothing on the mass destruction, genocide and stupidity of governments.
wumpus 4y ago
"Right to be forgotten" is an example of a European value.
- denton-scratch 4y ago
  Um, bad example. It was a misconceived bit of legislation. There's no logic behind that claimed right. What happened to my right to remember what I choose? Why shouldn't I publish my memories on a website? Anyway, it's not a "European value", it's a piece of legislation, so it falls under the "European jurisdiction" category, not the "European values" category. That complying with "European values" is part of the project goals is unfortunate. Who decides what are European values? There's no Declaration of European Values. We don't all have the same values. There are just European laws, and then a whole bunch of opinions that not all Europeans agree about.
  - permo-w 4y ago
    people really, truly, go out of their way to criticise the EU. there are things about the EU that deserve criticism, but this is reaching to the point of deliberate misunderstanding. no one is saying that a person who wants to be forgotten cannot be talked about on your website, or that you have to somehow deliberately delete your memories of that person. you are not a corporation. the right to be forgotten is about the right for you to be forgotten by a corporation. the right for you to not be in their database anymore

marginalia_nu 4y ago

I'm a bit skeptical EU-funding a bunch of professors is the way a search engine will be built. The primary goal for academics is to publish new findings, while what you need to build a search engine is rock solid CS and information retrieval basics. Academically, it's not very exciting. Most of it was hashed out in the 1980s or earlier.

margarina72 4y ago
One could have been skeptical about US-funding a bunch of university students to build a search engine a few years ago.
- harha 4y ago
  Search was quite broken back then. It got reasonably good at some point, now I’d argue it’s the content as a whole that’s gotten worse. I’m just skeptical if the EU bureaucrats will put the money in the right place, and if this is even the right approach.
  - margarina72 4y ago
    The project can fail, you are correct, but it does not take anything from any other projects, it is just an other initiative trying to contribute in the space. Parent commenter own search project Marginalia Search [1] could even benefit from it, or even maybe collaborate with it. It is not a winner-take-all situation, and we need various open initiative in this space to get out of the current conundrum we are in with Google stronghold on search. 1: https://search.marginalia.nu https://search.marginalia.nu
    - harha 4y ago
      Taxes are taking from funds for personal projects, and universities already get a lot of funding in Europe to do research, maybe they should focus on creating a better environment for students to become researchers or entrepreneurs instead. Overall I think there are better ways to improve search from an EU perspective by doing what they are supposed to be doing: - create a fair environment for companies to compete in, e.g., take a look how Google, Apple or Meta's assets are set up to make it harder for competitors, break that up - improve standards in eduction – it doesn't really make sense for all member countries to think of and maintain a good CS curriculum and they all seem to be pretty bad at it - make it easier to build something and get funded, and reward creating prosperity, don't tax it to death Just tested marginalia's random mode btw. Pretty cool, reminds me of the internet when I was a kid (edit: formatting)

antics9 4y ago

We need to develop a social aspect to search where results are also moderated and curated by humans in some kind of way.

topspin 4y ago
And when that curation produces results you find abhorrent? What then? Because I guarantee it would; a metaphysical certitude.
- antics9 4y ago
  Do you hang out with abhorrent people?
wumpus 4y ago
Blekko (2007-2015) did that -- apparently it wasn't enough for commercial success.

Extropy_ 4y ago

On first glance, I see the word "unbiased" immediately followed by "based on European values". Now, I'm no expert, but to me, that seems pretty biased.

radiojasper 4y ago
biased on European values
casualrandomcom 4y ago
"being unbiased" can be fairly honestly called a "value". Europeans apparently have chosen it as one of their own values. If you are arguing that being truly unbiased cannot ever be realized, I counter that neither truth nor justice can ever be truly realized, by this should not stop anyone from taking them as their values. signed by: an expert
- denton-scratch 4y ago
  It's not that there's anything wrong with aspiring to be unbiased; the problem arises when someone really thinks they are unbiased, or claims to be unbiased. You get this a lot in the MSM; the Guardian constantly exhorts it's readers to subscribe, to support unbiased journalism. I prefer reporting that wears its biases on its sleeve. Regrettably The Guardian squeezed out most of its most interesting writers, apparently by repeatedly spiking their stories. Yeah, Truth and Justice are platonic abstracts; nobody thinks they exist in the real world. But people do believe that unbiased reporting is possible. It isn't.

deleted 4y ago

[deleted]

s-xyz 4y ago

Correct me if I am wrong, but so the purpose is to create an index database, upon which custom search engines can be attached upon? Ie, the EU will crawl all pages on the web?

murphyslab 4y ago
The index is just the first step according to news articles: > Once the index has been created, the next step is to develop search applications. > The team at TU Graz will be particularly active here in the CoDiS Lab and will work on the conception and user-centric aspects of the search applications. This includes, for example, research into new search paradigms that enable searchers to have a say in how the search takes place. The idea is that there are different search algorithms or that you can influence the behavior of the search algorithms. For example, you could search specifically for scientific documents or for documents with arguments, include search terms that have already been used, or include documents from the intranet in the search. https://www.krone.at/2791083 https://www.krone.at/2791083

margarina72 4y ago

Interesting to see the amount of negative comments. Most of the negativity seems to come from the following points: - EU-funded project cannot succeed in tech because previous EU-funded projects have failed in tech in the past (and generally government-funded project in tech are suspicious) - Search is hard, and therefore it will fail - The project is underfunded Even if the points above can all be valid (though the obvious US-funded startup launched by a bunch of uni students seemed to have fared pretty well) it seems we are missing the point of this project. The proposal is to contribute to the creation of open building blocks necessary to enable others (including private US companies) to make better search products. Better search product are needed. Shall we remember HN of some of the intense conversations that happened here about Google failing us: - Google Search is Dying [1] - Every Google result now looks like an ad [2] - Google no longer producing high quality search results in significant categories [3] So while, yes search is a hard topic, we should welcome initiatives aiming at improving the ground infrastructure needed to lower the barrier to entry on this subject and hope it will allow many companies to build better search products (or inspire other initiatives to contribute in similar and even more successful way) 1: https://news.ycombinator.com/item?id=30347719 https://news.ycombinator.com/item?id=30347719 2: https://news.ycombinator.com/item?id=22107823 https://news.ycombinator.com/item?id=22107823 3: https://news.ycombinator.com/item?id=29772136 https://news.ycombinator.com/item?id=29772136

Proven 4y ago
[dead]
ffhhj 4y ago
If these thinkers can find a way to remove the incentives of spammers, misinformators, and stakeholders to pollute results, that would be a great achievement. It could be seen as a big economic game, simulating these actors might allow comming up with rules to balance this game and minimize pollution.
- petre 4y ago
  It might be possible using moderation and reporting abuse or registering with the search engine to aid moderation and banning abusers. This will probably also take down right wing websites, porn and online gambling.

hexage1814 4y ago

Will it censor the so called "hate speech"?

speedgoose 4y ago
I have never seen “hate speech” that wasn’t hate speech.

luis8 4y ago

Would it be hard to expand it as global endeavour?

solardev 4y ago
Governments would never agree on what to show or hide.

fspeech 4y ago

A crawler should see the web pages as a human would and read the pages as a human would and extract keywords based on the real displayed content. After that the search could be as simple as looking up a book in your library catalog. Think liberally about the search key, e.g. "<40% screen is display ads". If the search engine gets popular, you can then worry about scaling up to serve the results faster. Oh PageRank (R) can be done using real semantic links (actual human comments) rather than easily spammed hyperlinks. And you can group all similar pages and serve those as a single expandable/refinable entry.

alkonaut 4y ago
Yes, telling spam (spam blogs, copy-pasted versions of Stackoverflow, etc) from real organic content is easy for a human, so I can only assume it would be pretty easy for a computer too. At least until this becomes a practice that spam site makers have to consider and adapt. For example, given 10 similar articles where one has a well-known domain but few links and the others are all cross-linked only among a known spam-net, then just show the well known page (stackoverflow) and not the useless copycats. It seems doable for a computer to also navigate the page like a human and try to negotiate various cookiewalls and similar. If a page requires jumping through hoops to enter, rank it poorly. There should be a single decline/reject if there is a button at all, and the content ranked should be the content shown after rejecting. If it shows two buttons "Accept" and "Options" then just don't index that page.
- denton-scratch 4y ago
  > If a page requires jumping through hoops to enter, rank it poorly. Oh, how I wish that search engines would simply give up on encountering a paywall/cookiewall, and refuse to index the page.

psuresh 4y ago

Good to see more efforts from EU to negate big tech domination. Federation is the way forward.

tommica 4y ago

I really hope this project is successful - the heart seems to be in the right place!

tyiz 4y ago

Galileo, anyone? The EU is the trash bin of politicians who failed in their respective country. Not elected and saturated with piles of money. It is under control of the American Empire, thus designed to fail with a push of a button if necessary (€ is the key). Wake up!

qualudeheart 4y ago

This is an excellent idea to disintermediate Google. A search tool for the commons without the spyware and opportunities for search manipulation is very important, and the more actors work on it, the better. Perhaps cooperations with other tech savvy democracies could occur. India is full of skilled programmers. Unlike Russia or China it is easy to see Europe cooperate with it.

denton-scratch 4y ago
Hmm. India also has a government that is borderline fascist, promotes ethnic and religious hatred, and doesn't seem to believe in internet freedom (they keep shutting it down). And it's not the lack of skilled programmers that makes Western governments wary of cooperating with China and Russia. China and Russia have totalitarian governments that are terrified of the internet, and would like it to go away. I wouldn't want to include them in a project to make the internet work better.

jhoelzel 4y ago

I agree that we are sadly ending the chapter of free search but im quite sure this is just another roll call for some professors to make a quick buck like it always is. Search needs help its true, but 8 million are not even gonna move the needle and that other engines have that much money invested, just to maintain their network switches.

stiray 4y ago

What personally worries me the most is "Meet the partners" page. I didnt look for all but I did look who is partner from Slovenia. The most privacy invading ISP/Mobile company in our country selling statistics about their users (although anonymized, but walking on a thin line). I just hope it wont go down that drain. Regarding all the negativity about the government project. If we can run CERN we can surely do a web search project.

berry_sortoro 4y ago

Roark66 4y ago

Seriously... I really wanted to like this project, but it seems everything EU touches as of recently gets worse. From the webpafe that half of the time shows "Resource Limit exceeded" to a technology stack diagram on the bottom of this page https://openwebsearch.eu/the-project/ https://openwebsearch.eu/the-project/ being completely unreadable due to bad scaling. It is very disappointing really. Another example from the top of my head. Here in Poland we have ID cards(as every other EU country) . Those ID cards have to be renewed every now and then (10~15years). In last years an online system for government services was implemented including for renewal of those cards. One could take a photo with a mobile phone, submit an application and pick up a card from a gov office in few weeks time. Unfortunately, EU made a law that ID cards applications have to be acommpanied by biometrics (fingerprints) so this system has been thrown away. One has to physically go to the gov office, scan their fingerprints, apply for a new id card and then go again to pick it up... Ok, so what happens in 10 years time? They should have the fingerprints already, right? No. They take the fingerprints, they store them only until one picks up the id card and then they are deleted. There are no fingerprint database, they are not stored anywhere. The fingerprints are used only to ensure the same person that submitted the application picks up the document. It makes zero sense, other than to break the previous online system. Thanks EU.

ssmiler 4y ago
And why the hell the government will keep the fingerprints? So that once in 10 years I save 1h. The benefit is minor compared to bad things that can be done with a fingerprints DB (mass surveillance..)
- jld89 4y ago
  > And why the hell the government will keep the fingerprints? The government already does... just saying.
  - simiones 4y ago
    They just explained that it doesn't (assuming we trust them, of course).

hericium 4y ago

Is there a privacy policy in English? The only one I could find[1] is in German. [1] https://openwebsearch.eu/privacy-policy/ https://openwebsearch.eu/privacy-policy/

sazz 4y ago

Before Corona, I would have really welcomed this announcement. But to be honest, when someone says that the project will be "based on European values," I think it's stillborn. Who is defining those "European values"? Is it the European Commission? As a liberal person who still dreams of mature citizens who form their own opinions well-informed from a rich debate, I now see the "European values" quite critically. In Germany - if you criticize the Corona measures you are called a Nazi, if you criticize the Ukraine war you are a Russian troll. Are those then the "European values"? The European Commission has already established some projects to weight the information according to its will (SocialTruth, PROVENANCE, EUNOMIA, etc.). When a government agency talks about truth, then all the hairs on the back of my neck stand up. When governments speak of disinformation, it is usually only in the sense that the information does not correspond to their interests. I had to learn that the so-called fact checkers don't check facts, they just sell a counter opinion as a "fact". So for a European search engine, a filter is then placed upstream that filters out all disinformation? In the past, that was called censorship. Today, it's more like citizen service.

johnywalks 4y ago
> If you criticize the Corona measure you are a Nazi, if you criticize the Ukraine war you are a Russian troll? These notions are propagated by the US to the rest of the world. While EU politicians seem to go along, europeans do not tolerate it much and that much is reflected in society. If the EU or more gov sponsored search engines pop up, i have no doubt they will want to control them in their own way - i'm okay with that. Right now our only option is US controlled entities that answer to US govs and rules. At least here we can point fingers and hold politicians accountable if they try to influence things the wrong way. We can't do that with US companies.
- sazz 4y ago
  > At least here we can point fingers and hold politicians accountable if they try to influence things the wrong way. We can't do that with US companies. Germany is currently governed by a chancellor who has in fact been convicted of lying, has selective memory lapses as a co-responsible party in one of the biggest banking scandals, and was also instrumental in the political decisions to push ahead with dependence on Russian gas. So sorry if I doubt that politicians can be made accountable for anything.

reacharavindh 4y ago

I wish a project doesn't try to moonshot a thing that we already had.. I want the good old Google search where you type something, and all relevant web result pop out (without creepy ads, trackers, personalisation, shopping links to get you to spend). I understand it is very expensive to crawl the web, that's where an org like the EU can come in. I wish they bootstrapped a simple, open index that any startup can use to provide clean search fronts. Crawling the web, and indexing content can be quite expensive, if EU bootstrapped that infrastructure, and then say augmented it with federated crawling and indexing (sort of like the distributed computing MIT condor project), where people, Universities etc in the EU could contribute their spare computer time to crawling the web and indexing it for the EU index, making search better for everyone. Heck, I'd even put solar panels in my bike shed and hook it up to a PC to crawl the web while the sun is shining. I'm dreaming too much, I need coffee..

deleted 4y ago
[deleted]
mjburgess 4y ago
But the web is different; google much less so. There is no "old google search", because almost everything to search through is noise and spam. PageRank made user preferences as signalled by hyperlinks the key signal of quality. Who really hyperlinks any more? And how many, by percentage, are non-automated etc.? The web this works for is one of forums & personal websites. It's hard to say that today there are any properties of websites that are a reliable signal of quality. Hence the proliferation of voting sites (such as HN, etc.) which are little more than search engines augmented with reliable signals of user preferences.
- permo-w 4y ago
  >But the web is different; google much less so. There is no "old google search", because almost everything to search through is noise and spam the commenter is not referring to the search results, they're referring to the interface not having all the wank that google packs in to optimise for profit >Who really hyperlinks any more? who doesn't?

paul-j_ 4y ago

I hope it will have proper regex capability for narrowing the search results to more relevant content.

SirOibaf 4y ago

GAIA-X, is that you?

nickdothutton 4y ago

Some of you might be interested in the privacy focussed "French Google" Qwant. https://en.wikipedia.org/wiki/Qwant https://en.wikipedia.org/wiki/Qwant Plot twist: It's funded by Huawei.

sylware 4y ago

??? Where is the EU own and operated 2nm foundry?

denton-scratch 4y ago

I couldn't work out what "building blocks" they're going to produce. Are they going to produce a public index, that search engines can use? A set of APIs? > the researchers will develop the core of a European Open Web Index (OWI) as a basis for a new Internet Search in Europe. That sounds as if building an index is at least within scope. But isn't the design of the index one of the key differentiators between different search engines? Perhaps the OWI will just produce tools for building transparent, privacy-preserving search engines, rather than crawling the web itself. It's not clear (to me) from reading the site.

arjenpdevries 4y ago
The project is starting so not all your questions are answerable today, but, we definitly will produce an open web index, already by the end of the first year, with improvements for years two and three. We further deliver components to make search engines on top of this index. The project vision is that there will be many different search engines, not just 4 worldwide. Hoping to lead the way!

zerof1l 4y ago

It doesn't sound open and unbiased to me, because it will allow the government to censor on an index level.

frozencell 4y ago

> Unbiased One of the tech lead feed [0] looks quite biased towards Ukraine though. I hope it doesn't interfere with the search engine. [0] https://twitter.com/mgrani https://twitter.com/mgrani

tristanb 4y ago
There is no bias. Russia is a terrorist state that has brutally and illegally murdered civilians in a country they chose to invade. Hopefully it will be their continued downfall.
- frozencell 4y ago
  All countries are terrorist states if we look at it this way.

XCSme 4y ago

> Free, open and unbiased access to information How can something be unbiased yet somehow stop showing illegal search results?