5 ms·
Ask HN: What would make a next-gen search engine?
With new search engines like you.com popping up and Google facing regulatory scrutiny, it's an interesting time to consider what a next-gen search engine would look like.
What unsolved problems would a next-gen search engine potentially address?
What if you could have a conversation with a search engine, asking follow-up questions within the context of the previous ones?
- ggm 5y ago* Tree of results not list. Group by functions. * Confidence ranks you can interrogate. "Why did you rank high?" * on the fly refinement of specific hit terms. "Find more like this"
- RileyJames 5y agoWhat I’d like in a search engine? I’d like existing engines to be sources / indexes, upon which I’d add collaborative curation. I think uBlacklist is a good implementation. Braves proposed “goggles” is better, but as yet I don’t believe there are any implementations of it. [2] [1] https://github.com/iorate/ublacklist https://github.com/iorate/ublacklist [2] https://brave.com/wp-content/uploads/2021/03/goggles.pdf https://brave.com/wp-content/uploads/2021/03/goggles.pdf
- throwaway81523 5y agoHave a way to discard all results that don't contain your search terms.
- tboyd47 5y agoSearch engines that are extremely good at finding info on a specific thing. Search engines that filter out websites that don’t follow a certain code of conduct. Search engines that are local to a certain area. Search engines that are completely open-source.
- altdataseller 5y agoElasticsearch/OpenSearch is open source and can be a search engine.
- jmakov 5y agoCheckout http://tehnologije.ijs.si/TT09/TT-1th-October/Participants%20Ideas/SearchPoint.ppt http://tehnologije.ijs.si/TT09/TT-1th-October/Participants%2... and you.com
- gkjrnmtmt 5y agoRelated question. How can an individual start building a new search engine without initial seed funding? Bing’s free search tier is very easy to exhaust. My idea is to use some else’s search results and remove results that don’t fit my criteria
- phenkdo 5y agohttps://commoncrawl.org/ https://commoncrawl.org/ might be a good place to start.
- fuzzfactor 5y agoShould have an option to rank sites according to their lack of advertising.
- lkrubner 5y agoHistory. It’s been documented that Google no longer tries to index everything on the Web and is allowing older material to disappear. Tim Bray has a blog post about this. A full history would be useful, also the ability to run a search with the Page Rank of a given day. For instance, what site was the most linked node on July 19 of 2011?
- marginalia_nu 5y agoTo be fair, you can't index everything on the web, due to dynamic page generation and wildcard subdomains, web is effectively boundless. You can have an URL http://integer.example.com/N http://integer.example.com/N only that contains a link to http://integer.example.com/N+1 http://integer.example.com/N+1 All search engines need to make concessions. The tricky part is identifying the sorts of content that humans want to find.
- onionisafruit 5y agoI think the biggest challenge for a search engine is getting the corpus to search — the crawling and scraping. A collective of search engines could go in together on that together. Then they could go their own way on indexing it and making it useful.
- dredmorbius 5y agoOne possible solution to this would be for sites to offer their own indexes. If you think about it, search is an emergent sector that developed because there was no standard for sites to index themselves. Compiling that index and providing that would be far more efficient than trying to crawl billions (or is it trillions) of pages independently by multiple individual search engines, and would remove most of the incumbent advantage of such platforms. Instead we ended up with a few pieces of this --- robots.txt and sitemaps --- but not a full solution. Yes, you'd need to check for cheats (sample pages based on the index and look for signs of stuffing or misrepresentation, penalise accordingly). Reputation systems were based on inbound and outbound links, so those might also be offered. I'm not entirely sure this would work, but I'd be interested in seeing the concept fleshed out further.
- onionisafruit 5y agoThat’s an interesting point about search developing in an absence of a standard for sites to self-index. I can’t quite imagine what that standard index would look like.
- dredmorbius 5y agoIt would probably look much like the search index that search engines use. These consist of both individual terms and tuples. (Phrase search can be constructed reasonably reliably by chaining ngram tuples.) Rather than transferring the entire website, it would only be necessary to transfer the index itself. Mind that a sufficiently-large ngram-tuple index is itself large, I suspect limits to 3-word ngrams might be necessary, and should be reasonably sufficient. I'm kicking around the prospect of a DNS-like query mechanism, with cache-and-forward capabilities. That could both distribute search and indexing load. Reputation assessment and management remains the hard piece. Early computer science focused on the relatively easy problem of string search. At Web scale, the problem is of idnetifying reliable and useful sources. That itself isn't a static problem with a single solution as the landscape evolves in response to methods.
- dredmorbius 5y agoNB: I really like the follow-up question concept, though I'd like to see what you have in mind for that. Usually that's something I accomplish by adding or removing terms / conditions to my query. That's often useful, but frequently not, and remove classes of results (e.g., commerce sites, social media, SEO / content farms) is often a frustration. Poorly-detected date ranges would be another. User-definable ranking and exclusion criteria would be great. In the past few weeks on HN we've seen articles on: - How universally reviled autoplay video is. Excluding sites which do this from SERPs would be a strong incentive. - Paywalls and privacy invasions, similarly. - Known SEO-baiting sites, e.g., Pinterest and Quora. - Excessive JS. - User-hostile designs. Simple site quality and reputation would be a huge factor for me. I've increasingly taken to searching specific sites rather than general Web search just to be able to cut through the crap. Suggesting filters to apply might also be of interest --- say, "X filter returns / removes Y results". As noted in an earlier comment: establishing a self-indexing search standard, which would allow websites to create their own indices, and for those to be distributed to multiple search platforms. (Yes, cheats would need to be identified, and mechanisms for establishing reputation determined.) Better metadata search, and inclusion / exclusion by category, would be great. Search exclusive to scientific, technical, or academic sites, or exclusive of commercial, paywall, erotic, gaming sites, for example. (Such filters could of course be reversed if that was your kink.) Inclusion/exclusion of social media is probably another big one. Tools for leveraging site-specific search more cleanly, inspired by DDG's Bang! searches, might also be interesting. Really good date-ranged search. Google still beats DDG at this, though DDG can at least filter by past day/week/month/year, which is useful. Search inclusive of the Internet Archive's Web and other holdings would also be great. A search not just of Web space but of Web time. A true news / magazine archive search. Governmetnt records searches.
- marginalia_nu 5y agoI've experimented with labeling search results that contain js, amazon-links, video, analytics/tracking, etc. on search.marginalia.nu; you can also require/exclude results by those standards. Seems to work reasonably well, even if there are some false positives. > Better metadata search, and inclusion / exclusion by category, would be great. Search exclusive to scientific, technical, or academic sites, or exclusive of commercial, paywall, erotic, gaming sites, for example. (Such filters could of course be reversed if that was your kink.) Inclusion/exclusion of social media is probably another big one. I would like to do this, but I'm not quite sure how I'd implement the labeling. .edu and .ac.xx works for academia for the most part, but the rest is a lot harder. Could possibly be done approximately using graph adjacency, but I don't know how you'd get a true-false value out of it...
- manx 5y agoI'd like to see a modern web directory, that is built automatically, but can be corrected collaboratively. It should still feature search that finds websites or categories in the directory, but also allows to explore from these points and discover related content.