4 ms·
I always wonder if the days of search engines for specific topics could return. With LLM's providing less than accurate results in some areas, and Google, bing,
by ktallett 1y ago
I always wonder if the days of search engines for specific topics could return. With LLM's providing less than accurate results in some areas, and Google, bing, etc being taken over by adverts or well organised SEO, there feels like a place for accurate, specialised search.
- datadrivenangel 1y agoThe curation of an index of resources is what's needed for niche search
- dcist 1y agoWestLaw and Lexis Nexis provide this for legal search, but quite frankly, these services are subpar. It's amazing that these two companies rake in hundreds of millions but they are both slower than Google, Bing, Yandex, or any LLM service (ChatGPT, Claude, Gemini, etc.) while scouring a universe of text that is orders of magnitude smaller. The user experience is also terrible (you have to login and specify a client each and every time you attempt to use the service and both services log you out after a short -- in my opinion -- period of inactivity, creating friction and needless annoyance to the user). There's an opportunity there.
- ktallett 1y agoI haven't personally used the mentioned services as they aren't in my field, however what is the accuracy of their results? Are they double checked? I don't find LLMs particularly accurate in my field (that's being kind), if anything I find they make up sources that simply don't exist. I mean poor UX has no excuse but slow speed can be reasoned if it makes the quality of the service better.
- ordersofmag 1y agoHere’s a place to start if you want to go down the rabbit hole of how search at places like this is approached. https://haystackconf.com/us2022/talk-12/ https://haystackconf.com/us2022/talk-12/ https://www.youtube.com/watch?v=9vCMFIJRiKk https://www.youtube.com/watch?v=9vCMFIJRiKk
- ahi 1y agoLN and Westlaw's real service is their ubiquity. Every law student has access to it and every firm expects proficiency. While they generally suck, the last time I used it (looong time ago), their boolean search was quite nice. That kind of text search has mostly been replaced by non-deterministic black boxes which aren't great for legal research.
- piker 1y agoYou forgot to mention their claim of copyright over the bulk of, e.g. obscure state case law.
- ehecatl42 1y agoSo, you have to pay to access the law that you are subject to?
- piker 1y agoIf you want it digitized, yes, odd as that seems. You can go find individual prints of it or perhaps digital copies of opinions elsewhere, but those are also technically copyrighted in a lot of cases too.
- jfil 1y agoIn some jurisdictions, like Ontario, there are secret agreements that only allow 3 organizations to have digital access to Case Law (https://www.cameronhuff.com/blog/ontario-case-law-private/ https://www.cameronhuff.com/blog/ontario-case-law-private/). This says a lot about our society, and how much we still have to improve.
- throwup238 1y agoThey've also got the Microsoft effect going on. Usually at least one of their products like their personal information aggregator used for locating people (like when serving lawsuits) is mandatory for a firm so it's just easier for them bundle everything else in.
- raydenvm 1y agoWhich is not scalable, right?
- deleted 1y ago[deleted]
- cosmicgadget 1y agoIt's scalable if you are okay with not searching exhaustively.
- cosmicgadget 1y agoMy hope is that content self-indexes so instead curation it just has to be aggregated.
- econ 1y agoDepends how tiny the niche is. A few dozen domains is easily done by hand and worth having.
- cyanydeez 1y agoi know the answer is never distributed services, but if one could build a sufficiently complex SDK to make like a Blue Sky but for niche search indexes, you could chain a bunch of vetted resources together.
- fanwood 1y agoI already directly search on Wikipedia for most topics (with a search shortcut on URL bar)
- ktallett 1y agoWikipedia is useful up to a point for sure. I feel whether it could be a expansion of Wikipedia in it's current use case, but for emerging research and niche topics it can sometimes be less useful.
- wolfgang42 1y agoYeah, the (relative) rise of Kagi and Marginalia show that from a technical perspective, this is within the grasp of a dedicated hobbyist.[1] If Google continues their current trajectory, and overwhelming numbers of AI crawlers don’t cause an unsurmountable rise in CAPTCHA pages, I hope to see an upsurgence of niche search engines that focus on some specialty small enough that one or a few people can curate the content and produce a much better experience than the current crop of general Web search engines. Self-plug: I run such a search engine (for programmers) in my living room, at <https://search.feep.dev/ https://search.feep.dev/>. I don’t spend a ton of time maintaining it, so I’m interested to see what someone really dedicated could do. [1] I wrote a 2004-vs-2014 comparison, and things have only gotten better since then: https://search.feep.dev/blog/post/2022-07-23-write-your-own https://search.feep.dev/blog/post/2022-07-23-write-your-own
- iLoveOncall 1y agoPlease, Kagi doesn't even have 50,000 active members, it's definitely not "rising" to become a serious contender at any sort of market share, it's a micro-project. You just feel it's bigger than that because for some reason all of its 50,000 users post relentlessly about it on HN.
- wolfgang42 1y agoHence the (relative), yes. Did “dedicated hobbyist” not tip you off that I wasn’t thinking about how to maximize market share?
- cyanydeez 1y agoJust gotta build a search engine that properly contextualizes scams, bait & switch sites, SEO, and the rest, and you're back in business. To do that, you probably still need humans to properly curate the dataset, essentially hire 100 librarians and setup a work flow for them to continually prune results. Right now, everything is all batch processes. None of these LLMs use active feedback since there's no real models using updates.