4 ms·
To me what this shows is that it's really painful to try and use an "intelligent" algorithm for "content mediation" as if it were an actual search engine. Anyt
by photonthug 3y ago
To me what this shows is that it's really painful to try and use an "intelligent" algorithm for "content mediation" as if it were an actual search engine. Anything that spectacularly fails the "exact substring match" use-case is well on the way to becoming a content-mediator, and potentially as bad a content-moderator/thought-police/etc.
Unfortunately even before anyone started talking about "semantic web" or whatever, substring-match was declining in utility due to spam, so a substring-matcher was only as useful as it's ranking algorithm. So some cleverness in ranking is necessary, but when ranking becomes too clever, or irrelevant due to the arms race with spam, it's an almost inevitable slide from necessary rankers to mediators.
Since all the search/spam arms race is pretty much played out with giants like twitter/google anyway at this point, it is kind of hard to understand why we can't get and keep a separate and truly basic internet search engine that just works. I don't have a handy example, but in my experience even DDG & friends seem to routinely fail the "exact substring match" test for unusual phrases that I know I read verbatim last week.
What I (and I think others) really want is just substring-match tool plus a very minimal amount of extra cleverness for basic stuff like spelling variations, some fuzzing for dates/digits, directly adjacent stuff according to a simple thesaurus.
Not my area of expertise, but I would guess that there's a combination of problems that prevent this tool from existing.
a) Other search companies are all trying to do fancy-but-fuzzy "semantic distance" searches to emulate Google
b) It's still true in 2023 that just keeping an up to date index is a super hard problem no matter how many crawlers are in your army
c) Despite bots everywhere, the deeper web in social-media is just too login-walled for the rule-abiding crawlers to deal with
d) There's just no money in do-what-i-say-not-what-i-mean search tools
- galangalalgol 3y agoDDG seems unusually susceptible to highly ranking AI-generated overly verbose garbage. I can't recreate the same issue with google or bing, so I am not sure where it pulls it from. I can find those same results in google, it's just they are on the 2nd or 3rd page.