6 ms·
The consistent theme every time this comes up is that dealing with the sheer weight of the internet is almost impossible today. SEO spam is hard to fight and th
by lolinder 5y ago
The consistent theme every time this comes up is that dealing with the sheer weight of the internet is almost impossible today. SEO spam is hard to fight and the index gets too heavy. However, I wonder if this is a sign that we're looking at the problem wrong.
What if instead of even trying to index the entire web, we moved one step back towards the curated directories of the early web? Give users a search engine and indexer that they control and host. Allow them to "follow" domains (or any partial URLs, like subreddits) that they trust.
Make it so that you can configure how many hops it is allowed to take from those trusted sources, similar to LinkedIn's levels of connections. If I'm hosting on my laptop, I might set it at 1 step removed, but if I've got an S3 bucket for my index I might go as far as 3 or 4 steps removed.
There are further optimizations that you could do, such as having your instance not index Wikipedia or Stack Overflow or whatever (instead using the built-in search and aggregating results).
I'm sure there are technical challenges I'm not thinking of, and this would absolutely be a tool that would best serve power users and programmers rather than average internet users. Such an engine wouldn't ever replace Google, but I'd think it would go a long way to making a better search engine for a single user's (or a certain subset of users') everyday web experience.
- throwaddzuzxd 5y agoI wonder if we could use some kind of federation (ActivityPub?) to build an aggregate of the search indexes of a curated community. Something like a giant federated whitelist of domains to index.
- skyde 5y agowhat you are suggesting would make the problem of echo chamber (bubble) worse than it is today!
- Nasrudith 5y agoAwkwardly complaints about echo chamber as a problem tends to not refer to feedback dynamics (crudely but disambiguating refered to as circle jerk) so much "People disagree with me, the nerve of them!". It is not viable to have parties A through Z sharing the same world and all having absolute control over all others. We see this same complaint every time modernation comes up, let alone the fundamentals of democracy.
- loonster 5y agoBubbles are great if you are on the outside looking in at how a specific group thinks. Bubbles are horrible if you are on the inside trying to explore your thoughts.
- lessname 5y agoThis might work well in some situations (e.g. research, development), however it would also increase the effect of echo chambers I think.
- lolinder 5y agoPossibly, but I'm not convinced. Google's not exactly working against the echo chamber problem, and I think that's because to do so would be to work against its own reason for existing. There are two goals here that are fundamentally at odds with each other: 1) Finding what you're looking for. 2) Finding a new perspective on something. A search engine's job is to address the first challenge: finding something that the user is looking for. The search engine might end up serving both needs if they're looking for a new perspective on something, but if these two goals ever come into conflict with each other the search engine does (and I would argue it should) choose the first goal. Failing to do so will just lead to people ignoring the results.
- theduder99 5y agoecho chambers are what most people want :)
- dougSF70 5y agoecho chambers are what most people want :)
- jonathankoren 5y agoPart of the thing with echo chambers is that the search terms themselves can be indicative of a particular bubble. For example, there's a difference in the people that refer to the Bureau of Alcohol, Tobacco, and Firearms by the official initialism, "ATF", and those that use "BATF". There's a strong antigun control bent to the `"BATF" guns` query, compared to the `"ATF" guns` query. If you're indexing forums or social media, the same site is going to give back the bubbled responses, possibly without the person even being aware they're in a bubble. https://www.google.com/search?q=%22BATF%22+guns&client=safari&rls=en&sxsrf=AOaemvJS_Q3Nc1e40-VcQYu2xvyRTabBBQ:1638485361910&ei=cU2pYd3vNo65qtsPnZyYkA4&start=10&sa=N&ved=2ahUKEwid7NzPmcb0AhWOnGoFHR0OBuIQ8tMDegQIARA8&biw=967&bih=756&dpr=2&safari_group=9 https://www.google.com/search?q=%22BATF%22+guns&client=safar... https://www.google.com/search?q=%22ATF%22+guns&client=safari&rls=en&ei=z02pYZb0GY_PtQaTvrnADA&start=10&sa=N&ved=2ahUKEwjWl6n8mcb0AhWPZ80KHRNfDsgQ8NMDegQIARBJ&biw=1280&bih=970&dpr=2 https://www.google.com/search?q=%22ATF%22+guns&client=safari...
- lawwantsin17 5y agoI'm sure the algorithms are making echo chambers worse. Curating news opinion sites based on a prediction score of how often Chicken Little was right about the sky falling after the fact would surface reliable journalists and actual psychics!
- Nasrudith 5y agoThe retro idea of curation seems popular here but everybody forgets why it lost out in the first place. It just doesn't scale in the first place. Not to mention demands - people usually want tools which lower mental effort and are intuitive as opposed to ones which are precise but in an obtuse metric. Most would not find a hardware mouse that consisted of two keypads for X and Y coordinates and a left click and right click button very useful. Similarly everyone maintaining your their own index is cumbersome overkill in redundancy, processing power, and human effort in return for a stunted network graph which is worse for all metrics people usually actually care about. In terms of catching on even "antipattern search engines" that attempt to create an ideological echo chamber would probably catch on better. Short of search engine experiments/start up attempts the only other useful application I can see is "rude web-spidering" which deliberately disrespects all requests to not index pages left publicly accessible as search engines generally try not to be good tools for cracker wardriving for PR and liability reasons. It would be a good whitehat or greyhat tool as doors secured by politeness only are not secure.
- mclightning 5y agoThat's basically what I'm doing with my search "site:reddit.com" I wonder if anyone at Google is aware of this trend and taking notes.
- a-r-t 5y agoReddit is missing a huge opportunity by not improving their crappy search functionality.
- copperx 5y agoI estimate that about half of my searches have either site:reddit.com or site:news.ycombinator.com at the end. In fact, I have an autocomplete snippet on my Mac so I don't have to type all that.
- marksbrown 5y agoFYI this is exactly what the hashbangs in DDG do!
- supernovae 5y agoIt's flawed from the get go if reddit is the basis.
- loonster 5y agoAs much as I like to hate on reddit (I'm a permanently suspended user), not every sub there is trash. There are some great subs there on very specific niche topics.
- hardlianotion 5y agoI'm a permanently suspended member too (permanent for technical reasons), and I have never posted on there.
- myridium 5y agoBadge of honour I'd say. What was your transgression?
- loonster 5y agoSomeone asked about the Hunter Biden files. I responded with g n e w s . c o m . It took a few weeks, but they finally found it and suspended me for it. Others they suspended for mentioning the news organization that mentioned gnews.
- djwayne35 5y agoI agree, I think we are looking at the problem wrong. And this is a very insightful comparison with the linkedin levels of connections idea. I am working on something with this. One thing to point out is that when we think of searching through information, we are searching though an information structure aka graph of knowledge. Whatever idea or search term we are thinking of is connected to a bunch of other ideas. All those connected ideas represent the search space or the knowledge graph we are trying to parse. So one way in the past people have tried to approach this is they try to make a predefined knowledge graph or an ontology around a domain. They try to set up the structure of how the information should be and then they fill in the data. The goal is to dynamically create an ontology., Idk if anyone has really figured this out. But, Palantir with Foundry does something related. They sorta dynamically create an ontology ontop of a company's data. This lets people find relationships between data and more easily search through their data. Check this out to learn more https://sudonull.com/post/89367-Dynamic-ontology-As-Palantir-engineers-explain-this-to-the-CIA-the-NSA-and-the-military https://sudonull.com/post/89367-Dynamic-ontology-As-Palantir...
- hyperpallium2 5y agoI like the idea of a subset of the web, and for a niche purpose. Not sure about user-hosting. Capital is the huge barrier to entry today: Larry Page's genius was to extend google's tech, consumer-habit and PR barriers-to-entry into a capital-based advantage: massive geo server farms, giving faster responses. Consumers have a demonstrated huge preference for faster response.
- le-mark 5y agoI’ve often thought Alexa.com top n sites Would be a good starting point.
- kokanee 5y agoWhat if we allow users to upvote and downvote search results. Too many downvotes and you get dropped from the index.
- krapp 5y agoCompanies will simply hire people or purchase bots to downvote their competitors and upvote themselves, and then an entire economy will develop around gaming search engine algorithms, so that eventually search results will be completely useless. Basically, SEO. SEO is the real problem, not search engine algorithms. Those algorithms are a result of the arms race between Google and black-hat SEO BS. Remove SEO and search engines work just fine.