4 ms·
Free project idea for someone with more time on their hands than me: Classical search engines determine trust automatically, based on various factors including
by brundolf 5y ago
Free project idea for someone with more time on their hands than me:
Classical search engines determine trust automatically, based on various factors including "link neighborhoods" where trustworthy sites link to other trustworthy sites. These automated strategies are clearly breaking down; the spammers are winning the arms-race.
So maybe we need to go back to human-based trust.
People used to curate lists of websites, which partly solved this problem but didn't necessarily scale. I wonder if that idea could be supercharged.
Consider a browser extension that people install, which:
a) gives users a button to mark a site as trusted/favorited
b) tracks domains visited (and frequency)
Then, separately, you can manually add people you know personally to your "network". You trust them, so anything they trust is also something you might be able to trust. Manual favorites could be weighted higher than frequently-visited sites, and both could be displayed inline next to links on all pages you visit. You could also see which people the trust in a given link comes from, in case some of them consistently have bad judgement about these things and you want to remove them from your list. Then, finally, you could create a personalized search-engine that only indexes the sites determined to be trusted by your personal network.
Of course this would require placing a great amount of trust in the extension and service themselves, so maybe they would have to be open-sourced or self-hostable or something (a profit motive might create a huge amount of temptation to abuse the data). That's a stickier problem.
Edit: There was a little ambiguity left here about transitive trust; “friends of friends” type stuff. I think if this went on for unlimited hops, we’d be back at square one. So maybe it only uses direct contacts, or maybe some small N of hops (where longer ones are weighted lower?). Maybe this would be configurable, not sure.
Also re: privacy, maybe you could come up with a clever way to E2E encrypt the site visit data, even though it’s shared with many parties?
- sdeframond 5y agoInteresting. Another "sticky" problem is how to make a living out of this I guess...
- rsyring 5y agoHave the extension serve ads.
- Seirdy 5y agoThere's no need for an extension; plenty of websites are part of webrings or feature blogrolls. I'm in the process of adding one to mine. Throw in some microformats2 and/or schema.org structured data and you're good to go. Certain search engines specialize in this type of manually-curated content; I listed some in the "non-generalist search" section of my collection of indexing search engines: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html https://seirdy.one/2021/03/10/search-engines-with-own-indexe...
- whoibrar 5y agoI've just had the deja Vu reading this post! Took me a minute to realise, I did actually come across this post and your blog some time ago, and really enjoyed reading all the posts then and now again! Hoping to read more from you!
- eitland 5y agoSomeone will sooner or later mention the sybil attack. (At some points I feel it is thrown out as haphazardly as "correlation does not imply causation".) But I think you might be onto something. It won't necessarily be easy but I think it deserves more than a quick dismissal.
- throw10920 5y agoGP comment already explicitly addressed this: > Then, separately, you can manually add people you know personally to your "network". You trust them, so anything they trust is also something you might be able to trust. It doesn't matter how many sockpuppet accounts a marketing company creates if you don't click the "trust" button on any of them, and if the system is designed so that your local trust network only consists of what you trust directly and (weighted) transitively.
- foxfluff 5y agoI've been wanting[0] more or less the same for a long time now. Ideally the dataset should be public (and distributed) so that you could find bad actors and eliminate them from your network (i.e. when you see a bad result, you can figure out where it came from into your feed), as well as run your own tweaked algorithms. WoT alone is great, but it gets better still when you can tweak the algorithm. At that point, it doesn't need to be people you know personally; it just needs to be identities with a proven track record. If you can manage multiple identities, you could make ones e.g. for the explicit purpose of presenting a blacklist of bad sites you've found, or for providing a "front page" to a certain subject. Anyone is free to trust your identity or not, and choose exactly how to rank its actions. This whole thing could be expanded to tagging and other metadata as well as comments, and so on. [0] https://news.ycombinator.com/item?id=29691303 https://news.ycombinator.com/item?id=29691303
- oehpr 5y agoI'm so glad to see others talking about this. A system of distributed trust like this is the makings of decentralized moderation. Decentralized curation. I've also had this idea on my mind for a while: https://news.ycombinator.com/item?id=21975294 https://news.ycombinator.com/item?id=21975294
- dennis_moore 5y agoI like this idea of incorporating trust and reputation. As for the curation of websites not scaling, some time ago I thought about the possibility of a search engine where the user supplies a list of trusted websites (for example, university websites, blogs of people they admire), and the search engine ranks pages based on link distance to these websites.
- Corrado 5y agoI think the Keybase project would have been great for providing the "authentication" part of this solution. To bad it died on the vine of the Zoom purchase.
- PoignardAzur 5y agoThe biggest problem, doing that, is that you increase the pressure from bad actors to websites that are trusted by lots of people. So, for instance, the more weight you give to sites that are quoted by Wikipedia in your search rankings, the more content farms will have incentives to sneak edits that link to their sites. There are ways to counter that (eg moderation), but in general, defense is more expensive than offense.
- brundolf 5y agoRight but the idea is that you don’t use sites as sources of trust, you use people. Content farms won’t be paying your friends and family to shill websites to you.
- hutzlibu 5y ago"Like us on Facebook and get this 10 cent coupon or even the chance of winning something real!" Works suprisingly well already, sadly. So yes - at some point companies would pay people to vote for them. But I still would prefer that system (with some differences) over the default. Because the people I would trust, would not fall for the common scams.
- dangerlibrary 5y agoI mean, of course they will? A large fraction of facebook content is MLMs and mobile game ads from one's own contacts.
- nfc 5y agoThen you just stop trusting them for your search results, it's not like you are unfriending them.
- dangerlibrary 5y agoI think you are applying a solution that would work in a cooperative game. The problem is that this situation, like spam, is an inherently adversarial one - and one where one's adversaries are very motivated and have substantial (substantially more?) resources than you at the outset.
- batisteo 5y agoLike the Web-of-Trust or similar?
- thrtythreeforty 5y agoThis reminds me of Cory Doctorow's "whuffie" from Down and Out in the Magic Kingdom (man, that title takes me back!). Whuffie is, roughly, money determined by your social interactions. More importantly, others also have a queryable score that's weighted according to who you esteem highly - this sounds like what you're proposing!
- allochthon 5y agoThis was kind of the idea behind a side project I started a few years ago: https://blog.digraph.app/2020-06-13-democratization-of-search.html https://blog.digraph.app/2020-06-13-democratization-of-searc... In that post, I don't address the reputation management aspect as much, but it's central to making the whole thing work, and I think crowd-sourcing and a well-conceived reputation management system that can influence results are good next areas for exploration.
- JamesBarney 5y agoYou could also incorporate "reputation" of the author. Basically have a real person, the author stake their real identity on the quality of the blog post they wrote.
- nfc 5y agoI was thinking something similar recently, and I also believe it's an idea worth exploring. Something else to add to this conversation. There's an obvious difference between two cases in which a trusted person trusts a url: 1) Single contributor website (blog, personal page...): It seems that we could spread the trust the whole website in the algorithm (at least more than for the next case) 2) Multi contributor website (forum, newspaper): It seems the trust should be given at an URL level Something worth delving into if we are designing this trust based search engine in real-time here at HN ;)
- wdencker 5y agoThis is very much in the spirit of what we were trying to do with trove.to [1] — give people an easy way to curate & annotate lists of websites, and layer a social graph and endorsement system on top of those lists. The problem we encountered is that the vast majority of people are not hyper-organized list makers — the 1% rule of the internet [2]. To create a "human curated search engine" with any utility, you need a massive amount of manually-categorized data — data which most people are simply not interested in generating. This is why no social bookmarking site (e.g. delicious, pinboard, etc.) has ever taken off to hundreds of millions of users. I still think there's something exciting to be built here, but it will likely need to take a more "automated" approach as you suggested. [1] https://trove.to/ https://trove.to/ [2] https://en.wikipedia.org/wiki/1%25_rule_(Internet_culture) https://en.wikipedia.org/wiki/1%25_rule_(Internet_culture)
- brundolf 5y agoYeah- that’s the only reason I suggested the automatic tracking of visits by domain, despite the obvious privacy complications there
- ImaCake 5y agoReddit effectively has this information buried in its database. But they don’t really use it, preferencing issues of the day and recycled tik toks apparently.
- victor9000 5y agoExcept Reddit is now a top-down company, so I wouldn't expect them to innovate beyond what a typical MBA can regurgitate.