11 ms·
Show HN: I built a self hosted recommendation feed to escape Google's algorithm
I created this chrome extension for myself where I track my own behavior locally and recommend myself content from platforms I want content from (youtube/twitter/quora/etc) in a feed. I made it public just in case anyone else was interested.
I would rather have control over my own algorithm and own the data. Also, it gives me flexibility. Turns out I do like these feeds just not when I don't own it haha. Let me know what you think of my implementation?
- deleted 4y ago[deleted]
- themodelplumber 4y agoThat looks really helpful, nice job and thanks for the effort in documenting what it's doing. I have been working on something a little bit related recently, related to books. Kind of reverse-discovery, maybe. I noticed that my ebook libraries in apps pretty much always end up in the same ordering, or sets of orderings. So I would tend to forget about interesting books I bought, the longer it's been since I purchased them. I wrote some scripts that collect random thumbnail images from my various ebook and PDF collections every so many minutes and gather them into little collections. Like shuffling cards and dealing some out. It's very simple stuff but it's already been nice to have. I also noticed that my collection of Gutenberg texts didn't have the greatest thumbnails, so I made a script to pick random excerpts from N different Project Gutenberg books at intervals. Then it joins then to a single text file. Kind of like visiting the bookstore to browse through some books a bit, see what's interesting. Anyway, this interests-related stuff is enjoyable. I really like your idea for the discovery side of things, thanks for sharing it.
- jawerty 4y agoThank you for the words and for sharing. The excerpts into thumbnails idea is cool I like the concept of having a library where the cover is the first sentence of the first page or something like that.
- rolisz 4y agoLooks really cool! Is it hard to port a Chrome extension to Firefox?
- jawerty 4y agoThanks! For this implementation I don’t think it would be too hard since I'm using pretty basic features I would have to rewrite some of the message handling. However, chromes new manifest v3 would make it harder.
- tut-urut-utut 4y agoPlease do show HN once you have Firefox version. The idea looks very promising, but using Chrome is a show-stopper for me.
- jawerty 4y agoI absolutely will!
- splitrocket 4y ago
- MoOmer 4y agoForgive me if I've missed something, but what's the connection here?
- deleted 4y ago[deleted]
- whimsicalism 4y agoIt's one of the 6 options for content sourcing, which seems like a massive over-representation if you are only going to have 6.
- capableweb 4y agoAre people really rejecting a whole project because one of 6 options is Gab? Do you people not use yt-dlp (or youtube-dl) because some of the sources they support can have more "extreme" content as well? I'd understand it if the project was built by the same people who work at Gab, or meant for Gab, or something like that, so there really is an associated. But there is not, so why blame the project for it? For all we know, the author could be reading Gab because sometimes you're curious to see what all the extremists are up to.
- danjoredd 4y agoAgreed. This can be a helpful tool, even if one of the sources is kind of lame. Besides, since it is open source we can just erase that or add a different source.
- Grollicus 4y agoyt-dlp supports a lot more than 6 sources (check out https://github.com/yt-dlp/yt-dlp/tree/master/yt_dlp/extractor https://github.com/yt-dlp/yt-dlp/tree/master/yt_dlp/extracto...) so I don't think that's a useful comparison
- slaymaker1907 4y agoPersonally, I think the biggest problem with "the algorithm" is the fact that it is an algorithm. People often distinguish between things that they enjoy and things that they would recommend to others, but algorithms aren't easily able to make that distinction. However, all of what I just talked about is just a hypothesis and I'd be curious about how it works out for you and what traits it shares with other recommendation engines and what ends up being more unique. For example, does your algorithm just end up pushing for engagement (which also tends to push towards showing extremist content), or is it less likely to get caught up in rabbit holes?
- jawerty 4y agoThank you For the point. I’ve tried to mix up the sources with alternative media and there’s a 33% of randomness (still with engagement but all engagement not just what you have frequency with) I could try seeing how to experiment with adding topics that are completely outside your engagement I think I would find that useful as well.
- Phenix88be 4y agoYou create a google chrome extension to escape Google? I understand the purpose, but your browser of choice doesn't fit the purpose, it's just a bit sad :(
- nottorp 4y agoYep, this is ironic to say the least. I actually wanted to try it but the first step of escaping Google is switching to Firefox... so I couldn't install it.
- bckr 4y agoThey built an extension to "escape Google's [recommendation] algorithm". They're clearly not talking here about privacy, they're talking about managing their own attention. Their implementation does fit their purpose.
- Phenix88be 4y agoYes, I know, it just doesn't feel right to me, that's all
- deleted 4y ago[deleted]
- quest88 4y agoThanks for sharing, I've been thinking about this too. But I've also mulling over whether spending my time looking at _any_ endless algorithm suggestions is what I personally need. You know, happiness and meaning and all of that.
- suketk 4y agoExactly, algorithmic feeds as a whole are the problem. I wrote about this on my blog [0], but in a nutshell they slow you down, take you in unwanted directions and encourage consumption over action. [0] https://suketk.com/feeds-considered-harmful https://suketk.com/feeds-considered-harmful
- quest88 4y agoHey nice post! That’s exactly what I’ve been thinking about for the past few (many?) months!
- upupandup 4y agoThe problem with Google enhancement tools is that I almost always end up uninstalling it and going back to Google for the algorithm. Yeah I don't like it but for example Startpage was problematic. For some reason if it thinks you are searching porn (you are not) it wont show you the results. Not only this, the results end up being quite slow and lot of things I took for granted are unavailable. Like I got used to just using the address bar as a calculator/currency/stock look up. I don't even land on the Google page, just do the calc or look up info in the address bar.
- jawerty 4y agoYea I’ve had this as well. I made a tool like this specifically for YouTube 2 years ago and almost gave up on it because I kept going back to the regular algorithm. Here I tried addressing that by pruning the algorithm more for discoverability across the web. Im exploring content I never would have found with the regular algorithm. But this is also why Im working to make it extensible so people can cater what they want to discover and not
- O__________O 4y agoWhile I completely get being in control of your feeds, from rolling your own to understanding how & why the feeds you subscribe to work — also feel like it’s just as important to be aware of significant topics and beliefs that are not in your feeds. Case in point, as another commenter pointed out — it appears the OP feels GAB, commonly viewed as a far-right social media platform, should be listed as one of a handful of default sources. World needs more systems that actually reflect how the user fits into the world, allows people to understand differences and find shared beliefs — not systems that create information bubbles, enable isolation, and increase the likelihood of extremists.
- deleted 4y ago[deleted]
- oceanplexian 4y agoGAB is a type of bubble, and definitely a (US politic) right-leaning discussion forum, but my ears perked up when you used the term "Far Right". Is there such thing as a moderate-right social media platform? Does far-right mean "not supported by a majority of right-leaning people", or does it mean "extreme from the perspective of an average left-leaning person"? I'm not going to either defend Gab or put it down, but even if you deconstruct your statement, it's the product of an information bubble. And it can have different meanings depending on what type of information bubble you live in.
- Forgeties79 4y ago>Is there such thing as a moderate-right social media platform? I mean the knee jerk response for me is Facebook, though I’d say it’s solidly right overall and not moderate-right.
- lupire 4y ago
- danpalmer 4y agoThis is the one that came to mind for me too. In North America and Europe, Facebook skews older, and older people tend to be relatively moderate-right. Gab is certainly "far right", but that doesn't mean those views are uncommon. Centrist viewpoints have shrunk, pulling what was previously considered "far" viewpoints into the mainstream. I think it's undeniable that a website where many (most?) of the userbase would advocate against the peaceful transition of power in the US should be labelled "far right".
- aliqot 4y ago
- Xeoncross 4y agoIs there a tool that automatically forwards every URL + HTML of the page you visit to a webhook so you could write an endpoint that would index everything? If not, I would love to see this add a "forward to webhook" option. I would be happy to write up a real backend that parsed the content and indexed it. Actually, there are lots of OS projects for this: https://github.com/quickwit-oss/tantivy https://github.com/quickwit-oss/tantivy, https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic, https://github.com/mosuka/phalanx https://github.com/mosuka/phalanx, https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch, etc... I would think that with the thousands (tens of thousands?) of pages I would index just browsing each year it would be relativity easy to find ways to automatically expand the index to include links that appear multiple times in those pages or some other heuristic
- deleted 4y ago[deleted]
- jawerty 4y agoThat's really interesting. In general indexing with this is something I havent thought too much about. What would you like about a tool like this?
- collegeburner 4y agothis is kinda off topic but do anybody know about tiktoks recommendation algo? i've been working on a little algorithmic newsfeed project but wanna improve the recommendation system. im wondering if it's harder to get the same quality data as with short video, but still curious to know how they got recommendations that much better. is it just more/better data into the algo, like with rewatches etc tracked? or actual tech improvement? bc yt shorts and ig reels don't have the same recommendations even with about the same data.
- deleted 4y ago[deleted]
- jawerty 4y agoI don't know what Tiktok does personally but I would guess there is something to do with comment engagements like most recommendation algorithms we use. From viewing to liking to commenting.
- synergy20 4y ago"It tracks and stores your browsing habits (searches, clicks, content engagements, text input) locally" -- how is this technically possible, that a chrome extension can 'monitor' all my search/click/etc?
- Doubtme 4y agoby giving it permission
- jawerty 4y agoYes you give the extension permission to store the data in the browser using `chrome.storage` API. Also, part of the reason why I open sourced it is so you could see the implementation in case of any privacy concerns.
- frozencell 4y agoIt is self hosted and open source right, you can look if there is no communication with a server.
- warbeforepeace 4y agoI like https://techurls.com/ https://techurls.com/ for my tech news. It aggregates most of the common techsites into a single place. They also have other versions for finance and a few other topics.
- danielcberman 4y agoInstalled the plugin and very intriguing MVP. Thank you for pulling this together. I have two questions/feature requests. - Have you thought about adding edit functionality to your content sourcing under feed settings to add or remove sites? Wikipedia, Stack Overflow, and other sites come to mind. - Have you thought about adding an report export functionality of links recommended? Basically to help an individual understand why their copy of the algorithm is recommending specific items.
- jawerty 4y agoYes 100% i'm working on making it so you can add sources you want. Also going to most likely add the ones you mentioned as default sources. Also I didn't think about your 2nd point but I like it going to add to my todo list
- mt_ 4y agoIt's not just that Google is evil, but going after what the algorithm recommends is what is most dentrimental. It's like getting hooked on cyber cigarettes.
- jawerty 4y agoHere's a discord if you want to are interested in getting involved with the project or want to contact me directly :) https://discord.gg/C6sYF48f https://discord.gg/C6sYF48f
- VyseofArcadia 4y agoI'm about to nitpick, so first I want to say, nice job. It's a cool project. Don't let me being a nitpicky curmudgeon discourage you in any way. Ok, so, my understanding is that self-hosting is running a web service for personal use. This isn't self-hosted. It's a Chrome extension. It's like saying that you're "self-hosting" Microsoft Word because you have a Word window open on your desktop. (Or, slightly more similar, you're "self-hosting" magit because you have emacs installed locally.)
- jyrkesh 4y agoAs someone who hangs out in /r/selfhosting a fair amount, I get what you mean. But I also see it used frequently in the vernacular to distinguish between a managed/hosted web service (which may simply be a web "application" as far as the user is concerned), and software you can run on your own machine. To extend your analogy, running Word locally is in some sense "selfhosted" as opposed to using Word Online in a browser. (Though I'd argue that this isn't a great example because of how much the Word desktop app increasingly uses cloud services in order to work.) IMO, selfhosting here was descriptive in that the Chrome extension is storing all data and generating all recommendations locally, without the use of a managed service that would presumably be storing all of my history in someone's cloud instance. (I'm sure you're a great person, OP, but I wouldn't want to give you my entire browser history.) Yes, it's shipped in the Chrome web store, so OP could theoretically ship different code than what's showing in the open-source repo, but you actually _can_ selfhost even "further" by building the extension yourself (even if installing unsigned extensions in Chrome is harder than ever, I think it's still possible, and certainly Edge still allows it via Developer Mode).
- jawerty 4y agoCouldn’t of said it better myself also another reason why it’s open source always the option to take the code directly and load it into your chromium browser.
- akomtu 4y agoHow much work is it to get feed from others you've subscribed to? Crowd-sourced discovery of interesting stuff, basically.
- jawerty 4y agoI like this idea, and having comment sections as well. However, networking it would involve some sort of centralized server (peer to peer as a possibility). This would have to be a clear opt in sort of feature to address privacy issues with pushing your browsing habit data to a centralized server.
- akomtu 4y agoI've been thinking about something email-based. Once a week your device sends a email to your peers and they reply with their updates.