17 ms·
I have started doing something completely different than using bookmarks. I set up yacy[1] on a personal, internal server at my home, which I can access from al
by l72 4y ago
I have started doing something completely different than using bookmarks. I set up yacy[1] on a personal, internal server at my home, which I can access from all my devices, since they are always on my wireguard vpn.
Yacy is actually a distributed search engine, but I run in 'Robinson mode' as a private peer, to keep it isolated, as I just want a personal search of only sites I have indexed.
Anytime I come across something of interest, I index it with yacy, using a a depth of 0 (since I only want to index that one page, not the whole site). This way, I can just go to my search site, and search for something, and anything related that I've indexed before pops up. I found this works way better than trying to manage bookmarks with descriptions and tags.
Also, yacy will keep a cache of the content which is great if the site ever goes offline or changes.
If I need to browse, I can go use yacy's admin tools to see all the urls I have indexed.
I have been using this for several months and I am using this way more than I ever used my bookmarks.
[1] https://yacy.net/ https://yacy.net/
- rasulkireev 4y agoThis is insane! Thanks for recommendation! Was able to set it up relatively quickly and now have a personal search of all the things that I find interesting, this is insane!
- ComputerGuru 4y agoHow does yacy handle paywalls? Does it use the cookies from your browser instance or can it use bypass services like 12ft and co?
- omitmyname 4y agoThat's amazing! I wanted to make something similar. Thank you!
- Veen 4y agoI'd love something similar to automatically crawl and index every site I visit. I'm forever losing stuff. I know I saw it but I can't remember where.
- mttjj 4y agoThis is Mac only and I have no affiliation other than I like this developer but your request reminded me that he just launched this app: https://andadinosaur.com/launch-history-book https://andadinosaur.com/launch-history-book
- prepend 4y agoThat’s a really genius idea. I also like the author’s pricing mode. I was fearing some stupid “$10 is just the price of coffee. This is worth 24 coffees a year for the rest of your life” and have a reasonable $7 purchase price.
- kamray23 4y agothat's some expensive ass coffee considering i buy a bag for less than that and use it for a week or two, making tens of cups
- prepend 4y agoYou are not Starbucks’ market.
- kamray23 4y agowho the fuck is starbucks' market if a daily coffee costs over $2000 each year (assuming you do a 5-day workweek and throwing two months off the year to account for days when you don't get morning coffee from starbucks) you have to be swimming in money that makes me doubt that anyone would pay more than $2-4 and even that's overpriced as hell
- asselinpaul 4y agohttps://heyday.xyz https://heyday.xyz comes to mind
- fudged71 4y agoVortimo
- brokenkebab2 4y agoCan it cache indexed pages? If not how do you deal with disappearance/changes online?
- l72 4y agoYes, it does automatically keep a cache from when it indexed the site. I have it set to not automatically recrawl sites, so the cache is from when I added the site.
- kybernetikos 4y agoThis is great, and is something I've wanted for a while. I use pinboard which is supposed to have similar capabilities (click 'search full text', 'search mine' after turning on and paying for 'archiving'), but I've never been totally confident in it (pages would change, and the cached version was updated to a 404 page), and ended up letting my archiving subscription lapse. I think google used to offer something that did this as well as search all your local files, but I think that went the way of all gThings.
- mitchdoogle 4y agoThere's also https://historio.us/ https://historio.us/ - it's essentially same as what OP is doing but it's as easy as bookmarking
- kybernetikos 4y agoThat does look pretty cool, and unusually for a SaaS has chosen a pricing I think is reasonable for the service (not everything should be $9 a month!). Do you know if it does pdfs? That's a key thing I want in this kind of service.
- kerneis 4y agoI use Zotero for this and it shines at pdfs. It does webpage snapshots too. Discussed below: https://news.ycombinator.com/item?id=31849987 https://news.ycombinator.com/item?id=31849987
- aaaaaaaaata 4y agoAll of Stavros' stuff is reasonably priced and sexy.
- danesparza 4y agohttps://en.wikipedia.org/wiki/Google_Search_Appliance https://en.wikipedia.org/wiki/Google_Search_Appliance
- 4y ago
- ericcholis 4y agoThis is an incredible idea.
- pacifika 4y agoNice! I’ve been working on and off on a similar idea (searchable index of link contents) as a cli app eventually web web frontend. It’s on Python so packaging has been an issue.
- mitchdoogle 4y agoIf anybody is interested in this, there is also a service which offers very similar thing: https://historio.us/ https://historio.us/
- stavros 4y agoThat's mine! Feel free to drop a line if you need anything.
- cupofpython 4y ago"i like historious" as the only review at the bottom gave me a laugh for some reason
- ryan29 4y agoI have 3 tech related subscriptions; BorgBase, JetBrains, and historio.us. I self-host everything, but I've never found anything that replaces historio.us. It indexes just enough to always get me a complete copy of the data I want cached and the search results are just right. I often have about 1 page of search results when I'm looking for old info and I can pick out the page I'm looking for instantly. I use it a lot when I'm learning something new. As I'm looking for beginner info I'll often find more advanced stuff that I'd like to try or learn at some point, but I don't have a good enough understanding to know if it's truly useful info. I historify those sites and move on knowing that I can find it in my historio.us search at a later date. I also reference cached pages in a lot of my personal docs. I recently started using ArchiveBox for that, but the search doesn't make it a good replacement for the above use case. I've been using historio.us since 2011 (!) and have never found anything to replace it. Great job!
- stavros 4y agoThanks, I'm glad you like it! I really should give it some love, but I'd need to do a fairly sizable rewrite for most stuff...
- jrib 4y agoproduct looks very promising! Some UI feedback: I went to check out https://historio.us/ https://historio.us/ and on my macbook air I saw the top of the green button "see our plans". Took me a couple of clicks until I realized I had to scroll down to click the real "plans and pricing" button that was off my screen.
- jrochkind1 4y agoNice! How about getting it to automatically index your whole search history? Not what you're going for -- you don't have a list of specifically opted-in 'bookmarks' to browse. but I have often wanted "wait, what was that site involving X I was looking at maybe last week?"
- pmoriarty 4y agoIt would also be nice to be able to search through my aggregated browsing history on every device I use. Maybe I should open a feature request to Google/Fracebook to provide an API hook for that, since they probably already have all that information anyway.
- hbarka 4y agoThis. I wonder if there is a way to direct my searches first to the domains I have ever visited. Oftentimes I will search for something that I am sure I’ve hit before but can vaguely remember which result set it was that scored my search.
- nix23 4y agoYes works, you just need to use the proxy function of yacy, and everything get s indexed.
- jrochkind1 4y agoFacebook doesn't share data out in APIs!
- Jarwain 4y agoMy chrome history appears to aggregate my browsing across devices Anyways, so it should just be a matter of exposing that info
- CRConrad 4y agoFirefox probably does too, if you're logged in.
- a5huynh 4y agoA bit different, but I've been building something similar that runs locally: https://github.com/a5huynh/spyglass https://github.com/a5huynh/spyglass You create some rules for topics you want to index and it'll go out and crawl them. Searching through it is a global hotkey away.
- luckman212 4y agoSpyglass looks amazing! Thank you for this.
- a5huynh 4y agoThank you! Feel free to ping me on Github/Discord if you run into any issues, always looking to make it better :)
- srinathkrishna 4y agoThis is fascinating! I've been meaning to set something up on a spare rpi for this and I hadn't heard of yacy before. Thanks!
- dopidopHN 4y agoThanks this look neat. Can you easily share index accros clients? Edit : looks like the docker config allow to mount a arbitrary folder , that folder can be shared. I don’t need it to be concurrent proof. Again, thanks this look nice.
- deleted 4y ago[deleted]
- mejutoco 4y agoThis is how I always imagined the search engines of the future to work. All the data is local first and the user is in control.
- sanwrencho 4y agome too, when I first used delicious I was hoping that individual curators could provide a more in-depth "meta search" engine! Pocket is not bad, it provides some interesting links to new stuff on the web.
- jxm262 4y agoIve never even considered something like this before, but its genius! The offline caching sounds awesome. Thanks for sharing
- BiteCode_dev 4y agoNice I have wished for a while that browser would store the entire page of any bookmark you save automatically, and put a decent search engine on it. I wrote a script once to do it for my bookmarks, and it didn't even take that much space on my hard drive. Your system could be a Firefox addon, kinda like what scrapbook used to be, but automatic. Even with a note system, and storing metadata, Zotero style, but without the need for the dual setup.
- account42 4y agoHow mozzilla is not pursuing that kind of innovation in Firefox itself is beyond me. Instead they continously try to ape the Chrome UI and collect telemetry without permission so that they can give you "exciting" "new" features like colorways.
- hcarvalhoalves 4y agoDidn't know about yacy – interesting! Thank you
- Nition 4y agoWhat a good idea. A search engine like Kagi could support importing your existing bookmarks as a custom lense.
- RF_Savage 4y agoThat's an awesome idea. I'll have to look into this. Been looking for something like this.
- keyle 4y agoHow does the process of bookmarking goes then? meaning, how hard is it to add something to crawl with depth0 on a day to day. Can it be done with a bookmarklet?
- account42 4y agoThat sounds really cool. How difficult is this to set up? How confident are you that this will keep your bookmarks safe? Losing indexed data might not be such a big concern for a search engine that is intended to re-index everything continuously. I defnitely felt betrayed when I found out that Firefox automatically deletes old history :/ I guess they think that most users need to trash their profile to fix some obscure bug before that anyway...
- aguynamedryan 4y agoI'd love to hear more about how you set all this up, particularly, do you have some sort of extension or bookmarklet that submits the current page to yacy's crawler?
- johntash 4y agoThanks for this recommendation! Do you only index things with a depth of 0? I just set Yacy up a couple days after seeing this post, and didn't realize what that meant at first. Now I have 10s of 1000s of pages indexed after importing my bookmarks/history, and I'm wondering if it'd be more useful to _only_ index the pages I've visited/bookmarked, or if it'd also be good to crawl those sites further. I guess one distinction would be whether or not I thought I could use Yacy as a full-time replacement for google/ddg. It'd be nice if I could index "everything", but then have a toggle to search only my bookmarks/history or something similar to that.
- tecoholic 4y agoI really liked this setup. The only point of friction for me was adding the links to the index via the crawler everytime. So I created a Firefox Extenstion to do it directly from the address-bar. If someone is interested, you can download it from here: https://addons.mozilla.org/en-US/firefox/addon/yacy-it/ https://addons.mozilla.org/en-US/firefox/addon/yacy-it/ Limitation: It currently supports only YaCy running on localhost and unprotected. You might have to configure CORS as outlined here -> https://github.com/tecoholic/yacy-it#configuring-yacy https://github.com/tecoholic/yacy-it#configuring-yacy