6 ms·
Hister: A private search engine for the pages you visit and the files you keep
- jammaloo 25d agoA recent discussion about this tool https://news.ycombinator.com/item?id=49351802 https://news.ycombinator.com/item?id=49351802
- evilduck 25d agoSaw on Discord that they have to change their name, since https://histre.com https://histre.com sent them a letter.
- tamimio 25d agoIntegrate it with linkwarden so it searches the bookmarked pages.
- billbrown 25d agoIt does that already. https://hister.org/docs/import#importing-from-linkwarden https://hister.org/docs/import#importing-from-linkwarden
- tamimio 25d agoThanks, that looks interesting, I will test it out and see if it continuously sync or must be manually imported from time to time, or if it can replaces linkwarden entirely. My linkwarden instance also saves as a pdf not just html and bookmarks are in GB in size, if hister does it more efficiently it’s even better.
- 361994752 25d agoI had the same problem for a very long time but it is largely solved now. I started to simply ask chatgpt "hey I read something about x, y month ago but can't find it now". There is a surprisingly high chance chatbot can just give the exact answer back to me, usually with extra interesting reading materials as a plus.
- cobertos 25d agoIs all your browsing history already with chatgpt or something? This has an MCP server specifically so a workflow like that would work for you. This is just made to gold the data, and I'm a human accessible way should your AI fail you
- 361994752 25d agoNo I don't share anything with chatgpt. But I do have a $20 subscription, if it matters. I feel It's just capable enough to find what I want from my usually vague and inaccurate description.
- itsdesmond 25d agoOr something similar enough to what you wanted that you don’t notice it’s a different thing. That’s awful.
- rglullis 25d agoAnd now you are even more dependent on OpenAI... You didn't solve the problem, you are just trading pain points.
- randomblock1 25d agoNothing about this depends on the provider, you could spin up a local Qwen and give it a search tool like SearXNG or something. At this point local models are more than good enough for simple tasks like that. Using ChatGPT is just (usually) faster and simpler
- 361994752 25d agoThere should be no real vendor lock in in my opinion. You can ask the same question with pi + qwen (or any harness + good enough model) with internet access and it will work. Chatgpt is just one option came in handy.
- pixl97 25d ago
- bradrn 25d agoOoh, very nice! I have my own tool I’ve been using for this [https://github.com/bradrn/full-history-search/ https://github.com/bradrn/full-history-search/], and it’s incredibly useful, but it’s also pretty primitive. This one looks a lot nicer.
- jval43 25d agoGoogle Chrome did this in 2008. Full-text search over all visited pages, stored offline. It was very useful and I miss it. Nobody seems to remember it, even though it was a headline feature. Was removed in 2013, I think due to technical constraints. Will definitely try this.
- corney91 25d agoNow Chrome can't even remember what addresses you went to more than 90 days ago. It's such a nerfed browser.
- iririririr 25d agoStill boggles my mind that people went from ridiculing people who installed Ad companies browser toolbars, to actually running their browser. ...And don't give me the tired "it was faster". this horse was already beaten to death and it was 110% marketing.
- 93po 25d agoWhen chrome was literally first released it was in fact faster than IE and Firefox, by quite a lot, and had some nice features that neither had, too. But that gap didn't last for long.
- iririririr 23d agolol. it amuses me how much people are the marketing and now forces themselves to remember it wrobg. when it launched it was fast because it had LESS features.
- jval43 25d agoIt was so much faster you could not believe it. It was a revelation and even though "degoogling" was already a thing, Chrome was so much better than the rest that you just had to use it. Tabs actually worked without crashing your PC. The risks were clear from day one but Google and Chrome were great to both users and devs, and it stayed like that for a long time.
- Lio 25d agoThis is really cool. I like the idea of combining it with a offline Wikipedia cache.
- RobGR 25d agoI've been using this since the last time it came up on here. I don't have it index every page I visit, I use the browser plugin to tell it to index specific ones. It is useful for sure, but I think it will really shine once I've been using it long enough for it to build up a bigger index of things that are old enough that I've actually forgotten about them.
- asciimoo 25d agoOhi, author here! Thanks for posting Hister. Feel free to A.M.A. My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach. Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline result previews, so information remains searchable even when the original page changes or disappears. It supports full text and semantic search, can run entirely on your own machine, and includes a web interface, command line tools, and an MCP endpoint for assistant integrations. Website: https://hister.org/ https://hister.org/ Tiny read-only demo: https://demo.hister.org/ https://demo.hister.org/ Ps.: It looks like our name conflicts with a registered trademark in the US. The owner of the other project has asked us to change it, so we’ll probably need to comply sooner or later. Name suggestions are welcome! Ideally, the new name should be relatively short, sound good, and have an available .org domain. Thanks!
- Capricorn2481 25d agoThanks so much for this, I'm using it all the time. I self host a few things, but I'm using this the most.
- adfm 25d agohistro.org is available.
- sbeckeriv 25d agoI am happy to see the idea of history search more. I am on my 3rd version of my own. the use of local LLMs has made it easier to support features like weekly summarized and recipe extraction. I like the search ui. my projects become functional but never polished. https://github.com/sbeckeriv/memoir https://github.com/sbeckeriv/memoir
- kilroy123 25d agoI've been trying this out the past few weeks. I was literally just in there searching for a link 5 minutes ago. It's badly needed, and so far it's working well for me.
- pkamb 25d ago> Your own search engine — Hister is a private search engine for the pages you visit and the files you keep. Is there any site/project that works as a fully customizable personal front-end to all other SERPs? When I search for something, I always want a link to the best Wikipedia result. This should always be in the same place and have a giant icon/picture. Then there could be easily clickable links to the SERP pages for Google, DDG, etc. for that query. A big link to route it to your favorite LLM. Seems like you could have a really useful "homepage" for all searches that sat in front of all the other sites. It could be local only and would not require indexing the web. Also wouldn't be a files search thing, as Hister appears to be.
- taude 25d agoKind of related to this in that I built it to hoard knowledge from web pages I've visited along with implementing a Karpathy-style LLM Wiki, but the knowledge is collected automatically from sources I browse. I have it up on GitHub, but I don't think anyone should use my implementation. Loosely, what I built: * On each of my machines I have a cron job running that looks at all my web browser history (usualy it's inspecting the brower's SQLlite across firefox and chrome). If it matches my rule list: hacker news stories, certain reddits, etc. it'll grab the page, convert to markdown and drop in my Obsidian Vault incoming. * It has a whole de-duping architecture since I might open the same page on multiple machines. Uses the CloudFlare SQLITE D1 storage for tracking the processed links. * it'll then trigger the LLM to do some Karpathy wiki style taxonomy assignment to the articles, organize them, create an index etc. It's then available for my "bot" stuff to do writings for me.... I will probably write more about it at some point. I'm not certain it's totally useful and not just a yak-shave on hoarding knowledge. Ai-drafted article on this [1] Example AI-Drafted article based on some discussions the other day on Ollma vs LLama.cpp [2] [1] https://taude.xyz/posts/how-archivore-turns-browsing-into-a-wiki/ https://taude.xyz/posts/how-archivore-turns-browsing-into-a-... [2] https://taude.xyz/posts/skip-ollama-run-llama-cpp-directly-on-a-mac/ https://taude.xyz/posts/skip-ollama-run-llama-cpp-directly-o...
- skinfaxi 25d agoWhy don't you think people should use your implementation? Just curious
- rolandog 25d agoI guess he meant it in a "look, I made a cool thing that is useful to me, but don't go throwing away your current organizational method just yet nor leave it open to the internet" way.
- taude 25d agoI vibed most of it (and though I like the code that was output, I coached it with some custom skills). If I were to share it out, I'd probably clean up some of the tools to make their interfaces smaller and cleaner. And make the docs a lot simpler. I'd probably also create some form of dashboard so you can see what's happening (all the docs scraped and saved), etc. EDIT: it's also the type of thing that feels very personally customized for my needs. I encourage you to build something similar on the idea. Much like how Karpathy Wiki was suggestive and not a runtime to just use...
- tombert 25d agoInteresting, I actually very recently built a similar project [1]. I was unaware of this...If I were I probably wouldn't have bothered! [1] https://git.brucewillis.sexy/~tombert/fs_index https://git.brucewillis.sexy/~tombert/fs_index I promise, safe for work, despite the URL.
- testycool 25d agoI was fascinated by hister initially, but now I'm fixated on your URL.
- tombert 25d agoHah, I bought that URL a long time ago, mostly as an in-joke between me and some friends since Bruce Willis was my favorite actor. Then later I needed a dev URL for some work I was doing, and the only one I had free was brucewillis.sexy. Then I kept making subdomains on it for different things and eventually it just became the default. I like it.
- kanzure 25d agoCan you add viewed tweets?
- asciimoo 25d agoYes: https://hister.org/posts/how-hister-makes-the-social-web-searchable https://hister.org/posts/how-hister-makes-the-social-web-sea...
- MomsAVoxell 25d agoI attain this without involving an untrustworthy third party, with one simple trick: Print to PDF. Every single web page I’ve found interesting, since the advent of the Web, I have printed to PDF and stored locally for my own personal reference. Something like 80,000+ files - my own copy of my own Internet - indexable, searchable. Available offline. Something to read when I am far out to sea. There is no need to involve third parties in your Internet history - no matter how trustworthy they seem to want to appear. Print to PDF, and you’ve got everything you need, safe and sound.
- nottorp 25d agoExcept search. I want search. Going to try this project.
- MomsAVoxell 25d agoYeah, about search: $ pdfgrep -r -i -n -H "your mom" ~/PDFArchives/ Very effective, very fast, very private. Bonus points if the PDF filename itself is derived from a well formulated <title> tag, such that you can just use “ls” ..
- nottorp 24d agoYes, but that's a solution for the stuff you consciously save. While this is a solution for stuff you see but decide it was interesting weeks after you closed that tab and you only have a vague memory of what it was about.
- MomsAVoxell 23d agoWell, I consciously save anything I’ve been interested in for at least 2 minutes, it’s a perfectly good way to filter interests - and the PDF solution safeguards those interests from being exploited by third parties.
- krackers 25d agoYou could combine it with existing products like DevonThink that are meant for researchers organizing documents and provide tagging, semantic search, similarity graphs, and so on.
- computator 25d agoI'd like to use it, but I'm hesitant to use anything that isn't a reviewed and approved package in my Linux distribution. Even if the chance is 1% that a program I download has malware or security problems that even the author doesn't know about (eg., due to libraries used), odds are that my system's going to be compromised if I run 50 such programs. This extends to browser add-ons, bookmarklets, and extensions too. How do other people handle this dilemma? Even solution I can think of involves are a great amount of extra work.
- BrokenCogs 25d agoClone the repo and ask your llm of choice to audit any security risks
- computator 25d agoAnd then build from source. And make sure it doesn't auto-update so no new malware gets in. This is my idea of a lot of extra work.
- applfanboysbgon 25d agoBuilding from source is nowadays writing one sentence in a chat box. Your idea of extra work is either unbelievably lazy or perhaps outdated by a few years.
- itsdesmond 25d agoWhy even ask the question?
- edoceo 25d agoAfter that I run them in containers, no network access, ro filesystem.
- perlgeek 24d agoNot very practical for a solution that requires a browser plugin. The browser is where most of my interesting data is.
- torvald 25d agoBonus point for having an IRC channel for community forum.
- GlacierFox 25d agoI thought that said Hitler at first.
- Vaslo 25d agoIt's really great you've done all that work for importing from other apps - I use Karakeep and at first didnt know how much redundancy I have between the two, but now I am going to try this since I have so much of it already setup.
- ifh-hn 25d agoHad a double take as initially read it as Hitler...
- jamienk 25d agohttps://bugzilla.mozilla.org/show_bug.cgi?id=342916 https://bugzilla.mozilla.org/show_bug.cgi?id=342916 Can I add NOTES about pages? This might be a good spot to do that...? Maybe the interface can be in a web page instead of terminal? Before Google took off there was a vibrant ecosystem of FOSS dev around search, all different little aspects of it. Then after Google people stopped fiddling with search, search became "solved" or maybe "must be coded by the big boys". Shame. Thank you for this, looooong time coming
- asciimoo 25d agoThe default interface is web based: https://demo.hister.org/ https://demo.hister.org/ Notes are not supported yet, only labels. But it is a useful addition, added to my TODO.
- jjice 25d agoBeen running hister on my work machine for a month or two now. It's one of those things I don't look at until I need to, but when I do need it, it solves the dreaded hunting through the last month of browser history. I also have it index my Obsidian notes, which is another little bonus for global search. I did need to build up quite a few exclusion rules early on, but it's been hands off since.
- roschdal 25d agoImagine the security vulnerabilities
- w10-1 25d agofirst line of the open-source readme: ~download the binary for windows~ -- paved with good intentions?
- phyzome 25d agoRoughly in the same space is Zotero <https://www.zotero.org/ https://www.zotero.org/> which is intended as a way of organizing research materials but is also a great way of quickly archiving web pages you've visited, including quick tagging and full text search.
- machomaster 25d agoAnother very useful project is SingleFile. I was ecstatic when I found out about its existence. https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile "SingleFile helps you to save a complete web page into a single HTML file. SingleFile is a Web Extension (and a CLI tool) compatible with Chrome, Firefox (Desktop and Mobile), Microsoft Edge, Safari, Vivaldi, Brave, Waterfox, Yandex browser, and Opera."
- diarrhea 25d agoI use LinkDing for website bookmarking and storage/archiving needs. It is a great and stable piece of software. It archives using a Chrome web driver. Sometimes that fails, e.g. on Captchas, very heavy JS sites or sites behind login. Those I save with SingleFile and attach in LinkDing manually. Simple and works!
- autoexec 25d agoDon't most browsers do this natively? Firefox saves single pages as html. Chrome saves pages as mhtml which isn't supported in firefox but can be easily converted if there's ever a need.
- colbertw08 25d agoWill definitely be giving this a try. The marketing website looks good. Did you have an LLM do the design?
- liberian 25d ago[dead]
- etamponi 25d agoThis is amazing and something I'd like to integrate with my personal AI assistant. Is there a way to make it also index pages visited from an Android phone?
- Invictus0 25d agoat first glance i thought it was called hitler
- rao-v 25d agoI'd love a extension setting to only send tabs that were visible for ~4+ seconds. I built myself a little extension last year that tracks what information I was looking at, but focused on generating "new info" recaps for the day / week. I realized that I open / quick view a lot of pages and close them, which is a strong signal that I don't care about that specific page, and it shouldn't be a source of "new insights" that I learnt that day (since I probably don't care about that topic). I'd love to re-try a simpler version of that project that builds on Hister as a backend actually.
- kenanfyi 25d agoAgree on this. There area lots of links I click and immediately close, because I immediately recognize it's AI Slop for instance. No need to register these type of stuff immediately to the hister database. Adding a timer constraint would be really nice and I guess it might be quite easy to do it since it's only a concern in the browser extension.
- devdoshi 25d agoLove this kind of system, so many useful mashup apps to build on top!
- deleted 25d ago[deleted]
- valcarvalho 25d agoI've been using Hister for over 2 weeks. So far, so good. I really like the interface, the search, everything. Thank you!
- saimiam 25d agoCompletely random but o remember reading back in the day that Nostradamus had predicted the rise of a Hitler like figure except he had named a Hister, not the actual. https://www.history.co.uk/articles/did-nostradamus-really-predict-the-rise-of-adolf-hitler https://www.history.co.uk/articles/did-nostradamus-really-pr...
- jambalaya8 25d agoHi, was wondering if you would be alright with a github set private fork done by someone else if they got bored and wanted to add some private features? Thanks. Now back to my steak frites! :9
- Notkel 25d agoI was just thinking about a tool like this this morning.
- mattjbarnes 25d agoGreat product concept. I explored this some time ago and created a browser extension product. Check it out. https://stashpad.ai/ https://stashpad.ai/
- ferrule 25d agoThe local file indexing sounds useful. Often forget where I saved that important PDF and Spotlight fails me.
- alwillis 24d agoI have no affiliation but Find Any File [1] for macOS is amazingly good! [1]: https://findanyfile.app https://findanyfile.app
- rochansinha 25d agoI can see using this to index the thousands of bookmarks I have and my every growing reading list and see if my local model and find a pattern and/or help surface anything interesting related to the papers I read
- febed 25d agoOpera browser has this feature baked in a long time ago, before the Chromium era.
- Yashjain413 25d agoThis is pretty useful, actually. I’ve been using it for the past 2–3 days, and I didn’t find it through YC Hacker News but through some other channel. It’s great to see that so many other people are already using it and praising it—congrats on that! One of the most useful use cases for me, especially since I work in GTM, is keeping track of new ways to get replies from prospects, whether through cold outbound or things like SEO/GEO optimization. I read at least 2-3 articles a day on this, and it genuinely helps me figure out which ideas are worth trying because I can now keep track of everything I’ve read.
- deleted 25d ago[deleted]
- cardboardguru 25d agoIs the plugin compatible with Brave?
- IndiaInfraNotes 25d ago[flagged]
- Yehoshaphat 25d agoI use Hister as my default search engine. I’m quite happy with it. If I know I need to branch out, I use the double-bang before my query “!! Hacker News Fujitsu” and it kicks straight to the configured search engine (Kagi in my case).
- soapdog 24d agoWtf with everyone suggesting new names? Is this an LLM thing?
- frumiousirc 24d agoDo you have thoughts on the prospects for inter-operation between the "recoll" application and hister? Like, maybe an adapter that let's hister read recoll's xapian database/index? Or, a tool that converts/syncs beween hister's store and recoll's?
- szamski 24d ago[dead]
- pidgeon_lover 24d agoI use Betula (https://codeberg.org/bouncepaw/betula/ https://codeberg.org/bouncepaw/betula/) for this purpose of a self-hosted personal Google (for websites). (For file search, I use voidtools' Everything, and I'm not sure why anyone other than Microsoft would want to mix local file results and web results) This project looks cool, though I misread the name as "Hipster" initially [sips almond milk macha coffee]
- elestor 24d agoI used to use it, but it would use a lot of CPU and hurt battery life, is the idea that you self host on a different machine?
- ozim 24d agoMy issue with this is that as I had my notes that I curated there was never return on investment for doing that. Running this would be nice but it still takes time and still there is no ROI for me. Of course there will be people who find it useful but I am pretty much done with building knowledge bases or having todo lists. Stuff that I need to do or remember - everything else if I forget nothing happens and it doesn’t impact my life or work.
- amai 24d agoIt would be nice if it could also search through emails. I'm using emails to myself often as notes and bookmarks. Would be nice if they could be included in the search index.
- Unified-Mentor 24d ago[dead]
- itsmeduncan 24d ago[flagged]
- dang 24d agoCan you please not post AI-generated or AI-edited comments to HN? It's not allowed here - see https://news.ycombinator.com/newsguidelines.html#generated https://news.ycombinator.com/newsguidelines.html#generated and https://news.ycombinator.com/item?id=47340079 https://news.ycombinator.com/item?id=47340079. Of course, it's impossible to know for sure what was LLM processed or not, but some of your posts (like this one) have been getting classified that way.
- smilliken 24d agoI've been using histre.com for years, which builds a personal search engine for the pages you visit by adding a browser extension. It's freed me from the habit of compulsively bookmarking everything in case I might need it again. I also keep a local text index for grepping which is plenty fast even with years of history and ripgrep, and comes in handy often.
- chrisss395 24d agoCan Hister handle indexing of large local file shares? I've got 30-50TB of Word, PowerPoint, and Excel files with no easy way to search them currently, beyond Agent Ransack. Does Hister handle this well? If not, can anyone suggest other options?
- javatextbook 24d agoI just wanted to say thank you for not letting the LLM write the README.md for you n
- 1vuio0pswjnm7 24d agoA different approach For person using resource-constrained computers where CPU, memory and storage space is limited URLs from the local forward proxy log are extracted periodically and stored in compressed files (URL logs) (I also store post-data) The compression method used is old and unpopular: recursive pairing Compression ratio is better than gzip but worse than zstd, compression/decompression speed better than zstd but worse than gzip More recently a method was developed to search these compressed files Size of compression utility: 42.3K static binary Size of search utility: 102.4K static binary No Java Limitations include basic regex only (no back-references) and files must be line-oriented No decompression step is needed. IME, this search is very fast. If it is slow then this means the keyword is too common: refine the search With minor modification (insert a newline at the top of file) I can also search inside compressed tar files As a www user with underpowered computers doing relatively small jobs, these old, unpopular methods have proven to be fast and reliable for me When I'm searching more than just URL strings, e.g., dates, titles, etc., I reformat the data into SQL and store it in a text file Instead of storing large SQL database files, I store the text file compressed with recusrive pairing I can then search the compressed SQL using basic regex; the output is piped into sqlite3 to create a "results" SQL database, e.g., in memory For me, the speed of sqlite3 in creating relatively small databases is excellent Then I can query the "results.db" using SQL
- 1vuio0pswjnm7 24d ago*recursive