7 ms·

Show HN: Searchable offline archive of browser history

35 points by graderjs 5y ago

mooreed 5y ago
Looks fun reminds me of Google Desktop search from way back in the day - you know back in the days of google also selling a search appliance for enterprise data centers.
- connordoner 5y ago
  Lol… “Did you mean ‘the cloud’?”
  - Jaruzel 5y ago
    No, they didn't. Back in the day Google sold 'Google-In-A-Box' which was a rack mounted hardware device that you installed into your company's server rack (either on-prem or in a hosted data centre) and once configured it crawled all your internal resources and presented an internal search engine with all the power of Google search. Although a good idea in principle, I found it cumbersome to configure and it only (at the time) supported ftp and http based resources - if your company made heavy use of Windows file shares you couldn't index them without also serving them over ftp or http. I believe later versions improved on that though, but by then it was a dead product, and Google discontinued it.
    - connordoner 5y ago
      No, I know — I remember the CSA well. It was a play on words on Google’s “Did you mean?” functionality.
      - Jaruzel 5y ago
        Aha. :D
belkarx 5y ago
Interesting idea however I can't see many practical uses for this ... how often do you really need to search through the text of something you browsed some time ago and would it really offset the disk-space cost? I personally would have benefited from this on a few occasions however SQL+title-keywords+timestamps were effective enough and it's a rare occurrence. And if you need to archive something, why do it locally when there are projects like the Wayback machine (centralized is an argument against it I suppose), or why do it for everything?
- aryamaan 5y ago
  I want to do for the tweets all the time.
- deleted 5y ago
  [deleted]
- graderjs 5y ago
  Everyone is different. Sometimes even just saving someone that inconvenience and time and stress in that edge case where like you have this bookmark that you know that you had or this page that you visited but you can't find it again through a search engine, sometimes just saving people the headache of going through that is a really valuable thing like even if it only occurs a couple of times per year. Personally I couldn't understand why people were so into this project when I created it. I didn't understand the potential uses when I created it. I've liked that people have shown me, and also that I've had to market this way less than my other products. I created it as like you know a technical proof of concept kind of hack because I wanted to see if it was possible and then I kind of discovered that there were all these like archive nuts who were really into archiving. And then over time I started to think well sometimes I've got a bookmark but I can't remember how to find it and I couldn't find it in in the main search engine but I knew that I bookmarked it somewhere but I can't identify it from the title search. So having a full text search over the page content like a personal local search engine I thought it was going to be useful. But the main cool thing I think is the fact that you know you can have pages running as if they're on the internet when you're offline. So on the plane say you can have all of this you know mdn stuff archived all of these useful resources useful documents and then you can be running on the plane you don't have to use the Wi-Fi but you can have this like offline internet so you can keep working. That's me and my perspective maybe let me finish with the words of someone else who commented here just a little while ago but then deleted the comment, but I thought it was a really good perspective so I'll share it below > I thought the same thing 10 years ago. I have(had?) many thousands of bookmarks collected over 10 years. All of the most interesting things I ever read. Imagine my disappointment when I eventually realized most of them are now gone. Site is gone, URL is changed, videos taken down (so many of them it's amazing), websites weren't properly archived etc. > I tried to curl all my bookmarks for a half-assed solution but dynamic loading and unusable content presentation without running all js and saving dependencies, timestamps, replies etc makes it impossible. Even worse, it's impossible to discover something relevant when there is no search function. What you get is the page title and a URL. Usually both of them useless. I tried categorizing them and adding comments but that was a lot of work and led to me deleting stuff that are hard to categorize. I'm not a professional programmer and I'm definitely not fully aware of how modern websites work (particularly on the front end). > But if someone provided a locally run telemetry-free open-source (so it's auditable) searchable archive-as-you-browse tool that legitimately works reliably and with some kind of hard (technical) guarantee that it won't break with the next browser update, I'd pay good money for it.
aabbcc1241 5y ago
I made something similar, with and/or/not full text searching https://github.com/beenotung/personal-search-engine https://github.com/beenotung/personal-search-engine
- hamsterbase 5y ago
  I made something similar too https://github.com/hamsterbase/hamsterbase https://github.com/hamsterbase/hamsterbase Render and upload mhtml(chrome)、webarchive(safari) website. full text searching is WIP
throwanem 5y ago
Why build it with an intimate dependency on Chrome, and not as a general HTTP/HTTPS proxy? Not that it doesn't look neat, sure. But I do a lot of browsing from my phone, and I've just been spending some money and considering spending more on even a partial archiving solution for that. If this could support that use case, you'd have a chance at a paying customer for life. As it is, you've already told me you don't want me, or anyone with a similar use case, as a customer at any price. I assume there has to be some engineering justification for a tradeoff like that, but I admit I can't fathom what it might be.
- graderjs 5y ago
  > Not that it doesn't look neat, sure. But I do a lot of browsing from my phone, and I've just been spending some money and considering spending more on even a partial archiving solution for that. DK has this thing called "bookmark mode" where basically like you know it will only archive the stuff you bookmark. And because your bookmarks are synced from your Chrome mobile browsing to your Chrome desktop the next time you're on your desktop and running DK there it will detect those new bookmarks from your mobile browsing and it will archive and index them for you. > If this could support that use case, you'd have a chance at a paying customer for life. As it is, you've already told me you don't want me, or anyone with a similar use case, as a customer at any price. So there's plenty of space for you in the current system! :) > Why build it with an intimate dependency on Chrome, and not as a general HTTP/HTTPS proxy? I thought about doing that (not the general HTTP proxy idea but maybe I should look into that) I thought about doing that by somehow hacking a way to get the remote debugging port to work on the device (Android or iPhone) versions of Chrome. Basically I think the only way that it opens is if you have your phone plugged in to the Android debugger and hooked up the USB debugging or something like that so I can set it and it may be possible with like a little kind of USB type of dongle stuck in the bottom of your phone or something they may be a way to sort of, while the dongle is stuck in, you can archive your browser history... but I thought that's probably quite a number of steps from here to there.
  - throwanem 5y ago
    Well yeah, that's the thing about a proxy - ~every HTTP/HTTPS user agent supports them, and neither you nor your users need to mess around with Chrome-specific debugging stuff. Just set a proxy up in whatever browser, and all the traffic you want to archive goes through it with no extra effort. Fair warning, HTTPS proxying with packet inspection can be complicated since the proxy needs to terminate TLS in order to see the content; it's exactly like a MITM attack, but in this case performed among friends. This can be problematic in an environment where certificate pinning also exists, but that can be overcome for browsers at least if the root of the proxy's MITM cert is trusted in the system's or browser's certificate store. This is trivial as far as I know on all desktop OSes and on Android; I think you can only do it at the system level in iOS via an MDM profile, and I'm not sure how complicated those are to generate but iirc Boxen and similar OSS Mac config management tools do it, so may serve as a useful reference on the mechanics.
graderjs 5y ago
Replying to an (inexplicably, as it was high quality) deleted comment: > I thought the same thing 10 years ago. I have(had?) many thousands of bookmarks collected over 10 years. All of the most interesting things I ever read. Imagine my disappointment when I eventually realized most of them are now gone. Site is gone, URL is changed, videos taken down (so many of them it's amazing), websites weren't properly archived etc. > I tried to curl all my bookmarks for a half-assed solution but dynamic loading and unusable content presentation without running all js and saving dependencies, timestamps, replies etc makes it impossible. Even worse, it's impossible to discover something relevant when there is no search function. What you get is the page title and a URL. Usually both of them useless. I tried categorizing them and adding comments but that was a lot of work and led to me deleting stuff that are hard to categorize. I'm not a professional programmer and I'm definitely not fully aware of how modern websites work (particularly on the front end). > But if someone provided a locally run telemetry-free open-source (so it's auditable) searchable archive-as-you-browse tool that legitimately works reliably and with some kind of hard (technical) guarantee that it won't break with the next browser update, I'd pay good money for it. This is basically what DK is except, I have no guarantee as it depends on Chrome Remote Debugging Protocol (aka DevTools API.). I could hack you a guarantee by packaging it with a fixed version of Chrome (thank you for this idea I have never thought that that would be something people would give it their purchase on but thank you very much for informing me of this it's very valuable to know thank you) which would guarantee that all your sites archived up to the version of Chrome used to archive them would continue to work into the future because they worked with that version of Chrome in the first place...and then you would have access to that version of Chrome to run the archive. Aside from that we can have the hand waving heuristic argument (which is not a hard technical guarantee) that Chrome has invested a lot in puppeteer and the whole browser automation API thing is blowing up and it's very big among developers and supported by a variety of browser vendors (and on track to become a bona fide w3c standard if it's not already the web driver or web devital standard I mix up the name because there was a historical one as well) and there's lots of forks of browsers so the idea that some part of the puppeteer API (CRDP API), as in the ability to intercept requests, will go away at some point in the future seems kind of unlikely but it's not something you can guarantee. Although in the limit I think the idea of packaging it with a historical version of Chrome works because in the limit your archive size is going to be larger than the 60 to 100 MB size of the Chrome binary.
d4rkp4ttern 5y ago
Similar project I came across some time ago: Browser Parrot https://www.browserparrot.com/ https://www.browserparrot.com/
harabat 5y ago
Related: https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox HN thread: https://news.ycombinator.com/item?id=27842740 https://news.ycombinator.com/item?id=27842740

35 points by graderjs 5y ago

Looks fun reminds me of Google Desktop search from way back in the day - you know back in the days of google also selling a search appliance for enterprise data centers.

connordoner 5y ago
Lol… “Did you mean ‘the cloud’?”
- Jaruzel 5y ago
  No, they didn't. Back in the day Google sold 'Google-In-A-Box' which was a rack mounted hardware device that you installed into your company's server rack (either on-prem or in a hosted data centre) and once configured it crawled all your internal resources and presented an internal search engine with all the power of Google search. Although a good idea in principle, I found it cumbersome to configure and it only (at the time) supported ftp and http based resources - if your company made heavy use of Windows file shares you couldn't index them without also serving them over ftp or http. I believe later versions improved on that though, but by then it was a dead product, and Google discontinued it.
  - connordoner 5y ago
    No, I know — I remember the CSA well. It was a play on words on Google’s “Did you mean?” functionality.
    - Jaruzel 5y ago
      Aha. :D

belkarx 5y ago

Interesting idea however I can't see many practical uses for this ... how often do you really need to search through the text of something you browsed some time ago and would it really offset the disk-space cost? I personally would have benefited from this on a few occasions however SQL+title-keywords+timestamps were effective enough and it's a rare occurrence. And if you need to archive something, why do it locally when there are projects like the Wayback machine (centralized is an argument against it I suppose), or why do it for everything?

aryamaan 5y ago
I want to do for the tweets all the time.
deleted 5y ago
[deleted]
graderjs 5y ago
Everyone is different. Sometimes even just saving someone that inconvenience and time and stress in that edge case where like you have this bookmark that you know that you had or this page that you visited but you can't find it again through a search engine, sometimes just saving people the headache of going through that is a really valuable thing like even if it only occurs a couple of times per year. Personally I couldn't understand why people were so into this project when I created it. I didn't understand the potential uses when I created it. I've liked that people have shown me, and also that I've had to market this way less than my other products. I created it as like you know a technical proof of concept kind of hack because I wanted to see if it was possible and then I kind of discovered that there were all these like archive nuts who were really into archiving. And then over time I started to think well sometimes I've got a bookmark but I can't remember how to find it and I couldn't find it in in the main search engine but I knew that I bookmarked it somewhere but I can't identify it from the title search. So having a full text search over the page content like a personal local search engine I thought it was going to be useful. But the main cool thing I think is the fact that you know you can have pages running as if they're on the internet when you're offline. So on the plane say you can have all of this you know mdn stuff archived all of these useful resources useful documents and then you can be running on the plane you don't have to use the Wi-Fi but you can have this like offline internet so you can keep working. That's me and my perspective maybe let me finish with the words of someone else who commented here just a little while ago but then deleted the comment, but I thought it was a really good perspective so I'll share it below > I thought the same thing 10 years ago. I have(had?) many thousands of bookmarks collected over 10 years. All of the most interesting things I ever read. Imagine my disappointment when I eventually realized most of them are now gone. Site is gone, URL is changed, videos taken down (so many of them it's amazing), websites weren't properly archived etc. > I tried to curl all my bookmarks for a half-assed solution but dynamic loading and unusable content presentation without running all js and saving dependencies, timestamps, replies etc makes it impossible. Even worse, it's impossible to discover something relevant when there is no search function. What you get is the page title and a URL. Usually both of them useless. I tried categorizing them and adding comments but that was a lot of work and led to me deleting stuff that are hard to categorize. I'm not a professional programmer and I'm definitely not fully aware of how modern websites work (particularly on the front end). > But if someone provided a locally run telemetry-free open-source (so it's auditable) searchable archive-as-you-browse tool that legitimately works reliably and with some kind of hard (technical) guarantee that it won't break with the next browser update, I'd pay good money for it.

aabbcc1241 5y ago

I made something similar, with and/or/not full text searching https://github.com/beenotung/personal-search-engine https://github.com/beenotung/personal-search-engine

hamsterbase 5y ago
I made something similar too https://github.com/hamsterbase/hamsterbase https://github.com/hamsterbase/hamsterbase Render and upload mhtml(chrome)、webarchive(safari) website. full text searching is WIP

throwanem 5y ago

Why build it with an intimate dependency on Chrome, and not as a general HTTP/HTTPS proxy? Not that it doesn't look neat, sure. But I do a lot of browsing from my phone, and I've just been spending some money and considering spending more on even a partial archiving solution for that. If this could support that use case, you'd have a chance at a paying customer for life. As it is, you've already told me you don't want me, or anyone with a similar use case, as a customer at any price. I assume there has to be some engineering justification for a tradeoff like that, but I admit I can't fathom what it might be.

graderjs 5y ago
> Not that it doesn't look neat, sure. But I do a lot of browsing from my phone, and I've just been spending some money and considering spending more on even a partial archiving solution for that. DK has this thing called "bookmark mode" where basically like you know it will only archive the stuff you bookmark. And because your bookmarks are synced from your Chrome mobile browsing to your Chrome desktop the next time you're on your desktop and running DK there it will detect those new bookmarks from your mobile browsing and it will archive and index them for you. > If this could support that use case, you'd have a chance at a paying customer for life. As it is, you've already told me you don't want me, or anyone with a similar use case, as a customer at any price. So there's plenty of space for you in the current system! :) > Why build it with an intimate dependency on Chrome, and not as a general HTTP/HTTPS proxy? I thought about doing that (not the general HTTP proxy idea but maybe I should look into that) I thought about doing that by somehow hacking a way to get the remote debugging port to work on the device (Android or iPhone) versions of Chrome. Basically I think the only way that it opens is if you have your phone plugged in to the Android debugger and hooked up the USB debugging or something like that so I can set it and it may be possible with like a little kind of USB type of dongle stuck in the bottom of your phone or something they may be a way to sort of, while the dongle is stuck in, you can archive your browser history... but I thought that's probably quite a number of steps from here to there.
- throwanem 5y ago
  Well yeah, that's the thing about a proxy - ~every HTTP/HTTPS user agent supports them, and neither you nor your users need to mess around with Chrome-specific debugging stuff. Just set a proxy up in whatever browser, and all the traffic you want to archive goes through it with no extra effort. Fair warning, HTTPS proxying with packet inspection can be complicated since the proxy needs to terminate TLS in order to see the content; it's exactly like a MITM attack, but in this case performed among friends. This can be problematic in an environment where certificate pinning also exists, but that can be overcome for browsers at least if the root of the proxy's MITM cert is trusted in the system's or browser's certificate store. This is trivial as far as I know on all desktop OSes and on Android; I think you can only do it at the system level in iOS via an MDM profile, and I'm not sure how complicated those are to generate but iirc Boxen and similar OSS Mac config management tools do it, so may serve as a useful reference on the mechanics.

graderjs 5y ago

Replying to an (inexplicably, as it was high quality) deleted comment: > I thought the same thing 10 years ago. I have(had?) many thousands of bookmarks collected over 10 years. All of the most interesting things I ever read. Imagine my disappointment when I eventually realized most of them are now gone. Site is gone, URL is changed, videos taken down (so many of them it's amazing), websites weren't properly archived etc. > I tried to curl all my bookmarks for a half-assed solution but dynamic loading and unusable content presentation without running all js and saving dependencies, timestamps, replies etc makes it impossible. Even worse, it's impossible to discover something relevant when there is no search function. What you get is the page title and a URL. Usually both of them useless. I tried categorizing them and adding comments but that was a lot of work and led to me deleting stuff that are hard to categorize. I'm not a professional programmer and I'm definitely not fully aware of how modern websites work (particularly on the front end). > But if someone provided a locally run telemetry-free open-source (so it's auditable) searchable archive-as-you-browse tool that legitimately works reliably and with some kind of hard (technical) guarantee that it won't break with the next browser update, I'd pay good money for it. This is basically what DK is except, I have no guarantee as it depends on Chrome Remote Debugging Protocol (aka DevTools API.). I could hack you a guarantee by packaging it with a fixed version of Chrome (thank you for this idea I have never thought that that would be something people would give it their purchase on but thank you very much for informing me of this it's very valuable to know thank you) which would guarantee that all your sites archived up to the version of Chrome used to archive them would continue to work into the future because they worked with that version of Chrome in the first place...and then you would have access to that version of Chrome to run the archive. Aside from that we can have the hand waving heuristic argument (which is not a hard technical guarantee) that Chrome has invested a lot in puppeteer and the whole browser automation API thing is blowing up and it's very big among developers and supported by a variety of browser vendors (and on track to become a bona fide w3c standard if it's not already the web driver or web devital standard I mix up the name because there was a historical one as well) and there's lots of forks of browsers so the idea that some part of the puppeteer API (CRDP API), as in the ability to intercept requests, will go away at some point in the future seems kind of unlikely but it's not something you can guarantee. Although in the limit I think the idea of packaging it with a historical version of Chrome works because in the limit your archive size is going to be larger than the 60 to 100 MB size of the Chrome binary.

d4rkp4ttern 5y ago

Similar project I came across some time ago: Browser Parrot https://www.browserparrot.com/ https://www.browserparrot.com/

harabat 5y ago

Related: https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox HN thread: https://news.ycombinator.com/item?id=27842740 https://news.ycombinator.com/item?id=27842740