11 ms·
Webrecorder: Make an interactive copy of any web page that you browse
- boredgamer2 6y agoVery cool. This seems like such a "Duh. Users want this" feature. I wish it was integrated in Firefox years ago. I bookmark some sites and then come back years later, but perhaps they hijacked the URL or they later changed the URL's parameters. And then when I return, I get a 404. Instead of bookmarking, I'd love to "capture" the current info, divs, and graphics.
- Cymen 6y agoJoplin is a local thing you can run for this. Basically, saving a snapshot of a page. Has browser extensions. https://joplinapp.org/ https://joplinapp.org/ I like the ability to use tags too -- I've got various product/tech ideas and it's nice collecting information with it and not having to worry about the pages going away or changing.
- hanklazard 6y ago+1 for the Joplin web clipper. I really like using Joplin for notes, especially the command line version.
- PenguinCoder 6y agoNot a firefox extension, but I moved to using Polar Bookshelf[0] for exactly this reason. [0] https://getpolarized.io/ https://getpolarized.io/
- milofeynman 6y agoI have been searching for a tool exactly like this. Thank you for sharing!
- pzmarzly 6y agoWhile it's far from an ideal solution, you can throw your bookmarks periodically to archive.org's "Save Page Now!" service. It's easy to semi-automate it - here's how I use it with pinboard.in bookmark exports: https://pastebin.com/uUVE22RD https://pastebin.com/uUVE22RD
- holidaygoose 6y agoThis solution is on the user side, which is great because each person can get and manage saved pages for themselves. But if we're looking for a developer side solution, then making pages that last an order of magnitude longer may be better for everyone in the long run, e.g. https://jeffhuang.com/designed_to_last/ https://jeffhuang.com/designed_to_last/
- terrycody 6y agoarchive.fo is also very good to use.
- zxter 6y agoPinboard [0] has this feature with additional price. Just bookmark and you are done. Later you can text search in their content. [0] https://www.pinboard.in/ https://www.pinboard.in/
- derefr 6y agoDoes it capture your view of the page, though (i.e. use your own browser, with its cookie jar, to do the scrape)? I'd like to, for example, snapshot my Facebook feed.
- jabroni_salad 6y agoNo it does not. That means it also doesnt work for any news sites that you have subscriptions to. Im using the joplin web clipper pretty heavily for this purpose.
- zxter 6y agoTnen check out historysearch [0] If I remember correctly they index your history so it could include your history. You don't even need to bookmark. Of course I cannot vouch for their respect for privacy. [0] https://historysearch.com/ https://historysearch.com/
- ryanfox 6y agoI made an application[0] that does capture your view. It's screenshot-based. It works outside of a browser too - anything on your screen. All local too, so no privacy concerns :) [0] https://apse.io https://apse.io
- _curious_ 6y agoInteresting. How do you/does Apse make $
- oarsinsync 6y agoThe website shows a 'buy' button, and the pricing model.
- gildas 6y agoThere's an option for this ("Misc. > save the page of a newly created bookmark") in SingleFile [1]. [1] https://github.com/gildas-lormeau/SingleFile/ https://github.com/gildas-lormeau/SingleFile/
- MacSystem 6y agoAfraid of lose the contents on the page, I used to save it as PDF on cloud. Far from ideal but happened way too much I go back to the page and get a 404 page error or so.
- throwawaysea 6y agoI would love an automated solution for this that is also smart enough to use my logins to access subscription based news sites for archival purposes.
- kordlessagain 6y agoA smart crawler.
- m463 6y agoyou can do this, sort of. web developer menu toggle tools select ... select settings turn on "take a screenshot of the entire page" now you get a camera icon. clicking it snapshots the current webpage (as a veeery tall image)
- MasterYoda 6y agoYou could do that with the Firefox extension "SinleFile" [1]. It main purpose is to save a full webpage as one single html file with images etc encoded in the html file (no more messy html file + folder cluttering), very neat. There is also a variant that is a html/zip hybrid called SingleFileZ [2] that gets smaller saved file. In the settings you could set so that sites you bookmark is saved and also link the bookmark to the saved file locally if you want to. [1] https://addons.mozilla.org/en-US/firefox/addon/single-file/ https://addons.mozilla.org/en-US/firefox/addon/single-file/ [2] https://addons.mozilla.org/en-US/firefox/addon/singlefilez/ https://addons.mozilla.org/en-US/firefox/addon/singlefilez/
- gardaani 6y agoThat looks great! Firefox should have something like SingleFile built-in. Instead of just copying features from Chrome, they should lead the innovation and add useful features like this.
- renewiltord 6y agoI used this in like IE5 (I think) with MHTML. Though after looking a little, looks like SingleFile is more featureful (gets lazy loaded stuff etc)
- ordinary 6y agoI've been using SingleFile for a couple of months now, to auto-save every page I visit, and it works well. It's exactly as simple as it needs to be. My only real complaints are that it can be slow on large pages and that it pollutes your download history (every auto-save shows up as a download), both of which I suspect are due to limitations in the extension API.
- londons_explore 6y agoIf enough people did this, it would provide more pressure for websites not to just nuke their old URL schemes every time they switch from wordpress to Drupal...
- deleted 6y ago[deleted]
- Diederich 6y agoThe problem here is quite real; just because one has access to a remote resource now doesn't mean that access will remain, either in the short or long term. I'm not crazy about using a 3rd party service for it though. I've half-considered wiring up something with OBS to just record my web browser all day, but, besides the intense storage, indexing and searching is more than painful.
- jabroni_salad 6y agoIt looks like webrecorder has a downloadable version you can run entirely on local storage with no need for an account.
- greglindahl 6y agoWebrecorder is open source, and you can run your own instance if you like.
- mellosouls 6y agoYes - it's also a digital preservation project from a not-for-profit arts organisation. https://webrecorder.io/_faq https://webrecorder.io/_faq
- veenusav 6y agoI second
- ryanfox 6y agoHow about a local app? I am working on almost exactly what you're describing.
- deleted 6y ago[deleted]
- bacondude3 6y agoJust want to plug HTTrack [1] here as well. Not nearly as slick, but it's worked extremely well for me when Webrecorder couldn't do the job. Being usable through the command line also makes it useful for some projects WR can't do. [1]: http://www.httrack.com/ http://www.httrack.com/
- anigbrowl 6y agoSeconded, it ain't pretty but it's powerful.
- causality0 6y agoI came here to say the same thing. HTTRACK is incredibly handy especially when it comes to old pages. My personal use case is archiving small sites to make sure they don't go dark, for example webcomics. Having a single folder with all the comic image files is great.
- chrischen 6y agoI feel like the “Make website browsable offline” feature of yesteryears have been neglected. Somehow people assume everyone always has internet connectivity... but I need to save websites, docs, etc, for long flights or camping trips without signal. Safari annoyingly has a reading list feature that claims to cache the web page but annoyingly 50% of the time doesn’t. As with all apple cloud services there is just no way to explicitly sync.
- EamonnMR 6y agoOffline users don't generate data/ad impressions, so offline user experience isn't considered.
- giancarlostoro 6y agoOddly enough Mozilla gets crap for it but they bought Pocket which allows for offline reading. I think its fine for Mozilla to include Pocket. We seem to be fine installing Chrome which has much worse spyware. You can uninstall Pocket, you gotta go through more effort to uninstall Googles OS from Chrome.
- saurik 6y agoSince when is the Safari Reading List a cloud service? It might be that I just have most of iCloud disabled and that makes the system work as expected, but for me it is definitely 100% local (at which point I would presume failures are due to the mechanism it using to save things not working on all kinds of web pages: I honestly only use it for simple document sites, and so wouldn't know if it fails a lot on rich web app sites).
- saagarjha 6y agoI think perhaps they're annoyed that their Reading List isn't syncing between their devices and Apple hasn't put a button that forces the sync to occur.
- chrischen 6y agoYes, it is advertised as being able to make it available offline. But if you save it to reading list on iPhone, or your Mac, there's no guarantee it's available on the other device and often it even stops working on the device that I added to reading list.
- boromi 6y agoCan't we do this with OBS? What's the difference. Edit was going to download then saw it was an electron app.. no thanks.
- xfer 6y agoIt's Http Archive(HAR) recorder+indexer not a scree capture/mp4 video recorder program. You can use it with your webbrowser, you don't need an app. You can self-host the instance as well.
- dang 6y agoSurprisingly little prior discussion, but https://news.ycombinator.com/item?id=10838985 https://news.ycombinator.com/item?id=10838985 was related (2016).
- deleted 6y ago[deleted]
- seph-reed 6y agoThis seems like something that would work really well on the piHole level.
- jcahill 6y agoA pi isn't going to cut it here.
- edoceo 6y agoPi could for sure do the capture and save, it can run full or headless, and all the things. For personal itd be fine, not at scale tho
- dependenttypes 6y agoYou would have to intercept HTTPS connections and add your own CA to every computer that uses the pihole which is a pain.
- seph-reed 6y agoHmmm... https://security.stackexchange.com/questions/8145/does-https-prevent-man-in-the-middle-attacks-by-proxy-server https://security.stackexchange.com/questions/8145/does-https... I need to read more about CAs to figure out why the Pi couldn't fake it.
- jl6 6y agoI would love just a simple one-click print-current-page-to-PDF button that managed to capture the whole page, and did something intelligent about infinite scroll sites.
- eastendguy 6y agoTechnically interesting, but why would I use this over one of the many full page screenshot or "Print to PDF" browser extensions? That is what I use when I want to archive something.
- heinrichhartman 6y agoWhich browser and extension are you using? I tried a few, but could not get good results with any of them. print to pdf, output always looked terrible. Full page screenshot took ages to scroll throught the page hijacking my viewport. Those captures should happen in the background.
- eastendguy 6y agoFor manual capture I use https://chrome.google.com/webstore/detail/full-page-screen-capture/fdpohaocaechififmbbbbbknoalclacl https://chrome.google.com/webstore/detail/full-page-screen-c... and Chrome "Print to PDF". Layout issues are not important for my use case. For scheduled captures I automated this workflow with kantu: https://chrome.google.com/webstore/detail/uivision-rpa/gcbalfbdmfieckjlnblleoemohcganoc https://chrome.google.com/webstore/detail/uivision-rpa/gcbal... > Those captures should happen in the background. Yes, it would be nice if the Chrome extension api would allow full page screen captures to happen instanstly. Currently all extensions need to scroll up/down.
- gildas 6y ago> Currently all extensions need to scroll up/down. SingleFile does not. It can save lazy loaded contents without scrolling.
- jcahill 6y agoWeb archivists are coming at the problem with different requirements, related to fidelity and the systematic mirroring of address structure along with payload content and control information. By the sound of it, you likely wouldn't want to substantially change your personal filing system to have your page captures conform with those requirements.
- jcahill 6y agoWe use webrecorder for some interactive work at my workplace, a web archival nonprofit. If you're new to web archival, expect a learning curve.
- BiteCode_dev 6y agoFor a KISS version of this, there is the Single File add on: https://addons.mozilla.org/fr/firefox/addon/single-file/ https://addons.mozilla.org/fr/firefox/addon/single-file/ It will save any page as a standalone HTML file, including inlined external resources.
- mlok 6y agoI like the zipped-HTML version of this : https://addons.mozilla.org/fr/firefox/addon/singlefilez/ https://addons.mozilla.org/fr/firefox/addon/singlefilez/
- joepie91_ 6y agoDoesn't solve the same problem. Webrecorder is for saving browsing sessions, not individual pages.
- BiteCode_dev 6y agoHence the kiss
- abnry 6y agoI love this extension! I use it in place of bookmarks. I then run a cron job to move .html files from my downloads folder to a bookmarks folder. Then I generate thumbnails and an html index to easily browse my bookmarks. I feel so much more relaxed knowing I have can save with one click any good information I find on the web. Eventually I want to add NLP keyword extraction and categorization, and an internal search feature.
- ohmyblock 6y agoPocket has this as a paid feature
- heavyset_go 6y agoCheck out Shiori if you haven't, it does everything you want + archival.
- NelsonMinar 6y agoWow, neat! Is there a reliable screenshot version of this I can install on my own Linux system? Some sort of headless browser, I imagine. I realize this interactive system is much more powerful but it's overkill for an application I have in mind, a best effort single screenshot is fine. I've seen various attempts at doing this over the years and none have really worked reliably. I'd consider paying a service to do this but it seems like self-installable is better.
- heipei 6y agoCouple of options: You can build a small JavaScript package with a HTTP server to call your endpoint and puppeteer for actually running and interfacing with Chrome headless and then either Dockerize it or even run it as a Lambda / Google Cloud Function. Then there's https://github.com/browserless/chrome/ https://github.com/browserless/chrome/ which is both a commercial service as well as being Open Source. Plenty of other OSS and commercial options as well, ultimately it comes down to your use-case. Personally I've found reliably taking a screenshot at the right point in time challenging since a page is never really "done" loading, i.e. there could always be a refresh or timeout which loads another URLs, adds an iframe or changes something else entirely, so there is no single "yes, we're done loading now" event to wait for. Source: I run the service at https://urlscan.io https://urlscan.io where we have the same problem ;)
- asab 6y agoFirefox can screenshot the whole page - it's built in. I believe it can be done in headless as well. https://support.mozilla.org/en-US/kb/firefox-screenshots https://support.mozilla.org/en-US/kb/firefox-screenshots
- Nux 6y agoCan it screenshot lazy loading pages?
- williamdclt 6y agoI think it can be smart-ish about this (scrolling for you, but maybe I'm thinking about an extension). I've never found anything that handles virtualised lists/tables though, like screenshotting an entire Slack thread
- A4ET8a8uTh0 6y agoThis is neat. More and more stuff is hidden behind a login screen. Odd question. How does internet archive handle those types of pages these days? edit: types of
- mdaniel 6y agoI can't tell which spawned which, but there's a related discussion in r/DataHoarder about an extension that does that: https://old.reddit.com/r/DataHoarder/comments/ggyzoy/is_there_a_firefoxchrome_extension_that/ https://old.reddit.com/r/DataHoarder/comments/ggyzoy/is_ther... in which Webrecorder was mentioned yesterday
- amelius 6y ago> Webrecorder creates an interactive copy of any web page that you browse, including content revealed by your interactions such as (...) clicking buttons, and so forth. How can they guarantee this if the code may run on the server?
- netsharc 6y agoIt's probably a replay of what you did. I'm trying to think how the internals would be made using Javascript, but probably it's a case of recording changes to the DOM structure ("After 3 seconds, the password input field had the value 'hunter2'. After 5 seconds, a DIV appears with the the text 'Incorrect password'", etc).
- amelius 6y agoMakes sense, but I wouldn't call it an "interactive copy".
- jrochkind1 6y agoI am also just interpreting the copy on the OP, but I don't think it's just a replay -- I imagine they record all HTTP requests and responses, so can echo back the recorded response for any given JS query to server, when it's triggered. The page linked says: > Webrecorder takes a new approach to web archiving by capturing ("recording") network traffic and processes within the browser while you interact with a web page. Unlike conventional crawler-based web archiving methods, this allows even intricate websites, such as those with embedded media, complex Javascript, user-specific content and interactions, and other dynamic elements, to be captured and faithfully restaged. and: > Webrecorder creates an interactive copy of any web page that you browse, including content revealed by your interactions such as playing video and audio, scrolling, clicking buttons, and so forth. I think they record whatever HTTP transactions you trigger, so they can be played back when triggered in the replay.
- drewmol 6y ago>I think they record whatever HTTP transactions you trigger, so they can be played back when triggered in the replay. That could be a useful history for circumventing forced usability reductions when a site 'upgrades' but doesn't change the request structure and/or doesn't deny access/EOL a previous API version but just hides the old interface after a makeover. For example: I would love to have a recording of sessions from the previous version of ESPN's html fantasy football interface with a record of request to see if I could roll my own so to speak. (ESPN ruined the usability of the site by rewriting it in some Javascript framework hoping to annoy users into app conversions last year.)
- elil17 6y agoIs there a reason you can't just use archive.org for this?
- CGamesPlay 6y agoWell, at least two. For one, it might be desired to have a copy of the data yourself, rather than relying on some hosted service. Also, webrecorder would allow you to store authenticated pages.
- mosfets 6y agoI have been using this add-on in Chrome and thinks it's nice: https://chrome.google.com/webstore/detail/huula-web-clipper/holcfbedaepgjadjangfbmpdjnhkhdoa?hl=en https://chrome.google.com/webstore/detail/huula-web-clipper/...
- CGamesPlay 6y agoWebrecorder is quite good, but the necessity of using a separate program to record and to replay made this less desirable for me. I ended up making an app that integrated the two (conceptually, no technical relationship), which is rather outdated now but available here, should anyone be interested: https://github.com/CGamesPlay/chronicler https://github.com/CGamesPlay/chronicler
- sudhirkhanger 6y agoThere's Chrome's history, saved pages in Pocket, Bookmarks, possibly separate history in my mobile, etc. But the main issue is if that how do I get back the results when I want them in a seamless manner. How do I get them when I do a regular search using Google? Retrieval is the main problem.
- imvetri 6y agoFeedback on video: Video looks like machine recorded. Suggestion: During playback, smoothen the mouse movement. Videos will have smooth design appeal to sell your product better.
- ernsheong 6y agoI am the author of https://www.pagedash.com https://www.pagedash.com, which is a service that simply saves the entire page statically, and also makes an attempt at a dynamic save (allowing all initial JS to run). No network interaction is captured. Try it if you want something simpler. We've been around since 2017. See previous Show HN https://news.ycombinator.com/item?id=15653206 https://news.ycombinator.com/item?id=15653206
- fareesh 6y agoSlightly unrelated - but how would one dump the entire state of JS objects and the DOM onto the disk and then retrieve it at a later date? Hypothetical example - I'm playing some HTML5 game and I want to turn off my pc and return to it exactly as it was.
- busymom0 6y agoI am mostly a noob to JS but could this work? https://stackoverflow.com/questions/2934787/view-list-of-all-javascript-variables-in-google-chrome-console https://stackoverflow.com/questions/2934787/view-list-of-all...