4 ms·

MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a s

MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).

flatroze 7y ago
Apparently everybody knew about MHTML but me Ü I'm going to look into that format and see if I could enhance monolith to output proper MHTML, among other additions and improvements. Thank you for the info!
- masklinn 7y ago
  I don't know that it would be a very useful thing to do at least in the short term: there's a bunch of "web archive" formats out there and the common thread between them is that they're custom archive formats, you need special clients or support for those formats: * mthml encodes the page as a multipart MIME message (using multipart/related), essentially an email (you're usually able to open them by replacing the .mth by .eml) * WARC is its own thing with its own spec * WAFF is a zipfile, not sure about the specifics * webarchive is a binary plist, not sure about the specifics either Your tool generates straight HTML which any browser should be able to open. It probably has more limitations, but it doesn't require dedicated client / viewer support. Maybe once you've got all the fetching and extracting and linking nailed down it would be a nice extension to add "output filters", but that seems more like a secondary long-term goal, especially as those archive formats are usually semi-proprietary and get dropped as fast as they get created (WARC might be the most long-lived as it descends from the Internet Archive's ARC, is an ISO standard and is recognised as a proper archival format by various national libraries).
  - mftrhu 7y ago
    There isn't much to WAFF. Each WAFF file can contain more than one saved page. Each page needs to be contained within its own folder (whose name is usually the timestamp of when the page was saved, but it doesn't matter AFAICT). There can be an `index.rdf` file in there, to specify metadata and which file to open, but otherwise you should look for an `index.SOMETHING` file - usually `index.html`. E.g. test.maff `-- 1566561512/ |-- index.rdf |-- index.html `-- index_files/ `-- ??? When I was messing around with archiving things locally I settled on WAFF, because it's pretty much trivial to create and to use. Even if your browser does not support it, you just need to unpack it to a tempdir and open the index file.
- japanuspus 7y ago
  I had never heard about MHTML either. Another use case could be embedding markdown source for the HTML in the document as well. This would allow single-file documents with figures that could be edited as light markup (with some tooling) and be viewed by anyone with a browser. This is something I have been dreaming about for years! Tbh. I had arrived at the conclusion that Mime would be great, but it never struck me that someone had already made a "standard" of mime and HTML.
Causality1 7y ago
Always struck me as quite odd MHTML fell out of favor. Back in the day when I wanted to preserve a web page it was the logical choice since you could just click "save as archive".
- flatroze 7y ago
  We'll bring it back, don't you worry!
- da_chicken 7y ago
  Blame XMLHttpRequest, Flash, JS, and embedded video. It doesn't make sense to archive a document when the necessary interactive content elements will essentially fail when opened offline.
  - paggle 7y ago
    What would be cool is one package with the original HTML file, the fully rendered DOM written out as a second file+assets package, and a PDF just in case the first two get fucked.
  - jordwalke 7y ago
    You can first prerender the page with Chrome in headless mode (see my other comment), and then convert it into a single document using an inlining tool (such as the OP's). That way the JS will run and render the page (see my other comments here for an example).
masklinn 7y ago
> Similarly stores assets as base64'd data URIs and saves it as a single file. Does it? IIRC it stores assets as MIME attachments, hence the "M": the result is not HTML (which this would I assume be), it's a multipart MIME message whose root is an HTML document. edit: in fact when downloading mht files osx / safari misrecognises them as exported emails and appends the "eml" extension.
jordwalke 7y ago
I'm not aware of a way to save as MHTML from Chrome in headless mode (from the command line). Are you?
- bhl 7y ago
  You can either use the flag --save-page-as-mhtml from CLI [1] or a library such as puppeteer. [1] https://techdows.com/2019/06/google-removes-save-page-as-mhtml-flag-from-chrome.html https://techdows.com/2019/06/google-removes-save-page-as-mht... [2] https://github.com/GoogleChrome/puppeteer https://github.com/GoogleChrome/puppeteer
jordwalke 7y ago
It appears that (at least) Safari cannot open mhtml files. The benefit of a tool such as what the OP shared is that it can produce plain html pages that are openable by anyone. (also, I tried mhtml in Chrome using the proper flag and it doesn't appear to store/inline/render static assets correctly).
bhl 7y ago
One issue with MHTML is that it does not seem to be currently supported by iframes. The use case I was working on was comparing search results from Google and DuckDuckGo by simply scraping and downloading to later embed. For that, I used a cli tool from an open source library [1]. MHTML seems like a nice format but I'm not sure if there's a library to convert them into stand-alone HTML files. [1] https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile Edit: This question just came to mind. If MHTML saves images using base64, and base64 dataurl images have a limit size, how would you save extremely large photos? Take for example the cover image of this article https://story.californiasunday.com/gone-paradise-fire https://story.californiasunday.com/gone-paradise-fire. When I saved the page in MHTML format, the re-rendered image showed up quite blurry compared to the original. Was the size limit the cause? [2] https://stackoverflow.com/questions/12637395/what-is-the-size-limit-of-a-base64-dataurl-image https://stackoverflow.com/questions/12637395/what-is-the-siz...
- app4soft 7y ago
  > One issue with MHTML is that it does not seem to be currently supported by iframes. There no needs to insert MHTML page into IFrame! If you need insert MHTML content into IFrame, just convert it to HTML+JS firstly.
- Springtime 7y ago
  > MHTML seems like a nice format but I'm not sure if there's a library to convert them into stand-alone HTML files. Many years ago used a program that could convert them but the name escapes me. A brief search shows a few results that appear to do a similar conversion, potentially they may be of use. > This question just came to mind. If MHTML saves images using base64, and base64 dataurl images have a limit size, how would you save extremely large photos? From what I've read there's no official limit for base64 encodings though IE/Edge limit them to 4GB according to caniuse.com. Haven't personally encountered an image or GIF that was too large not to be saved in an MHT (I have over 10k MHTML files saved, some image and GIF heavy ones up to 200MB each). Also a sibling comment corrected me about the data URI use, MHTML uses a separate scheme but nevertheless still uses base64 for encoding. For that example article you linked it seems likely to be the way the program you're using is handling the Javascript-loaded images. I saved it in Vivaldi (Blink-based engine) and the main image displayed at full res when opened locally while the other images didn't, while when saved with a pre-Quantum Firefox using the UnMHT addon it saved all the images at their fully loaded resolutions. Some MHTML saving implementations clearly have advantages over others it would seem.

flatroze 7y ago

Apparently everybody knew about MHTML but me Ü I'm going to look into that format and see if I could enhance monolith to output proper MHTML, among other additions and improvements. Thank you for the info!

masklinn 7y ago
I don't know that it would be a very useful thing to do at least in the short term: there's a bunch of "web archive" formats out there and the common thread between them is that they're custom archive formats, you need special clients or support for those formats: * mthml encodes the page as a multipart MIME message (using multipart/related), essentially an email (you're usually able to open them by replacing the .mth by .eml) * WARC is its own thing with its own spec * WAFF is a zipfile, not sure about the specifics * webarchive is a binary plist, not sure about the specifics either Your tool generates straight HTML which any browser should be able to open. It probably has more limitations, but it doesn't require dedicated client / viewer support. Maybe once you've got all the fetching and extracting and linking nailed down it would be a nice extension to add "output filters", but that seems more like a secondary long-term goal, especially as those archive formats are usually semi-proprietary and get dropped as fast as they get created (WARC might be the most long-lived as it descends from the Internet Archive's ARC, is an ISO standard and is recognised as a proper archival format by various national libraries).
- mftrhu 7y ago
  There isn't much to WAFF. Each WAFF file can contain more than one saved page. Each page needs to be contained within its own folder (whose name is usually the timestamp of when the page was saved, but it doesn't matter AFAICT). There can be an `index.rdf` file in there, to specify metadata and which file to open, but otherwise you should look for an `index.SOMETHING` file - usually `index.html`. E.g. test.maff `-- 1566561512/ |-- index.rdf |-- index.html `-- index_files/ `-- ??? When I was messing around with archiving things locally I settled on WAFF, because it's pretty much trivial to create and to use. Even if your browser does not support it, you just need to unpack it to a tempdir and open the index file.
japanuspus 7y ago
I had never heard about MHTML either. Another use case could be embedding markdown source for the HTML in the document as well. This would allow single-file documents with figures that could be edited as light markup (with some tooling) and be viewed by anyone with a browser. This is something I have been dreaming about for years! Tbh. I had arrived at the conclusion that Mime would be great, but it never struck me that someone had already made a "standard" of mime and HTML.

Causality1 7y ago

Always struck me as quite odd MHTML fell out of favor. Back in the day when I wanted to preserve a web page it was the logical choice since you could just click "save as archive".

flatroze 7y ago
We'll bring it back, don't you worry!
da_chicken 7y ago
Blame XMLHttpRequest, Flash, JS, and embedded video. It doesn't make sense to archive a document when the necessary interactive content elements will essentially fail when opened offline.
- paggle 7y ago
  What would be cool is one package with the original HTML file, the fully rendered DOM written out as a second file+assets package, and a PDF just in case the first two get fucked.
- jordwalke 7y ago
  You can first prerender the page with Chrome in headless mode (see my other comment), and then convert it into a single document using an inlining tool (such as the OP's). That way the JS will run and render the page (see my other comments here for an example).

masklinn 7y ago

> Similarly stores assets as base64'd data URIs and saves it as a single file. Does it? IIRC it stores assets as MIME attachments, hence the "M": the result is not HTML (which this would I assume be), it's a multipart MIME message whose root is an HTML document. edit: in fact when downloading mht files osx / safari misrecognises them as exported emails and appends the "eml" extension.

jordwalke 7y ago

I'm not aware of a way to save as MHTML from Chrome in headless mode (from the command line). Are you?

bhl 7y ago
You can either use the flag --save-page-as-mhtml from CLI [1] or a library such as puppeteer. [1] https://techdows.com/2019/06/google-removes-save-page-as-mhtml-flag-from-chrome.html https://techdows.com/2019/06/google-removes-save-page-as-mht... [2] https://github.com/GoogleChrome/puppeteer https://github.com/GoogleChrome/puppeteer

jordwalke 7y ago

It appears that (at least) Safari cannot open mhtml files. The benefit of a tool such as what the OP shared is that it can produce plain html pages that are openable by anyone. (also, I tried mhtml in Chrome using the proper flag and it doesn't appear to store/inline/render static assets correctly).

bhl 7y ago

One issue with MHTML is that it does not seem to be currently supported by iframes. The use case I was working on was comparing search results from Google and DuckDuckGo by simply scraping and downloading to later embed. For that, I used a cli tool from an open source library [1]. MHTML seems like a nice format but I'm not sure if there's a library to convert them into stand-alone HTML files. [1] https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile Edit: This question just came to mind. If MHTML saves images using base64, and base64 dataurl images have a limit size, how would you save extremely large photos? Take for example the cover image of this article https://story.californiasunday.com/gone-paradise-fire https://story.californiasunday.com/gone-paradise-fire. When I saved the page in MHTML format, the re-rendered image showed up quite blurry compared to the original. Was the size limit the cause? [2] https://stackoverflow.com/questions/12637395/what-is-the-size-limit-of-a-base64-dataurl-image https://stackoverflow.com/questions/12637395/what-is-the-siz...

app4soft 7y ago
> One issue with MHTML is that it does not seem to be currently supported by iframes. There no needs to insert MHTML page into IFrame! If you need insert MHTML content into IFrame, just convert it to HTML+JS firstly.
Springtime 7y ago
> MHTML seems like a nice format but I'm not sure if there's a library to convert them into stand-alone HTML files. Many years ago used a program that could convert them but the name escapes me. A brief search shows a few results that appear to do a similar conversion, potentially they may be of use. > This question just came to mind. If MHTML saves images using base64, and base64 dataurl images have a limit size, how would you save extremely large photos? From what I've read there's no official limit for base64 encodings though IE/Edge limit them to 4GB according to caniuse.com. Haven't personally encountered an image or GIF that was too large not to be saved in an MHT (I have over 10k MHTML files saved, some image and GIF heavy ones up to 200MB each). Also a sibling comment corrected me about the data URI use, MHTML uses a separate scheme but nevertheless still uses base64 for encoding. For that example article you linked it seems likely to be the way the program you're using is handling the Javascript-loaded images. I saved it in Vivaldi (Blink-based engine) and the main image displayed at full res when opened locally while the other images didn't, while when saved with a pre-Quantum Firefox using the UnMHT addon it saved all the images at their fully loaded resolutions. Some MHTML saving implementations clearly have advantages over others it would seem.