5 ms·

$ pdftotext | grep sometext ^^ first advantage Also: a) Formatting is not lost, it just changes to fit the default paper size I've got selected (A4) but doesn

$ pdftotext | grep sometext ^^ first advantage Also: a) Formatting is not lost, it just changes to fit the default paper size I've got selected (A4) but doesn't really make much difference, since its a snapshot, and b) URL is right there in the Header of the PDF, and is clickable, so no - not really an issue. This archive also functions as a bookmark collection as well as an offline copy for future reference .. (Disclaimer: may be that your browser is borking the PDF's. Not the case with Safari, anyway, but ymmv..)

pvg 6y ago
You can grep the saved archives and they often save working copies of local interactive content in a way PDF doesn't. Internal structure and annotation is also preserved. I'm not sure I understand the formatting comment, you seem to be saying formatting is not lost and supporting that with an example of how formatting is lost. Don't get me wrong, it should definitely be easier to save, index and otherwise manipulate web pages. But out of the the trivial methods, 'print to PDF' is one of the poorer methods.
- fit2rule 6y ago
  It depends on the site - but I haven't found 'lost formatting' to be an issue at all - since, when I want to do a granular search I'm using 'pdftotext' to search on plaintext, and when I find a PDF of interest, I open it and can go directly back to the web page from which it was printed by way of the footer/header which contains a clickable URL. Most of the time though, the formatting isn't an issue. It depends on the site though - some authors produce stuff that doesn't look good as PDF, even if the content is still there. That doesn't bug me much.
  - pvg 6y ago
    Ok, so we seem to agree print-to-pdf loses formatting. I share your interest in and fascination with this (weirdly irksome and edgecasey) problem but just about any modern browser provides better facilities for saving web pages with higher fidelity than 'print to pdf'. Print to pdf is so easy to beat, you'd have to go out of your way to find a way to not-beat it - say, saving just 'page source'.
    - fit2rule 6y ago
      PDF gives me an off-line readable version of the website, and is pretty compatible with pdftotext as a pipeline tool... I'm not sure that formatting is such a huge issue - I'm yet to find a web page I can't extract some meaningful info from, later on ..
      - pvg 6y ago
        an off-line readable version of the website As does 'Save as: Web Archive' in Safari or, if you tell it to store offline, 'Add to Reading List'. In Chrome you can save pages to .mht. All of these are single-keystroke, better ways to locally archive a web page.
        fit2rule 6y ago
        Cool. Let me know when I can extract the full text contents from such files using common, built-in tools on your average fresh install of MacOS/Linux. PDF works just fine. It presents a feasible view of the original data, and allows for data harvesting with ease.
        pvg 6y ago
        Let me know when I can extract the full text You can extract the full text from these (with whatever tools you like) with better fidelity than you can from a pdf, which is a lossy conversion from the same source. This seems to barely merit debating, unless I'm missing something. PDF works just fine. I'm sure it works for you and I'm not harbouring any delusions I'm going to talk you out of your decades-established workflow. But for anyone looking for ways to keep track of web pages, thinking about building tools in this space, etc - no, PDF is not a good way to archive web pages, either manually or programmatically.
        fit2rule 6y ago
        >PDF is a lossy conversion I am not finding this to be true. Pretty much every PDF I have has been usable for extracting the text content - unless the Web authors intentionally work to obfuscate/disable this functionality, i.e. using images to display text content. >PDF is not a good way to archive web pages, either manually or programmatically. I disagree, entirely, with your conclusion - you haven't made a strong argument. 20,000+ fully-searchable, indexable, accessible-in-offline PDF files vs. your opinion so far. I don't see any of the issues you've stated are insurmountable - in fact, I find the reality to be completely the opposite to your stated opinion. Please expand on this if you have the energy.
ben509 6y ago
Grep is not much of an advantage when Mac, Windows and Linux all have as-you-type full-text search of common formats like HTML and PDF.
- anonymoushn 6y ago
  Can you point me to the as-you-type full-text search on Mac for a directory transitively containing 20,000 pdfs?
  - salutis 6y ago
    I use Apple’s built-in Spotlight search to search in Markdown files and PDF documents, albeit I do not have 20,000 of them. Would it be a problem for Spotlight to index 20,000 PDF documents?
  - ben509 6y ago
    Sure, Spotlight has been around for at least a decade; it's the little magnifying glass at the right end of the main menubar. It definitely indexes PDFs with a text layer as well as other document formats; I've been running them through OCR for ages and have 28,000 in my records. You can also install extensions for other formats. It indexes your whole disk; if there's a limit as to how many files it will index, I've yet to reach it. I checked Apple's developer documentation, and they don't mention one.
- fit2rule 6y ago
  Pro's for grep/silversearcher/etc.: quite fast, quite efficient, redirect-able to other tools, maybe a little sqlite here, maybe a bit of xml/svg there and most important of all: private, under my control... mdimporter: yeah, sure, just no .. mmkay?

pvg 6y ago

You can grep the saved archives and they often save working copies of local interactive content in a way PDF doesn't. Internal structure and annotation is also preserved. I'm not sure I understand the formatting comment, you seem to be saying formatting is not lost and supporting that with an example of how formatting is lost. Don't get me wrong, it should definitely be easier to save, index and otherwise manipulate web pages. But out of the the trivial methods, 'print to PDF' is one of the poorer methods.

fit2rule 6y ago
It depends on the site - but I haven't found 'lost formatting' to be an issue at all - since, when I want to do a granular search I'm using 'pdftotext' to search on plaintext, and when I find a PDF of interest, I open it and can go directly back to the web page from which it was printed by way of the footer/header which contains a clickable URL. Most of the time though, the formatting isn't an issue. It depends on the site though - some authors produce stuff that doesn't look good as PDF, even if the content is still there. That doesn't bug me much.
- pvg 6y ago
  Ok, so we seem to agree print-to-pdf loses formatting. I share your interest in and fascination with this (weirdly irksome and edgecasey) problem but just about any modern browser provides better facilities for saving web pages with higher fidelity than 'print to pdf'. Print to pdf is so easy to beat, you'd have to go out of your way to find a way to not-beat it - say, saving just 'page source'.
  - fit2rule 6y ago
    PDF gives me an off-line readable version of the website, and is pretty compatible with pdftotext as a pipeline tool... I'm not sure that formatting is such a huge issue - I'm yet to find a web page I can't extract some meaningful info from, later on ..
    - pvg 6y ago
      an off-line readable version of the website As does 'Save as: Web Archive' in Safari or, if you tell it to store offline, 'Add to Reading List'. In Chrome you can save pages to .mht. All of these are single-keystroke, better ways to locally archive a web page.
      - fit2rule 6y ago
        Cool. Let me know when I can extract the full text contents from such files using common, built-in tools on your average fresh install of MacOS/Linux. PDF works just fine. It presents a feasible view of the original data, and allows for data harvesting with ease.
        pvg 6y ago
        Let me know when I can extract the full text You can extract the full text from these (with whatever tools you like) with better fidelity than you can from a pdf, which is a lossy conversion from the same source. This seems to barely merit debating, unless I'm missing something. PDF works just fine. I'm sure it works for you and I'm not harbouring any delusions I'm going to talk you out of your decades-established workflow. But for anyone looking for ways to keep track of web pages, thinking about building tools in this space, etc - no, PDF is not a good way to archive web pages, either manually or programmatically.
        fit2rule 6y ago
        >PDF is a lossy conversion I am not finding this to be true. Pretty much every PDF I have has been usable for extracting the text content - unless the Web authors intentionally work to obfuscate/disable this functionality, i.e. using images to display text content. >PDF is not a good way to archive web pages, either manually or programmatically. I disagree, entirely, with your conclusion - you haven't made a strong argument. 20,000+ fully-searchable, indexable, accessible-in-offline PDF files vs. your opinion so far. I don't see any of the issues you've stated are insurmountable - in fact, I find the reality to be completely the opposite to your stated opinion. Please expand on this if you have the energy.

ben509 6y ago

Grep is not much of an advantage when Mac, Windows and Linux all have as-you-type full-text search of common formats like HTML and PDF.

anonymoushn 6y ago
Can you point me to the as-you-type full-text search on Mac for a directory transitively containing 20,000 pdfs?
- salutis 6y ago
  I use Apple’s built-in Spotlight search to search in Markdown files and PDF documents, albeit I do not have 20,000 of them. Would it be a problem for Spotlight to index 20,000 PDF documents?
- ben509 6y ago
  Sure, Spotlight has been around for at least a decade; it's the little magnifying glass at the right end of the main menubar. It definitely indexes PDFs with a text layer as well as other document formats; I've been running them through OCR for ages and have 28,000 in my records. You can also install extensions for other formats. It indexes your whole disk; if there's a limit as to how many files it will index, I've yet to reach it. I checked Apple's developer documentation, and they don't mention one.
fit2rule 6y ago
Pro's for grep/silversearcher/etc.: quite fast, quite efficient, redirect-able to other tools, maybe a little sqlite here, maybe a bit of xml/svg there and most important of all: private, under my control... mdimporter: yeah, sure, just no .. mmkay?