4 ms·

This is really cool, nice work. In addition to censorship-circumvention, it also provides a way to read the content of potentially-malicious sites without risk

This is really cool, nice work. In addition to censorship-circumvention, it also provides a way to read the content of potentially-malicious sites without risk of browser vulnerabilities being exploited, and without revealing your IP address, User-Agent, etc. Now we can all browse the web like we're Richard Stallman :) EDIT: > try downloading it, if it is a file smaller than 5 MiB; I found that even pages smaller than 5MB get rendered to PDF. Are you using headless Chrome or similar to implement this? It seems Javascript isn't getting executed (which is probably wise, I was just surprised it renders the pages so faithfully if it's not also running some sort of engine that will execute JS). (Update: UA is "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/67.0.337") EDIT2: It doesn't seem to support chunked transfer-encoding: > Your request [...] has been unsuccessful due to following error: > file size unknown: "content-length" header is missing

heinrich5991 9y ago
>it also provides a way to read the content of potentially-malicious sites without risk of browser vulnerabilities being exploited PDFs have the same kind of vulnerabilities, they can even execute JS (JS is probably not contained in the ones generated by the author). >and without revealing your IP address, User-Agent, etc. Yep. >> try downloading it, if it is a file smaller than 5 MiB; >I found that even pages smaller than 5MB get rendered to PDF. I believe the text is supposed to mean "if it's not a web page but a file [offered for download], it will download the file if it's less than 5MB".
- Yetanfou 9y ago
  Some PDF readers can execute JS which can be embedded in PDF files. Just use a reader which doesn't support embedded JS (or switch it off if it does support it) to avoid this whole class of vulnerabilities.
boramalper 9y ago
> Are you using headless Chrome or similar to implement this? Yes, with pyppeteer[0] (Python 3 port of puppeteer). > I found that even pages smaller than 5MB get rendered to PDF. Web-pages (URLs that contain no file-extensions, or certain ones such as ".php", ".aspx" etc.) are rendered into PDFs. Generated PDFs must also be less than 5 MiB[1]. > It doesn't seem to support chunked transfer-encoding: True. I need to determine the size of a file before downloading it completely, and I used "content-length" header for that. I can see how it should also be possible for chunked transfer-encoding, but I'm not willing to implement it myself to be honest, and I doubt if it's supported by requests library. Thanks for the feedback! [0]: https://miyakogi.github.io/pyppeteer/ https://miyakogi.github.io/pyppeteer/ [1]: An arbitrary limit. Might change it in the future.
- jstanley 9y ago
  I'd be extremely surprised if chunked encoding is not supported by the standard python http requests libraries.
  - boramalper 9y ago
    I meant, a way to get the file-size when chunked transfer-encoding is used. I cannot imagine any, and I don't think requests has some magic way to calculate it (or more likely STOP after downloading n bytes). Maybe `stream=True` and `.read(5 * 1024 * 1024)` and check EOF, if not, abort.

heinrich5991 9y ago

>it also provides a way to read the content of potentially-malicious sites without risk of browser vulnerabilities being exploited PDFs have the same kind of vulnerabilities, they can even execute JS (JS is probably not contained in the ones generated by the author). >and without revealing your IP address, User-Agent, etc. Yep. >> try downloading it, if it is a file smaller than 5 MiB; >I found that even pages smaller than 5MB get rendered to PDF. I believe the text is supposed to mean "if it's not a web page but a file [offered for download], it will download the file if it's less than 5MB".

Yetanfou 9y ago
Some PDF readers can execute JS which can be embedded in PDF files. Just use a reader which doesn't support embedded JS (or switch it off if it does support it) to avoid this whole class of vulnerabilities.

boramalper 9y ago

> Are you using headless Chrome or similar to implement this? Yes, with pyppeteer[0] (Python 3 port of puppeteer). > I found that even pages smaller than 5MB get rendered to PDF. Web-pages (URLs that contain no file-extensions, or certain ones such as ".php", ".aspx" etc.) are rendered into PDFs. Generated PDFs must also be less than 5 MiB[1]. > It doesn't seem to support chunked transfer-encoding: True. I need to determine the size of a file before downloading it completely, and I used "content-length" header for that. I can see how it should also be possible for chunked transfer-encoding, but I'm not willing to implement it myself to be honest, and I doubt if it's supported by requests library. Thanks for the feedback! [0]: https://miyakogi.github.io/pyppeteer/ https://miyakogi.github.io/pyppeteer/ [1]: An arbitrary limit. Might change it in the future.

jstanley 9y ago
I'd be extremely surprised if chunked encoding is not supported by the standard python http requests libraries.
- boramalper 9y ago
  I meant, a way to get the file-size when chunked transfer-encoding is used. I cannot imagine any, and I don't think requests has some magic way to calculate it (or more likely STOP after downloading n bytes). Maybe `stream=True` and `.read(5 * 1024 * 1024)` and check EOF, if not, abort.