9 ms·

Quick and dirty way to rip an eBook from Android

192 points by Amorymeltzer 5y ago

account-5 5y ago
I like this, it seems like the sort of thing I might do, except I would probably spend way too long extracting the actual text and attempting to produce a "better" pdf. I would probably fail, and end up back with what this article produced.
deleted 5y ago
[deleted]
rahimnathwani 5y ago
I wonder if this works for apps that have set WindowManager.LayoutParams.FLAG_SECURE
- Farbklex 5y ago
  You can still use the tool scrcpy to stream a video of the screen via USB to a PC.
  - edent 5y ago
    Huh! I just tried this and you are right. I would have thought scrcpy wouldn't be able to see those screens. Thanks!
    - iggldiggl 5y ago
      Only works up to and including Android 11, though.
  - VMtest 5y ago
    Yea, I have even tried to capture a screenshot on a computer with scrcpy before that was not possible in the app that restricts screenshot
- jeroenhd 5y ago
  Not without some kind of root access. But you can usually find an app that doesn't have that flag and can still decode DRM.
  - goldcd 5y ago
    I just pop my phone on the photo-copier. Damp postit on the ADF, auto-scrolls the screen on each cycle.
    - alufers 5y ago
      You are joking right? If not, I'd kill for a video of this setup working.
      - goldcd 5y ago
        Sadly I am. Although I'm idly just considering putting a magnet on the scan-head of a flatbed scanner. Could then just use tasker to detect the passing of the magnet as a 'page-down' scroll on the screen.
        arnoooooo 5y ago
        the "adb shell input" command as used in the linked article would be simpler.
  - alufers 5y ago
    You can either use root access or disassemble the app, remove the flag, and then assemble it again. Impossible to do if the app uses SafetyNet though.
- Gigachad 5y ago
  That setting is essentially malware and it’s a huge shame that android added it.
  - wiseowise 5y ago
    It's really not.
    - Gigachad 5y ago
      Why do you think that? I made the claim because it’s a feature that acts against the user. The user wants to take a screenshot and there is never a good reason to stop them from doing that on their own device.
  - tjoff 5y ago
    I'm flabbergasted that there at least isn't a developer-option to disable it.
    - josephcsible 5y ago
      You can use things like https://github.com/veeti/DisableFlagSecure https://github.com/veeti/DisableFlagSecure to disable it if you're willing to unlock your bootloader and install Xposed. But yes, it's incredibly user-hostile of Google to provide this without giving the user a built-in way to override it.
      - tommek4077 5y ago
        The point is, for every stupid DRM "solution" you come up with, some technical capable guy will find a way to crack this. Happens since more then 40 years in computer sience.
        Gigachad 5y ago
        Unfortunately DRM is winning. It's failed before because the user could always drop down to a lower level and gain access, but now DRM is baked in to the hardware and OS of the devices people have. Those tricks to disable the secure flag feature do not work if the app uses safety net which can detect a rooted device and is backed up by boot chain attestation and hardware security chips. Sure, for each part there is usually some exploit known, but the whole thing is getting much much harder to the point where if the average user said "How do I screenshot this app" the only real answer is to take a photo of the screen.
- Too 5y ago
  I always thought that flag was to protect the users sensitive information, like banking logins. Seems I was wrong and it’s mostly used for DRM purposes.
  - rahimnathwani 5y ago
    I've only encountered it with banking apps, so perhaps you were right.
    - stonewareslord 5y ago
      Yeah it's a shame. Android allows for a lot of cool apps that have access to control or record other apps. I assume FLAG_SECURE was added for banking apps and the like (it's on for my banking apps), so it's a great addition. If Android really cared about its users, all it would have to do is permit user-prompted screenshots and screen recordings but still block other app access. I can't think of a time where I'd want my own phone preventing me from screenshotting, but I can see the use case for general screen access prevention. I don't personally know a root-obly way to disable this, but xposed has the disable_flag_secure module that I'm weary of using because it removes the flag everywhere.
userbinator 5y ago
I suppose this could be called the "semi-analog hole". I'd probably analyse the app first because these protections are usually quite trivial. I remember one a long time ago (PC-based), which was basically a PDF "encrypted" using a short XOR key and appended to the end of the executable.
yosito 5y ago
Does anyone have any advice on how I can pay authors to get a copy of their eBook without going through Amazon or Google? I really don't want Big Tech or advertising companies profiling me based on the books I read, but I absolutely want to pay authors for their work. I've considered torrenting books and just sending a check or something to authors, but I'd love to find a better way.
- juglans 5y ago
  I recently came across a site called Leanpub [https://leanpub.com/ https://leanpub.com/] while searching for Jeff Geerling's ebook on 'Ansible for Devops'. The company looks to be ethical, and is aimed at Technical authors who wish to self publish ebooks. Each ebook sold displays how much the author will receive from the purchase (usually 80%). Most Leanpub books are available in PDF, EPUB, and MOBI. If you buy a Leanpub book, you get free updates for as long as the author updates the book [which Jeff Geerling seems to regularly do] The big plus is that Leanpub books don't have any DRM copy-protection.
- edent 5y ago
  There are a few of methods I use. First, my local library lends eBooks. In the UK, authors get paid for each book that is borrowed. Secondly, sites like hive.co.uk sell eBooks outside of the Amazon / Google services. You can read them on any device which supports Adobe's DRM - or you can remove the DRM yourself. Finally, go to the publisher's website. Some of them sell books directly.
  - steelframe 5y ago
    > First, my local library lends eBooks. In the UK, authors get paid for each book that is borrowed. I'm in the US, and I wonder how it's done here. I also get all of my eBooks and audiobooks from my local library. Just yesterday I was perusing titles at Barnes & Noble and putting stuff that looked interesting on my library queue. I ended up actually buying dead-tree manga books from there, since I really can't get into eBook-formatted manga for some reason. If it's illustration-heavy, I still want printed stuff. For the audiobooks I don't even have to strip DRM. The ODM files provide links where you can pull down unprotected MP3 files. For the eBooks I keep a Windows VM just for running an older version of Kindle for PC, Calibre with DeDRM, and Adobe Digital Editions. I know there are websites from where I can just directly download the DRM-stripped eBooks, but I figure that going through the library means that somehow some kind of revenue is making it to the authors. I'd be more than happy to tip the authors directly, but I'd imagine that publishers forbid that sort of thing contractually.
fouc 5y ago
>The text is "sandwiched" behind the image of the page, so you can't see it but can search for it. I already have a bunch of these PDFs that have this. Does anyone know how to extract this text out? In particular I'd like to make it into a nice mobi/epub file, but calibre doesn't do this on its own.
- edent 5y ago
  At a basic level, you can copy and paste it out. CTRL+A, CTRL+C, CTRL+V. Done. You can also use Poppler's PDFtoText https://en.wikipedia.org/wiki/Pdftotext https://en.wikipedia.org/wiki/Pdftotext Calibre struggles with some PDFs, especially those that don't contain metadata about chapter headings etc. That's a failing on the PDF, not Calibre.
- deleted 5y ago
  [deleted]
- lelandfe 5y ago
  As always, Ghostscript is the way. $ gs sDEVICE=txtwrite -o output.txt input.pdf
  - sandreas 5y ago
    Awesome, I did not know that this works :-) Thank you.
    - stonewareslord 5y ago
      Additionally, some PDFs (like textbooks) insert intentionally mangled embedded text to prevent copying/pasting. You can rasterize them and push them through pdfsandwich/ocrmypdf to wipe these hidden layers and re-ocr them. Haven't done it in a while, but looks like this should work: https://askubuntu.com/a/845119 https://askubuntu.com/a/845119 Obviously replace convert with mogrify ./* after backing them up
      - lelandfe 5y ago
qiqitori 5y ago
I once did something similar with a Windows ebook reader that nicely let you copy up to 10 characters to the clipboard. I wrote an Autohotkey script that would mouse down->move the mouse->mouse up->copy to a variable, and then start again. Lost the source code but never came across that format again. Still have a video of the program running: https://youtu.be/umou8uuNIPU https://youtu.be/umou8uuNIPU
- account-5 5y ago
  Nice use of autohotkey, it doesn't get enough love. I learned to program using autohotkey to automate parts of my job.
quanto 5y ago
The workhorse behind this script is pdfsandwich http://www.tobias-elze.de/pdfsandwich/ http://www.tobias-elze.de/pdfsandwich/ which itself uses the OCR engine Tesseract. Unfortunately, Tesseract is not quite accurate even for nicely aligned, printed computer texts. When I was working on an OCR problem within my firm, I was surprised to find that it was not a solved problem, and there are still public and for-profit efforts to improve the state of the art.
- smegger001 5y ago
  I would imagine it could be a "Hard Problem". I mean look at all of the problem anyone has ever had with Unicode now imagine trying to reverse engineer it form images of characters in random font without the bit-stream of the charecter, the benefit of textual context (ie it that a Latin script letter O an zero or O shaped letter from another character set used in a quote name or excerpt from another language), or metadata in file header. I think this will probably be one of those thing that will get better asymptotically until we have sufficiently advanced general intelligence, as alot of it will be context dependent.
  - Too 5y ago
    Could an ocr reader do a first pass on the easy characters to guess the font, eliminating the problem of random font for the second pass of parsing more difficult characters.
musicale 5y ago
> DRM on textbooks is an annoyance. For computer science books, it's little more than a fig-leaf. The idea that you have to jump through hoops to format shift an ebook (or other item in digital form) that you paid for extremely annoying. Apple's iTunes music store tracks went DRM-free with iTunes Plus in 2007. Amazon also sells DRM-free music. Why are iBooks and Kindle so late to the party?
kaetemi 5y ago
Back when Google Books still showed entire pages for every search result, you could just script a search for each page number, and rip an entire book that way.

192 points by Amorymeltzer 5y ago

I like this, it seems like the sort of thing I might do, except I would probably spend way too long extracting the actual text and attempting to produce a "better" pdf. I would probably fail, and end up back with what this article produced.

deleted 5y ago

[deleted]

rahimnathwani 5y ago

I wonder if this works for apps that have set WindowManager.LayoutParams.FLAG_SECURE

Farbklex 5y ago
You can still use the tool scrcpy to stream a video of the screen via USB to a PC.
- edent 5y ago
  Huh! I just tried this and you are right. I would have thought scrcpy wouldn't be able to see those screens. Thanks!
  - iggldiggl 5y ago
    Only works up to and including Android 11, though.
- VMtest 5y ago
  Yea, I have even tried to capture a screenshot on a computer with scrcpy before that was not possible in the app that restricts screenshot
jeroenhd 5y ago
Not without some kind of root access. But you can usually find an app that doesn't have that flag and can still decode DRM.
- goldcd 5y ago
  I just pop my phone on the photo-copier. Damp postit on the ADF, auto-scrolls the screen on each cycle.
  - alufers 5y ago
    You are joking right? If not, I'd kill for a video of this setup working.
    - goldcd 5y ago
      Sadly I am. Although I'm idly just considering putting a magnet on the scan-head of a flatbed scanner. Could then just use tasker to detect the passing of the magnet as a 'page-down' scroll on the screen.
      - arnoooooo 5y ago
        the "adb shell input" command as used in the linked article would be simpler.
- alufers 5y ago
  You can either use root access or disassemble the app, remove the flag, and then assemble it again. Impossible to do if the app uses SafetyNet though.
Gigachad 5y ago
That setting is essentially malware and it’s a huge shame that android added it.
- wiseowise 5y ago
  It's really not.
  - Gigachad 5y ago
    Why do you think that? I made the claim because it’s a feature that acts against the user. The user wants to take a screenshot and there is never a good reason to stop them from doing that on their own device.
- tjoff 5y ago
  I'm flabbergasted that there at least isn't a developer-option to disable it.
  - josephcsible 5y ago
    You can use things like https://github.com/veeti/DisableFlagSecure https://github.com/veeti/DisableFlagSecure to disable it if you're willing to unlock your bootloader and install Xposed. But yes, it's incredibly user-hostile of Google to provide this without giving the user a built-in way to override it.
    - tommek4077 5y ago
      The point is, for every stupid DRM "solution" you come up with, some technical capable guy will find a way to crack this. Happens since more then 40 years in computer sience.
      - Gigachad 5y ago
        Unfortunately DRM is winning. It's failed before because the user could always drop down to a lower level and gain access, but now DRM is baked in to the hardware and OS of the devices people have. Those tricks to disable the secure flag feature do not work if the app uses safety net which can detect a rooted device and is backed up by boot chain attestation and hardware security chips. Sure, for each part there is usually some exploit known, but the whole thing is getting much much harder to the point where if the average user said "How do I screenshot this app" the only real answer is to take a photo of the screen.
Too 5y ago
I always thought that flag was to protect the users sensitive information, like banking logins. Seems I was wrong and it’s mostly used for DRM purposes.
- rahimnathwani 5y ago
  I've only encountered it with banking apps, so perhaps you were right.
  - stonewareslord 5y ago
    Yeah it's a shame. Android allows for a lot of cool apps that have access to control or record other apps. I assume FLAG_SECURE was added for banking apps and the like (it's on for my banking apps), so it's a great addition. If Android really cared about its users, all it would have to do is permit user-prompted screenshots and screen recordings but still block other app access. I can't think of a time where I'd want my own phone preventing me from screenshotting, but I can see the use case for general screen access prevention. I don't personally know a root-obly way to disable this, but xposed has the disable_flag_secure module that I'm weary of using because it removes the flag everywhere.

userbinator 5y ago

I suppose this could be called the "semi-analog hole". I'd probably analyse the app first because these protections are usually quite trivial. I remember one a long time ago (PC-based), which was basically a PDF "encrypted" using a short XOR key and appended to the end of the executable.

yosito 5y ago

Does anyone have any advice on how I can pay authors to get a copy of their eBook without going through Amazon or Google? I really don't want Big Tech or advertising companies profiling me based on the books I read, but I absolutely want to pay authors for their work. I've considered torrenting books and just sending a check or something to authors, but I'd love to find a better way.

juglans 5y ago
I recently came across a site called Leanpub [https://leanpub.com/ https://leanpub.com/] while searching for Jeff Geerling's ebook on 'Ansible for Devops'. The company looks to be ethical, and is aimed at Technical authors who wish to self publish ebooks. Each ebook sold displays how much the author will receive from the purchase (usually 80%). Most Leanpub books are available in PDF, EPUB, and MOBI. If you buy a Leanpub book, you get free updates for as long as the author updates the book [which Jeff Geerling seems to regularly do] The big plus is that Leanpub books don't have any DRM copy-protection.
edent 5y ago
There are a few of methods I use. First, my local library lends eBooks. In the UK, authors get paid for each book that is borrowed. Secondly, sites like hive.co.uk sell eBooks outside of the Amazon / Google services. You can read them on any device which supports Adobe's DRM - or you can remove the DRM yourself. Finally, go to the publisher's website. Some of them sell books directly.
- steelframe 5y ago
  > First, my local library lends eBooks. In the UK, authors get paid for each book that is borrowed. I'm in the US, and I wonder how it's done here. I also get all of my eBooks and audiobooks from my local library. Just yesterday I was perusing titles at Barnes & Noble and putting stuff that looked interesting on my library queue. I ended up actually buying dead-tree manga books from there, since I really can't get into eBook-formatted manga for some reason. If it's illustration-heavy, I still want printed stuff. For the audiobooks I don't even have to strip DRM. The ODM files provide links where you can pull down unprotected MP3 files. For the eBooks I keep a Windows VM just for running an older version of Kindle for PC, Calibre with DeDRM, and Adobe Digital Editions. I know there are websites from where I can just directly download the DRM-stripped eBooks, but I figure that going through the library means that somehow some kind of revenue is making it to the authors. I'd be more than happy to tip the authors directly, but I'd imagine that publishers forbid that sort of thing contractually.

fouc 5y ago

>The text is "sandwiched" behind the image of the page, so you can't see it but can search for it. I already have a bunch of these PDFs that have this. Does anyone know how to extract this text out? In particular I'd like to make it into a nice mobi/epub file, but calibre doesn't do this on its own.

edent 5y ago
At a basic level, you can copy and paste it out. CTRL+A, CTRL+C, CTRL+V. Done. You can also use Poppler's PDFtoText https://en.wikipedia.org/wiki/Pdftotext https://en.wikipedia.org/wiki/Pdftotext Calibre struggles with some PDFs, especially those that don't contain metadata about chapter headings etc. That's a failing on the PDF, not Calibre.
deleted 5y ago
[deleted]
lelandfe 5y ago
As always, Ghostscript is the way. $ gs sDEVICE=txtwrite -o output.txt input.pdf
- sandreas 5y ago
  Awesome, I did not know that this works :-) Thank you.
  - stonewareslord 5y ago
    Additionally, some PDFs (like textbooks) insert intentionally mangled embedded text to prevent copying/pasting. You can rasterize them and push them through pdfsandwich/ocrmypdf to wipe these hidden layers and re-ocr them. Haven't done it in a while, but looks like this should work: https://askubuntu.com/a/845119 https://askubuntu.com/a/845119 Obviously replace convert with mogrify ./* after backing them up
    - lelandfe 5y ago

qiqitori 5y ago

I once did something similar with a Windows ebook reader that nicely let you copy up to 10 characters to the clipboard. I wrote an Autohotkey script that would mouse down->move the mouse->mouse up->copy to a variable, and then start again. Lost the source code but never came across that format again. Still have a video of the program running: https://youtu.be/umou8uuNIPU https://youtu.be/umou8uuNIPU

account-5 5y ago
Nice use of autohotkey, it doesn't get enough love. I learned to program using autohotkey to automate parts of my job.

quanto 5y ago

The workhorse behind this script is pdfsandwich http://www.tobias-elze.de/pdfsandwich/ http://www.tobias-elze.de/pdfsandwich/ which itself uses the OCR engine Tesseract. Unfortunately, Tesseract is not quite accurate even for nicely aligned, printed computer texts. When I was working on an OCR problem within my firm, I was surprised to find that it was not a solved problem, and there are still public and for-profit efforts to improve the state of the art.

smegger001 5y ago
I would imagine it could be a "Hard Problem". I mean look at all of the problem anyone has ever had with Unicode now imagine trying to reverse engineer it form images of characters in random font without the bit-stream of the charecter, the benefit of textual context (ie it that a Latin script letter O an zero or O shaped letter from another character set used in a quote name or excerpt from another language), or metadata in file header. I think this will probably be one of those thing that will get better asymptotically until we have sufficiently advanced general intelligence, as alot of it will be context dependent.
- Too 5y ago
  Could an ocr reader do a first pass on the easy characters to guess the font, eliminating the problem of random font for the second pass of parsing more difficult characters.

musicale 5y ago

> DRM on textbooks is an annoyance. For computer science books, it's little more than a fig-leaf. The idea that you have to jump through hoops to format shift an ebook (or other item in digital form) that you paid for extremely annoying. Apple's iTunes music store tracks went DRM-free with iTunes Plus in 2007. Amazon also sells DRM-free music. Why are iBooks and Kindle so late to the party?

kaetemi 5y ago

Back when Google Books still showed entire pages for every search result, you could just script a search for each page number, and rip an entire book that way.