6 ms·

Scraping data from an app: real world example

117 points by DantesTravel 4y ago

nothasan 4y ago
I have worked with the site mentioned in the article’s API previously. I am not sure why they used the overhead of a "scraping framework" when it was just JSON they needed to parse.
- matheusmoreira 4y ago
  Perhaps they're used to the Scrapy APIs and tools? I agree that an HTTP requests library would have been sufficient but maybe it was just easier for them to use the framework they're used to.
IMTDb 4y ago
Most app have some form of SSL pinning system in place which means that you have to perform additional work to allow the proxy to decrypt the HTTPS traffic.
- 1vuio0pswjnm7 4y ago
  "Most app have some form of SSL pinning system in place..." I would like to see the data, if any, supporting this statement. I would expect some apps would use pinning, but most would not. Google recommends against it. https://developer.android.com/training/articles/security-ssl#Pinning https://developer.android.com/training/articles/security-ssl...
  - franga2000 4y ago
    I reverse engineer Android apps for work and pinning is present in all but the lowest effort apps I encounter.
  - forgotmypw17 4y ago
    >Caution: Certificate Pinning is not recommended for Android applications due to the high risk of future server configuration changes, such as changing to another Certificate Authority, rendering the application unable to connect to the server without receiving a client software update. This actually applies to websites and browsers as well.
    - metadat 4y ago
      Why isn't there a site-controlled fallback setting for this? Does this not make sense? Abu given website's beet interest is to continue to be reachable.
      - tremon 4y ago
        Every escape hatch in the certificate validation is also an additional avenue for attack. For example, using a DNS record to override certificate pins makes DNS cache poisoning much more valuable to the attacker.
        forgotmypw17 4y ago
        Every layer of security is also an additional accessibility hurdle.
        metadat 4y ago
        Got it, thanks @tremon.
  - oefrha 4y ago
    I would say based on personal observation that the more scrape-worthy an app is, the more likely it has cert pinning. Rather obvious if you think about it, really. High value targets especially from big shops tend to have other measures like complex MACs that make scraping hell. I’m sure most largely-worthless-to-scrape apps don’t employ cert pinning.
    - mmsnberbar66 4y ago
      > other measures like complex MACs that make scraping hell. Do you have examples of these techniques?
      - oefrha 4y ago
        Recent example I encountered: TikTok web API has dynamically generated parameters X-Bogus, msToken and _signature (could be slightly wrong, it’s been a while) that are verified server-side. I haven’t reversed their mobile app so not sure if they also employ MACs there, but I’ve seen these from other apps in the past. And it’s harder when employed in an app; on the web you’ll be reversing (obfuscated) JavaScript in a readily available debugger, whereas for an app you’ll likely be reversing from disassembly.
  - txtsd 4y ago
    Most games I've tried to examine have had cert pinning enabled.
- ratg13 4y ago
  Can someone suggest some resources to understand the additional work needed to decrypt the pinned https traffic?
  - IMTDb 4y ago
    Android: https://httptoolkit.tech/blog/frida-certificate-pinning/ https://httptoolkit.tech/blog/frida-certificate-pinning/ iOS: https://nabla-c0d3.github.io/blog/2013/08/20/intercepting-the-app-stores-traffic-on-ios/ https://nabla-c0d3.github.io/blog/2013/08/20/intercepting-th...
    - matheusmoreira 4y ago
      This is really helpful, thanks!
matheusmoreira 4y ago
This is a great method. It's essentially creating a custom client for their server. Fiddler makes interception of encrypted traffic from apps a lot easier than the last time I tried this. Really nice.
anyfactor 4y ago
That is really nice. The last time I attempted scraping an app was using an android emulator (bluestacks), then using maybe Wireshark or Charles for getting the API endpoint. It didn't work for some reason though. I don't remember the exact error and I am kinda skeptical about app scraping being this easy.
- rOOb85 4y ago
  I recently did something similar with good results (I found the api endpoints I was interested in) using the official Android emulator and https://github.com/mitmproxy/mitmproxy https://github.com/mitmproxy/mitmproxy I did have to jump through some hoops with the emulator and pushing my own ssl cert to it's RO system partition. But it was a few commands and easy enough.
  - anyfactor 4y ago
    Thank you. I am going to try out your solution :) appreciate it.
shafin_ 4y ago
Previously i had some success with this https://httptoolkit.tech/ https://httptoolkit.tech/ and running the app on android emulator
hirebackenddev 4y ago
jesterson 4y ago
I use Burp for same purposes. Very convenient and solves the problem of MITM certificates.
wantlotsofcurry 4y ago
There's also PCAPdroid [1] which you can run straight from your phone with no root. Works with https traffic too when you enable the mitm setting. [1]: https://github.com/emanuele-f/PCAPdroid https://github.com/emanuele-f/PCAPdroid

117 points by DantesTravel 4y ago

I have worked with the site mentioned in the article’s API previously. I am not sure why they used the overhead of a "scraping framework" when it was just JSON they needed to parse.

matheusmoreira 4y ago
Perhaps they're used to the Scrapy APIs and tools? I agree that an HTTP requests library would have been sufficient but maybe it was just easier for them to use the framework they're used to.

IMTDb 4y ago

Most app have some form of SSL pinning system in place which means that you have to perform additional work to allow the proxy to decrypt the HTTPS traffic.

1vuio0pswjnm7 4y ago
"Most app have some form of SSL pinning system in place..." I would like to see the data, if any, supporting this statement. I would expect some apps would use pinning, but most would not. Google recommends against it. https://developer.android.com/training/articles/security-ssl#Pinning https://developer.android.com/training/articles/security-ssl...
- franga2000 4y ago
  I reverse engineer Android apps for work and pinning is present in all but the lowest effort apps I encounter.
- forgotmypw17 4y ago
  >Caution: Certificate Pinning is not recommended for Android applications due to the high risk of future server configuration changes, such as changing to another Certificate Authority, rendering the application unable to connect to the server without receiving a client software update. This actually applies to websites and browsers as well.
  - metadat 4y ago
    Why isn't there a site-controlled fallback setting for this? Does this not make sense? Abu given website's beet interest is to continue to be reachable.
    - tremon 4y ago
      Every escape hatch in the certificate validation is also an additional avenue for attack. For example, using a DNS record to override certificate pins makes DNS cache poisoning much more valuable to the attacker.
      - forgotmypw17 4y ago
        Every layer of security is also an additional accessibility hurdle.
      - metadat 4y ago
        Got it, thanks @tremon.
- oefrha 4y ago
  I would say based on personal observation that the more scrape-worthy an app is, the more likely it has cert pinning. Rather obvious if you think about it, really. High value targets especially from big shops tend to have other measures like complex MACs that make scraping hell. I’m sure most largely-worthless-to-scrape apps don’t employ cert pinning.
  - mmsnberbar66 4y ago
    > other measures like complex MACs that make scraping hell. Do you have examples of these techniques?
    - oefrha 4y ago
      Recent example I encountered: TikTok web API has dynamically generated parameters X-Bogus, msToken and _signature (could be slightly wrong, it’s been a while) that are verified server-side. I haven’t reversed their mobile app so not sure if they also employ MACs there, but I’ve seen these from other apps in the past. And it’s harder when employed in an app; on the web you’ll be reversing (obfuscated) JavaScript in a readily available debugger, whereas for an app you’ll likely be reversing from disassembly.
- txtsd 4y ago
  Most games I've tried to examine have had cert pinning enabled.
ratg13 4y ago
Can someone suggest some resources to understand the additional work needed to decrypt the pinned https traffic?
- IMTDb 4y ago
  Android: https://httptoolkit.tech/blog/frida-certificate-pinning/ https://httptoolkit.tech/blog/frida-certificate-pinning/ iOS: https://nabla-c0d3.github.io/blog/2013/08/20/intercepting-the-app-stores-traffic-on-ios/ https://nabla-c0d3.github.io/blog/2013/08/20/intercepting-th...
  - matheusmoreira 4y ago
    This is really helpful, thanks!

matheusmoreira 4y ago

This is a great method. It's essentially creating a custom client for their server. Fiddler makes interception of encrypted traffic from apps a lot easier than the last time I tried this. Really nice.

anyfactor 4y ago

That is really nice. The last time I attempted scraping an app was using an android emulator (bluestacks), then using maybe Wireshark or Charles for getting the API endpoint. It didn't work for some reason though. I don't remember the exact error and I am kinda skeptical about app scraping being this easy.

rOOb85 4y ago
I recently did something similar with good results (I found the api endpoints I was interested in) using the official Android emulator and https://github.com/mitmproxy/mitmproxy https://github.com/mitmproxy/mitmproxy I did have to jump through some hoops with the emulator and pushing my own ssl cert to it's RO system partition. But it was a few commands and easy enough.
- anyfactor 4y ago
  Thank you. I am going to try out your solution :) appreciate it.

shafin_ 4y ago

Previously i had some success with this https://httptoolkit.tech/ https://httptoolkit.tech/ and running the app on android emulator

hirebackenddev 4y ago

jesterson 4y ago

I use Burp for same purposes. Very convenient and solves the problem of MITM certificates.

wantlotsofcurry 4y ago

There's also PCAPdroid [1] which you can run straight from your phone with no root. Works with https traffic too when you enable the mitm setting. [1]: https://github.com/emanuele-f/PCAPdroid https://github.com/emanuele-f/PCAPdroid