5 ms·

Reverse-engineering the Kayak app with mitmproxy

73 points by shbhrsaha 12y ago

xrjn 12y ago
Great writeup, there's definitely potential to do something cool, especially if it's possible to get around any tracking and the following price manipulation. I tried installing your demo client, however on running it I got the following error: root@kayak:~/kayak-mobile-client# python client.py Departure airport code: LBG Destination airport code: HAM Departure date (MM/DD/YY): 12/26/14 Traceback (most recent call last): File "client.py", line 56, in <module> searchid = json.loads(r.text)["searchid"] KeyError: 'searchid'
- shbhrsaha 12y ago
  Appreciate your kind words. Have you set up your UUID and HASH according to the post? You'll actually need to run the app through mitmproxy yourself to get your unique credentials. Then, just drop those into the variables at the top of the script. I just added some error handling to the script to make this a bit more obvious!
ianlevesque 12y ago
A useful technique for sure. The only technique I know to slow this down is to use certificate pinning, but it's probably pointless. Some of your users are probably extremely motivated (like ours [1]) and it's obvious to them that what they are doing is unsupported. 1. http://difm.eu/dox/ http://difm.eu/dox/
- nnd 12y ago
  Certificate pinning would help to prevent sniffing traffic with tools like mitmproxy, but once you decompile the binary, you can disable certificate validation. I'm curious, in your case did you consider preventing using your API by third-party clients?
101914 12y ago
Note that mitmproxy, which requires a Python install, is not necessary to monitor what is being sent out from your computing device. The same results can be achieved using only socat and the openssl binary. While I understand the terminology is popular, I would not call this "reverse-engineering"; to me this is simply viewing your own traffic. I believe users have a right to see the traffic they (or the apps they use) are sending, and for security reasons alone they should monitor what is being sent. https plus third party CA usage complicates such transparency, making proxying techniques necessary. I wish more users would view their own traffic. Keep up the good work.
- userbinator 12y ago
  I've been MITM'ing my connections ever since I discovered the Proxomitron ( http://en.wikipedia.org/wiki/Proxomitron http://en.wikipedia.org/wiki/Proxomitron ) - it lets me customise webpages and filter out the crap before it gets to any browser, and works in all the browsers on the system; even the ones embedded in apps. It has a logging feature to show traffic too. It would be great if a proxy could be run directly on the mobile device, as then it could be used anywhere along with the apps its monitoring. As you said HTTPS is a bit of a pain since it's actively designed to resist such "attacks", but as long as there's the option to specify your own CAs it will work. Certificate "unpinning" is still a manual process, however...
  - dmix 12y ago
    How would you use a proxy on a mobile phone? Just curious. I'm not familiar with Proxomitron, so I assume adding filters like privoxy or captures? People have done this in ways by hacking the built-in VPN system I believe.
    - guiambros 12y ago
      You just point to your Proxy server under Settings / WiFi. Alternatively you may want to install your proxy SSL CA on your mobile device, so you can see inside HTTPS connections (if supported). Charles Proxy has some easy-to-use pointers [1][2]. [1] http://www.charlesproxy.com/documentation/faqs/ssl-connections-from-within-iphone-applications/ http://www.charlesproxy.com/documentation/faqs/ssl-connectio... [2] http://www.charlesproxy.com/documentation/faqs/using-charles-from-an-iphone/ http://www.charlesproxy.com/documentation/faqs/using-charles...
chrisan 12y ago
Another great mitm proxy is Charles Proxy http://www.charlesproxy.com/ http://www.charlesproxy.com/. It has a really nice UI for looking at JSON responses such as the Kayak. Sometimes a collapsible tree is invaluable in looking through a response. The easy filtering and formatting is primarily why I like it so much. Here is how it handles SSL for various ways http://www.charlesproxy.com/documentation/using-charles/ssl-certificates/ http://www.charlesproxy.com/documentation/using-charles/ssl-... Here is a screen shot of my iPhone Kayak app request for comparison http://imgur.com/gvKB6fr http://imgur.com/gvKB6fr
- mrbombastic 12y ago
  I love Charles Proxy, I use it all the time for debugging service issues. Breakpoints and the ability to easily mock responses is really invaluable.
- nnd 12y ago
  How is it better than mitmproxy? Also, it's not free/open-source.
  - brad0 12y ago
    In this case you get what you pay for. You get filtering. Local and remote mapping. SSL decryption. Request and response detail views. Breakpoints. The list goes on. Seriously, it's the best $50 I spent as part of my developer toolkit.
    - droope 12y ago
      If you are going to pay, I'd suggest Burp Suite.
    - scr4ve 12y ago
      Charles is definitely good, but to be fair, you get all of that (except mappings) in mitmproxy as well. :-)
vertak 12y ago
Does anyone know the possible legal repercussions open-sourcing a web service's API when the company doesn't explicitly grant permission? This is really neat, but could also raise the ire of a service that doesn't offer an API for a reason.
- declan 12y ago
  There are two obvious issues, as I see it. #1 is whether someone's liable for poking around inside the Kayak app in the way the author of the post did. #2 is whether it's legal to "open-source" the API by disclosing this information, once obtained. My quick take is that #1 likely violates Kayak's terms of use for the app, its web site, or both. The TOU for the website has a host of prohibitions, including don't "bypass or circumvent other measures employed to prevent or limit access to Our Website," no deep-linking (such as a link to the API endpoint), and as a catchall also prohibits all inappropriate-as-defined-by-Kayak activity. I haven't read the app TOU, but it would be standard practice to prohibit reverse engineering of the app itself. So we don't really need to reach point #2, which raises some 1A issues the first doesn't. (Note I'm excluding DMCA and CFAA issues because Kayak isn't our sue-happy friends at the MPAA or RIAA.) As a practical matter, though, the author of the linked post, Shubhro Saha, appears to be an undergrad, so is probably judgment-proof and not the most likely target of litigation.
  - moyix 12y ago
    Just to add to this: the relevant case law seems to be Bowers v. Baystate Technologies [1], which held that EULA clauses prohibiting reverse engineering are enforceable. I personally think this is a terrible decision, since it also prevents a lot of good uses of reverse engineering (see some examples, like fighting censorship and diagnosing software vulnerabilities, in a recent paper of mine [2]). [1] http://en.wikipedia.org/wiki/Bowers_v._Baystate_Technologies http://en.wikipedia.org/wiki/Bowers_v._Baystate_Technologies [2] https://mice.cs.columbia.edu/getTechreport.php?techreportID=1588&disposition=inline&format=pdf https://mice.cs.columbia.edu/getTechreport.php?techreportID=...
- nnd 12y ago
  I'm not a legal expert, but I bet there is a clause in TOS to prevent that. I wonder if anyone actually got sued for violating TOS this way though.
vertak 12y ago
Does anyone know the possible legal repercussions open-sourcing a web service's API when the company doesn't explicitly grant permission? This is really neat, but could also raise the ire of a service that doesn't offer an API for a reason.
dthakur 12y ago
> From that folder, get the mitmproxy-ca-cert.pem file onto your mobile device by emailing it to yourself, for example. Then follow certificate installation steps for iOS or Android. You can just go to http://mitm.it http://mitm.it on the device. It's a 'magic domain' for the proxied host. See http://mitm.it/doc/certinstall/webapp.html http://mitm.it/doc/certinstall/webapp.html
- nnd 12y ago
  I've tried this website on iPhone 6, and it redirects to the standard version of the website instead.
ketralnis 12y ago
I don't know about Kayak's economics, but at least at Hipmunk we pay our data providers per search and it's really quite expensive. If they aren't offering an API anymore, it's probably because it was too pricey to operate. You could easily cost a travel search company thousands of dollars very very quickly using an API they don't want you to use. I don't know if it's illegal or not, but it's certainly immoral.
- deleted 12y ago
  [deleted]
- uaygsfdbzf 12y ago
  How is using the API any different to using the website or app? Sounds like you need to move your access controls one layer down, into the API. If the app does rate limiting, the API should too/instead. Take the github approach and let open source devs write innovative apps based on your open API!
  - rdeboo 12y ago
    The difference is that Kayak is not able to show you ads if you use the API, so they generate less income from API usage compared to (web)app usage.
- edward 12y ago
  I'm happy to pay to use a flight search API. Kayak and Hipmunk could provide an API and pass on the cost to the API consumers.
  - nnd 12y ago
    Sabre has a free flight search API: https://developer.sabre.com/docs/read/REST_APIs https://developer.sabre.com/docs/read/REST_APIs
  - kremi 12y ago
    The problem with any paid API is that the API consumer needs to make money somehow so that he can offer the price of the service. How will the API consumer make money? I don't think Ads will generate enough revenue -- especially not on mobile. Kayak will get some revenue from the bookings resulting from the searches, and would need to circle back some of this revenue back to the API consumer. This is technically complicated because it requires a lot of tracking infrastructure. In the end, a service that would make sense is a lot more complicated than simply having an endpoint and saying it's X dollars per Y calls.
nnd 12y ago
I wouldn't call this reverse-engineering. How are UUID and HASH are generated? Are they unique to every installation?
javiercr 12y ago
Interesting! We've used a similar technique to reverse-engineering mobile apps from different banks in order to create a Ruby gem to fetch bank data (balance and transactions). https://github.com/ismaGNU/bankScrap https://github.com/ismaGNU/bankScrap

73 points by shbhrsaha 12y ago

xrjn 12y ago

Great writeup, there's definitely potential to do something cool, especially if it's possible to get around any tracking and the following price manipulation. I tried installing your demo client, however on running it I got the following error: root@kayak:~/kayak-mobile-client# python client.py Departure airport code: LBG Destination airport code: HAM Departure date (MM/DD/YY): 12/26/14 Traceback (most recent call last): File "client.py", line 56, in <module> searchid = json.loads(r.text)["searchid"] KeyError: 'searchid'

shbhrsaha 12y ago
Appreciate your kind words. Have you set up your UUID and HASH according to the post? You'll actually need to run the app through mitmproxy yourself to get your unique credentials. Then, just drop those into the variables at the top of the script. I just added some error handling to the script to make this a bit more obvious!

ianlevesque 12y ago

A useful technique for sure. The only technique I know to slow this down is to use certificate pinning, but it's probably pointless. Some of your users are probably extremely motivated (like ours [1]) and it's obvious to them that what they are doing is unsupported. 1. http://difm.eu/dox/ http://difm.eu/dox/

nnd 12y ago
Certificate pinning would help to prevent sniffing traffic with tools like mitmproxy, but once you decompile the binary, you can disable certificate validation. I'm curious, in your case did you consider preventing using your API by third-party clients?

101914 12y ago

Note that mitmproxy, which requires a Python install, is not necessary to monitor what is being sent out from your computing device. The same results can be achieved using only socat and the openssl binary. While I understand the terminology is popular, I would not call this "reverse-engineering"; to me this is simply viewing your own traffic. I believe users have a right to see the traffic they (or the apps they use) are sending, and for security reasons alone they should monitor what is being sent. https plus third party CA usage complicates such transparency, making proxying techniques necessary. I wish more users would view their own traffic. Keep up the good work.

userbinator 12y ago
I've been MITM'ing my connections ever since I discovered the Proxomitron ( http://en.wikipedia.org/wiki/Proxomitron http://en.wikipedia.org/wiki/Proxomitron ) - it lets me customise webpages and filter out the crap before it gets to any browser, and works in all the browsers on the system; even the ones embedded in apps. It has a logging feature to show traffic too. It would be great if a proxy could be run directly on the mobile device, as then it could be used anywhere along with the apps its monitoring. As you said HTTPS is a bit of a pain since it's actively designed to resist such "attacks", but as long as there's the option to specify your own CAs it will work. Certificate "unpinning" is still a manual process, however...
- dmix 12y ago
  How would you use a proxy on a mobile phone? Just curious. I'm not familiar with Proxomitron, so I assume adding filters like privoxy or captures? People have done this in ways by hacking the built-in VPN system I believe.
  - guiambros 12y ago
    You just point to your Proxy server under Settings / WiFi. Alternatively you may want to install your proxy SSL CA on your mobile device, so you can see inside HTTPS connections (if supported). Charles Proxy has some easy-to-use pointers [1][2]. [1] http://www.charlesproxy.com/documentation/faqs/ssl-connections-from-within-iphone-applications/ http://www.charlesproxy.com/documentation/faqs/ssl-connectio... [2] http://www.charlesproxy.com/documentation/faqs/using-charles-from-an-iphone/ http://www.charlesproxy.com/documentation/faqs/using-charles...

chrisan 12y ago

Another great mitm proxy is Charles Proxy http://www.charlesproxy.com/ http://www.charlesproxy.com/. It has a really nice UI for looking at JSON responses such as the Kayak. Sometimes a collapsible tree is invaluable in looking through a response. The easy filtering and formatting is primarily why I like it so much. Here is how it handles SSL for various ways http://www.charlesproxy.com/documentation/using-charles/ssl-certificates/ http://www.charlesproxy.com/documentation/using-charles/ssl-... Here is a screen shot of my iPhone Kayak app request for comparison http://imgur.com/gvKB6fr http://imgur.com/gvKB6fr

mrbombastic 12y ago
I love Charles Proxy, I use it all the time for debugging service issues. Breakpoints and the ability to easily mock responses is really invaluable.
nnd 12y ago
How is it better than mitmproxy? Also, it's not free/open-source.
- brad0 12y ago
  In this case you get what you pay for. You get filtering. Local and remote mapping. SSL decryption. Request and response detail views. Breakpoints. The list goes on. Seriously, it's the best $50 I spent as part of my developer toolkit.
  - droope 12y ago
    If you are going to pay, I'd suggest Burp Suite.
  - scr4ve 12y ago
    Charles is definitely good, but to be fair, you get all of that (except mappings) in mitmproxy as well. :-)

vertak 12y ago

Does anyone know the possible legal repercussions open-sourcing a web service's API when the company doesn't explicitly grant permission? This is really neat, but could also raise the ire of a service that doesn't offer an API for a reason.

declan 12y ago
There are two obvious issues, as I see it. #1 is whether someone's liable for poking around inside the Kayak app in the way the author of the post did. #2 is whether it's legal to "open-source" the API by disclosing this information, once obtained. My quick take is that #1 likely violates Kayak's terms of use for the app, its web site, or both. The TOU for the website has a host of prohibitions, including don't "bypass or circumvent other measures employed to prevent or limit access to Our Website," no deep-linking (such as a link to the API endpoint), and as a catchall also prohibits all inappropriate-as-defined-by-Kayak activity. I haven't read the app TOU, but it would be standard practice to prohibit reverse engineering of the app itself. So we don't really need to reach point #2, which raises some 1A issues the first doesn't. (Note I'm excluding DMCA and CFAA issues because Kayak isn't our sue-happy friends at the MPAA or RIAA.) As a practical matter, though, the author of the linked post, Shubhro Saha, appears to be an undergrad, so is probably judgment-proof and not the most likely target of litigation.
- moyix 12y ago
  Just to add to this: the relevant case law seems to be Bowers v. Baystate Technologies [1], which held that EULA clauses prohibiting reverse engineering are enforceable. I personally think this is a terrible decision, since it also prevents a lot of good uses of reverse engineering (see some examples, like fighting censorship and diagnosing software vulnerabilities, in a recent paper of mine [2]). [1] http://en.wikipedia.org/wiki/Bowers_v._Baystate_Technologies http://en.wikipedia.org/wiki/Bowers_v._Baystate_Technologies [2] https://mice.cs.columbia.edu/getTechreport.php?techreportID=1588&disposition=inline&format=pdf https://mice.cs.columbia.edu/getTechreport.php?techreportID=...
nnd 12y ago
I'm not a legal expert, but I bet there is a clause in TOS to prevent that. I wonder if anyone actually got sued for violating TOS this way though.

vertak 12y ago

dthakur 12y ago

> From that folder, get the mitmproxy-ca-cert.pem file onto your mobile device by emailing it to yourself, for example. Then follow certificate installation steps for iOS or Android. You can just go to http://mitm.it http://mitm.it on the device. It's a 'magic domain' for the proxied host. See http://mitm.it/doc/certinstall/webapp.html http://mitm.it/doc/certinstall/webapp.html

nnd 12y ago
I've tried this website on iPhone 6, and it redirects to the standard version of the website instead.

ketralnis 12y ago

I don't know about Kayak's economics, but at least at Hipmunk we pay our data providers per search and it's really quite expensive. If they aren't offering an API anymore, it's probably because it was too pricey to operate. You could easily cost a travel search company thousands of dollars very very quickly using an API they don't want you to use. I don't know if it's illegal or not, but it's certainly immoral.

deleted 12y ago
[deleted]
uaygsfdbzf 12y ago
How is using the API any different to using the website or app? Sounds like you need to move your access controls one layer down, into the API. If the app does rate limiting, the API should too/instead. Take the github approach and let open source devs write innovative apps based on your open API!
- rdeboo 12y ago
  The difference is that Kayak is not able to show you ads if you use the API, so they generate less income from API usage compared to (web)app usage.
edward 12y ago
I'm happy to pay to use a flight search API. Kayak and Hipmunk could provide an API and pass on the cost to the API consumers.
- nnd 12y ago
  Sabre has a free flight search API: https://developer.sabre.com/docs/read/REST_APIs https://developer.sabre.com/docs/read/REST_APIs
- kremi 12y ago
  The problem with any paid API is that the API consumer needs to make money somehow so that he can offer the price of the service. How will the API consumer make money? I don't think Ads will generate enough revenue -- especially not on mobile. Kayak will get some revenue from the bookings resulting from the searches, and would need to circle back some of this revenue back to the API consumer. This is technically complicated because it requires a lot of tracking infrastructure. In the end, a service that would make sense is a lot more complicated than simply having an endpoint and saying it's X dollars per Y calls.

nnd 12y ago

I wouldn't call this reverse-engineering. How are UUID and HASH are generated? Are they unique to every installation?

javiercr 12y ago

Interesting! We've used a similar technique to reverse-engineering mobile apps from different banks in order to create a Ruby gem to fetch bank data (balance and transactions). https://github.com/ismaGNU/bankScrap https://github.com/ismaGNU/bankScrap