5 ms·
At a former job, we reverse engineered the trading APIs of most American retail stock brokerages (Fidelity, E-Trade, Robinhood, TD Ameritrade, etc). We did it b
by danielvaughn 2y ago
At a former job, we reverse engineered the trading APIs of most American retail stock brokerages (Fidelity, E-Trade, Robinhood, TD Ameritrade, etc). We did it by rooting an iPhone and using Charles Proxy to grab the unencrypted traffic.
I learned a lot from that experience, and it's also just plain fun to do. We did get some strongly worded letters from Robinhood though, lol. They tried blocking our servers but we just set up this automated system in Digital Ocean that would spin up a new droplet each time we detected a blockage, and they were never able to stop us after that.
Fun times.
- dangsux 2y ago[dead]
- packetlost 2y agoI did almost this exact same thing back in 2015~ ish when I was in high school over Christmas break. I reverse engineered the anime streaming site Crunchyroll's API via their Android and PS3 app using some HTTP proxy application and trial + error. I ended having a proper HLS-based streaming player and Android TV app back when their Android app was still Flash based. It was lots of fun!
- danielvaughn 2y ago2015/2016 was exactly when I was doing the above job. We could've hired you as an intern!
- packetlost 2y agoMaybe! I considered pursuing a job there at the time, but I opted to get a degree instead. Being located in the Midwest would've made it rather challenging anyways.
- packetlost 2y agoAlso this would have been winter of 2014 and into early 2015, so a bit before :) Sadly the CR forums are gone, so my rather popular thread that had feedback and support is long gone.
- IAmGraydon 2y agoYou were able to connect to those APIs without auth? As far as I know, they all require it.
- danielvaughn 2y agoNo we would use our own accounts that we sourced from either the CEO/CTO or someone else.
- IAmGraydon 2y agoWhy couldn't they block you then? They should have been able to quickly disable the accounts.
- danielvaughn 2y agoOur product was built for end users, so the traffic coming from our servers could technically be from any account. But as to why we weren't blocked during testing, that I'm not sure about. It's been about 8 years since I did that work - I assume we had someone's account who wasn't obviously connected to the company.
- wdb 2y agoSounds like a version of Plaid lol
- danielvaughn 2y agoYep it was very similar, just with more of a focus on retail trading.
- lesuorac 2y agoCan't one just list all of digital ocean's ip blocks? Like sure then you can add in hertzer or w/e and keep adjusting but idk if somebody keeps ban dodging by using the same provider it seems like you'd just try banning that provider early on?
- danielvaughn 2y agoYeah idk, they should've been able to but for some reason they didn't.
- strunz 2y agoProbably because they'd potentially be blocking "legitimate" users
- bigiain 2y agoSame sort of timeframe, a project I worked on used netwoking via mobile hotspots on a bunch of Android phones with SIMs from a provider that used CGNAT. If the target websites wanted to block that, they'd be blocking well over 10% of all mobile phones in Australia. (Hmmm, all the devices we used then would have just stopped working with the shutdown on the 3G network here. I wonder if it's all broken, or if they've upgraded all those devices to 4/5G ones?)
- dunham 2y agoQuite a few people do that. My grocery store's app and the Tesla app stop working if I bring up my vpn through DO. (I first set that up years ago because the legoland hotel's wifi was blocking reddit.)
- esskay 2y agoThe trick is to use the same provider. I did it with the Expedia api for a while. At the time they were using AWS so running it on Lambda made it very tricky for them to do much about it. They were hardly about to block all AWS IP's and risk their own services or any of the "real" partners having issues.
- ratedgene 2y ago
- TeMPOraL 2y ago> We did get some strongly worded letters from Robinhood though, lol. Unsurprisingly, the most sleazy players are the first ones to go after someone accessing their services in ways they didn't anticipate or intend :).
- 3abiton 2y agoThis reminds of an internship I did long time ago at a "insight generation company" that use to reverse engineer all do the flights/booking website for price forecasts. Getting paid for automating scraping/reverse engineering APIs was always fun.
- foxbarrington 2y agoI tried to use Charles on Robin Hood, but it looks like they use cert pinning to prevent it.
- danielvaughn 2y agoyeah I'm sure it's much harder today. I did this work like 8 or 9 years ago and I think fewer controls were in place at the time.
- mdaniel 2y agoIt reminds me of the expression "locks are to keep honest people out," in that code which runs on a device you control is code that you control: https://github.com/shroudedcode/apk-mitm#readme https://github.com/shroudedcode/apk-mitm#readme
- animuchan 2y ago[Frida](https://frida.re/ https://frida.re/) is fantastic for de-pinning certs in applications. Can be fiddly, but when it works, it just works™.
- getcrunk 2y agoWouldn’t they block/suspend the account/credentials?
- danielvaughn 2y agoThe product was for end users so the traffic was coming from any account. They never discovered the accounts we used for testing. At least, the American ones didn't. When we finished with US companies, we went for Singapore. Our CEO was friendly with an investor who had an account at some big Singaporean trading firm called Lim Tan. He gave us his account credentials and I began working. A few days later my boss comes over to my desk and says "stop whatever you're doing right now." Apparently my traffic had set off so many alarm bells that the CTO of Lim Tan was woken up at 2am. They permanently banned the investor, which I felt really bad about. What's crazy is that I wasn't even doing anything weird. I was just poking a bit at their authentication methods. That was when I learned that Singapore tech doesn't fuck around.
- theturtletalks 2y agoI wonder if using something like puppeteer or playwright to actually make the server think everything is being done client-side would still raise flags.
- pyrale 2y agoScraping a well-built API at human speed often isn't terribly useful, and once you start ramping up the scraping, it's account creation/patterns/use frequency that will set an alarm. Faking real user clients won't prevent these alarms.
- danielvaughn 2y agoThe purpose of our work wasn’t scraping - we were building a unified UI where our users could trade with any vendor of their choosing. Kinda like Plaid but specifically for retail stock trading. So the goal was to implement their trading API.
- jamesfinlayson 2y agoI remember a friend telling me years ago some scraping that happened where they worked - they scraped results from a bunch of different websites to create SEO websites and they had some setup using tor to avoid getting blocked. One of the websites that the company actually depended on apparently rendered results using a whole assortment of visually identical but structurally (HTML-wise) different methods which were returned randomly to hamper scrapers. They eventually gave that up because it turned out TV closed captions can be downloaded as XML and they had what the company needed.
- ratedgene 2y agoI've done the same for use in foreign currency exchanges. The adventure of reverse engineering protocols and finding security checks, etc was more fun than the actual accomplishment lol.
- DeathArrow 2y agoI wonder why didn't hey block all Digital Ocean IP ranges.They only need residential customers to access Robin Hood, so they can block everything else.
- danielvaughn 2y agoThere could be some legitimate proxy services set up on DO? Not sure, always wondered why.
- TrickyRick 2y agoIt's not exactly neither hard nor expensive to buy a residential IP proxy services to get around that.
- pyrale 2y agoOn the defender side, it's much funnier to poison the data of identified scrapers than to immediately ban them. Let them work out that their data has been altered for a while, clean up their datasets, and work to understand what identifies them as scrapers.
- TrickyRick 2y agoDefinitely, but it's also a lot more complex to present credible looking false data than to simply reject a request.
- biosboiii 2y agoBanning IP Ranges from datacenters is pretty easy and common, surprising they could not combat that.