3 ms·

Is there any brokerage who’s terms of service or fraud liability protections aren’t entirely waived by using a service like this? Do end users provide the crede

by koolba 5y ago

Is there any brokerage who’s terms of service or fraud liability protections aren’t entirely waived by using a service like this? Do end users provide the credentials for their brokerage accounts? I can’t imagine any infosec officer or general counsel at a brokerage agreeing to screen scraped data collection. They already get enough legal threats for bad pricing / bad execution, this would be an entirely new can of worms.

timothygoltser 5y ago
We do sanctioned integrations with brokerages wherever OAuth integrations are available such that we don't have to collect credentials, but for the ones without public APIs we do need to collect credentials (though we never store them). Unfortunately, the industry is in a mode where brokerages will develop anti-screen-scraping technology and data aggregators will develop new and creative solutions to evade detection - the demand for access to retail accounts is too great for screen scrapers to stop what they're doing, and the regulatory and technical risks are too great for brokerages to lean back and allow screen scrapers to do what they do. Many of the major brokerages are realizing that this isn't ideal and are starting to build out public APIs. We're working to develop relationships with the holdouts and convince them that exposing a public API is the only sustainable long-term solution to their screen scraping problem.
- tehwebguy 5y ago
  Can you explain how you collect credentials but don’t store them?
  - timothygoltser 5y ago
    We collect a user's credentials (usernames + passwords) and exchange them immediately for access and refresh tokens. We use the term "credentials" as we've seen it used colloquially - access and refresh tokens certainly fall under the formal definition of credentials, but we usually call them "sensitive tokens" or something similar. Additionally, in the case of an API like Webull's which accepts a salted MD5 hash of the user's password, the user's plaintext password will never touch our servers.
danielvaughn 5y ago
I used to do this exact thing as a job, and it's crazy how little US brokers do to protect their APIs (at least back when I was doing it around 2015), given the security implications. I'm not even a hacker and I was able to reverse engineer 90% of the US brokerages without a problem. Compare that to Singapore. I spent about an hour toying with the Lim & Tan API before our account got shut down. Apparently the fucking CTO was woken up in the middle of the night by all the alarm bells I set off lol. edit: when I say "without a problem", I mean that in relative terms. We got plenty of strongly-worded letters asking us to stop, and they played whack-a-mole with our instances. But we never faced a serious challenge to what we were doing.
- timothygoltser 5y ago
  We've had the same experience in 2022. An afternoon with an HTTPToolkit interceptor is usually all it takes to reverse engineer a brokerage's API; we spend most of our time paving over various brokerage-specific idiosyncrasies and responding to breaking changes in their APIs. We've discovered that building sane abstractions over the various retail brokerages in the US is a hard design problem given the variability in supported assets, account types, order strategies, authentication strategies, etc. Thankfully we haven't had to field any calls from irate, sleep-deprived CTOs yet - running experiments against APIs during local business hours only could be a good hack for avoiding those :)

by koolba 5y ago

timothygoltser 5y ago

We do sanctioned integrations with brokerages wherever OAuth integrations are available such that we don't have to collect credentials, but for the ones without public APIs we do need to collect credentials (though we never store them). Unfortunately, the industry is in a mode where brokerages will develop anti-screen-scraping technology and data aggregators will develop new and creative solutions to evade detection - the demand for access to retail accounts is too great for screen scrapers to stop what they're doing, and the regulatory and technical risks are too great for brokerages to lean back and allow screen scrapers to do what they do. Many of the major brokerages are realizing that this isn't ideal and are starting to build out public APIs. We're working to develop relationships with the holdouts and convince them that exposing a public API is the only sustainable long-term solution to their screen scraping problem.

tehwebguy 5y ago
Can you explain how you collect credentials but don’t store them?
- timothygoltser 5y ago
  We collect a user's credentials (usernames + passwords) and exchange them immediately for access and refresh tokens. We use the term "credentials" as we've seen it used colloquially - access and refresh tokens certainly fall under the formal definition of credentials, but we usually call them "sensitive tokens" or something similar. Additionally, in the case of an API like Webull's which accepts a salted MD5 hash of the user's password, the user's plaintext password will never touch our servers.

danielvaughn 5y ago

I used to do this exact thing as a job, and it's crazy how little US brokers do to protect their APIs (at least back when I was doing it around 2015), given the security implications. I'm not even a hacker and I was able to reverse engineer 90% of the US brokerages without a problem. Compare that to Singapore. I spent about an hour toying with the Lim & Tan API before our account got shut down. Apparently the fucking CTO was woken up in the middle of the night by all the alarm bells I set off lol. edit: when I say "without a problem", I mean that in relative terms. We got plenty of strongly-worded letters asking us to stop, and they played whack-a-mole with our instances. But we never faced a serious challenge to what we were doing.

timothygoltser 5y ago
We've had the same experience in 2022. An afternoon with an HTTPToolkit interceptor is usually all it takes to reverse engineer a brokerage's API; we spend most of our time paving over various brokerage-specific idiosyncrasies and responding to breaking changes in their APIs. We've discovered that building sane abstractions over the various retail brokerages in the US is a hard design problem given the variability in supported assets, account types, order strategies, authentication strategies, etc. Thankfully we haven't had to field any calls from irate, sleep-deprived CTOs yet - running experiments against APIs during local business hours only could be a good hack for avoiding those :)