3 ms·

While we try to form relationships with all of the brokerages we integrate with (including the ones we reverse engineer), we do still have to do the abuse/block

While we try to form relationships with all of the brokerages we integrate with (including the ones we reverse engineer), we do still have to do the abuse/block dance with their protective systems occasionally. We've found that cutting out headless browsers has made it much easier to do this, for two main reasons: 1) Some brokerages have fairly sophisticated anti-screen-scraping protections, but their private APIs are comparatively undefended. 2) It's generally more difficult to create protective systems for private APIs, since there are fewer ways to fingerprint non-browser clients.

ceejayoz 5y ago
I think "direct integrations" as a euphemism from "using reverse-engineered private APIs in violation of their terms" is dishonest.
- timothygoltser 5y ago
  That's a fair point - we've been trying to determine the best way to compress "using reverse-engineered private APIs in violation of their terms, but in practice brokerages don't really take enforcement actions against this," and "direct integrations" is what we came up with. We'll work on finding a better way to express this and are open to suggestions.
  - alistairclark7 5y ago
    I think "unofficial API" is probably the most succinct phrase without being dishonest. A direct integration does imply some kind of professional relationship with the other company.
    - seandoh 5y ago
      This sounds good - thank you for the feedback.
- sjtgraham 5y ago
  It's not as long as they expressly state there is no relationship or endorsement with the brokerages where there isn't one.
  - ceejayoz 5y ago
    Perhaps. The current site and documentation appears to lack anything of the sort.
    - sjtgraham 5y ago
      It’s a brand new company. Give them a chance to work these things out.
      - drekk 5y ago
        Well, they're taking shots at players like Plaid (who use majority direct oAuth integrations) while doing the exact same kind of skirting around companies that don't play nice and share permissioned user data. Aggregators aren't scraping because they enjoy using headless browsers, the US doesn't let end users own their data and this is the industry workaround. I don't expect new players to change the system so much as I expect them to accurately represents it to those outside of fintech
svnt 5y ago
> 2) It's generally more difficult to create protective systems for private APIs, since there are fewer ways to fingerprint non-browser clients. There are? I think what you meant to say is “they haven’t worried much about key distribution, yet”?
- timothygoltser 5y ago
  In a sense. There are certainly protections they can implement against non-browser clients; what I meant is that it would be more difficult for them to implement the usual sorts of defenses you see against screen scraping since the surface area available for fingerprinting is much smaller. We haven't seen brokerages expend significant effort on this, and the industry seems to be moving in the direction of providing more open access to APIs, so we're (cautiously) optimistic that we will be able to convince them to provide more sanctioned integration paths such that we don't need to continue playing cat-mouse with them.
  - rezbull 5y ago
    What incentives do mainstream brokerages have to building public APIs? It seems to me data asymmetry is the name of the game in Wall Street and it’s no different for retail brokers.
namdnay 5y ago
“I ask my neighbours for some of their apples, but if they don’t give them, I go into their garden at nighttime with a big ladder…”
- not_exactly__ 5y ago
  Wrong analogy. You’re describing theft. It’s More like, my neighbor planted an apple tree in their yard and told me, “this is your tree. You can pick any fruits off of it, anytime you’d like.” I told the jam making shop down the street that they can come get apples on this tree anytime if they give me some of the apple jam they end up making. My neighbor didn’t explicit permit me to do this, but also didn’t forbid me. I think? So now the jam making shop owner goes and grab my apple on my neighbor’s tree. Maybe the solution is just to have the jam maker give the neighbor a can?
  - namdnay 5y ago
    Except that it isn’t you inviting the jam maker, it’s the jam maker saying “I have a direct integration with the guy who manages your apple tree” , when they actually mean “I’ll dress up like you and pick the apples when nobody’s looking”

ceejayoz 5y ago

I think "direct integrations" as a euphemism from "using reverse-engineered private APIs in violation of their terms" is dishonest.

timothygoltser 5y ago
That's a fair point - we've been trying to determine the best way to compress "using reverse-engineered private APIs in violation of their terms, but in practice brokerages don't really take enforcement actions against this," and "direct integrations" is what we came up with. We'll work on finding a better way to express this and are open to suggestions.
- alistairclark7 5y ago
  I think "unofficial API" is probably the most succinct phrase without being dishonest. A direct integration does imply some kind of professional relationship with the other company.
  - seandoh 5y ago
    This sounds good - thank you for the feedback.
sjtgraham 5y ago
It's not as long as they expressly state there is no relationship or endorsement with the brokerages where there isn't one.
- ceejayoz 5y ago
  Perhaps. The current site and documentation appears to lack anything of the sort.
  - sjtgraham 5y ago
    It’s a brand new company. Give them a chance to work these things out.
    - drekk 5y ago
      Well, they're taking shots at players like Plaid (who use majority direct oAuth integrations) while doing the exact same kind of skirting around companies that don't play nice and share permissioned user data. Aggregators aren't scraping because they enjoy using headless browsers, the US doesn't let end users own their data and this is the industry workaround. I don't expect new players to change the system so much as I expect them to accurately represents it to those outside of fintech

svnt 5y ago

> 2) It's generally more difficult to create protective systems for private APIs, since there are fewer ways to fingerprint non-browser clients. There are? I think what you meant to say is “they haven’t worried much about key distribution, yet”?

timothygoltser 5y ago
In a sense. There are certainly protections they can implement against non-browser clients; what I meant is that it would be more difficult for them to implement the usual sorts of defenses you see against screen scraping since the surface area available for fingerprinting is much smaller. We haven't seen brokerages expend significant effort on this, and the industry seems to be moving in the direction of providing more open access to APIs, so we're (cautiously) optimistic that we will be able to convince them to provide more sanctioned integration paths such that we don't need to continue playing cat-mouse with them.
- rezbull 5y ago
  What incentives do mainstream brokerages have to building public APIs? It seems to me data asymmetry is the name of the game in Wall Street and it’s no different for retail brokers.

namdnay 5y ago

“I ask my neighbours for some of their apples, but if they don’t give them, I go into their garden at nighttime with a big ladder…”

not_exactly__ 5y ago
Wrong analogy. You’re describing theft. It’s More like, my neighbor planted an apple tree in their yard and told me, “this is your tree. You can pick any fruits off of it, anytime you’d like.” I told the jam making shop down the street that they can come get apples on this tree anytime if they give me some of the apple jam they end up making. My neighbor didn’t explicit permit me to do this, but also didn’t forbid me. I think? So now the jam making shop owner goes and grab my apple on my neighbor’s tree. Maybe the solution is just to have the jam maker give the neighbor a can?
- namdnay 5y ago
  Except that it isn’t you inviting the jam maker, it’s the jam maker saying “I have a direct integration with the guy who manages your apple tree” , when they actually mean “I’ll dress up like you and pick the apples when nobody’s looking”