4 ms·

I've done a lot of work involving scraping, data analysis and reversing private API access from mobile applications. A lot of this work is legally defensible (i

I've done a lot of work involving scraping, data analysis and reversing private API access from mobile applications. A lot of this work is legally defensible (if you do it correctly) and not particularly unethical. After doing a bunch of this type of work I started to accrue subject matter expertise in the area that led other companies to come to me with more questionable projects in mind. I turned down many of these, but two projects stand out to me as unethical or borderline. In fact, this was the reason I eventually stopped working with these companies. The first project was for a large, (now) well-known fintech company. They needed to develop login integrations with consumer banks to acquire customer account information for verification purposes. But many such banks didn't particularly want to grant them any special API access. More importantly, these banks typically forbid scraping and made it explicitly difficult by implementing JavaScript-based computational measures required on the client in order to successfully login. I helped this company develop methodologies for bypassing the anti-scraping measures on several banking websites. However, I stopped working on this because 1) I felt uncomfortable with the cavalier way they were ignoring banks' refusals, then using the reversed integrations and onboarded customers as a bargaining chip for more formal partnerships, and 2) performing huge amounts of analytics on customer data acquired as part of the account verification process. The second project was for a tech startup working on insurance and credit analytics. This company is one of several that popped up in recent years to use machine learning and social data in order to develop a more "complete" credit score (in their eyes). They had an impressive team of machine learning researchers but their data acquisition team was comparatively mediocre. So I worked with them to improve their acquisition methodologies for a variety of social media websites. I stopped working with them for three reasons: 1) fundamentally, I lost faith that their product was actually generating a meaningful signal over traditional means, 2) I was worried that the data they were collecting might introduce spurious correlations or illegal biases, and 3) if any team was going to do this correctly, I didn't think this particular team was the qualified one to do it.

jscholes 8y ago
> A lot of this work is legally defensible (if you do it correctly) Could you expand on the correctness aspect? I'm currently working in this space for what I believe to be good reasons (to improve the accessibility of a particular service for visually impaired users). But I'm eager not to abuse my position and knowledge.
- sjtgraham 8y ago
  Depends on jurisdiction OP operates in, but clean room reverse engineering is one technique, i.e. OP reverse engineers a piece of software, documents how it works at a high level, and hands off the documentation to legal counsel who reviews it for IP infringement etc. The doc is then handed off to a completely different team/engineer who implements the API contract documented by OP.
- throwawaymath 8y ago
  To be clear, I'm talking about scraping. I think the sibling commenter is talking about developing competing products via reverse engineering ("clean room implementation"). I am also not a lawyer, so I can only tell you the guidance I received from one for the projects I worked on. Technically speaking you can scrape data in a legally defensible way if you do not need to accept any terms of service explicitly prohibiting scraping in the course of grabbing the data. The distinction is that browsewrap T&C have plausible deniability, but clickwrap T&C do not. And if you receive a cease and desist order, you abide by it with a mea culpa. This also means you don't scrape so loudly as to be noticed, which has the happy side effect of probably not disrupting the target's service. But again: The grey areas of ethics are a separate question from legality. Please engage a lawyer for your specific work.
  - user5994461 8y ago
    Depends on the jurisdiction. For instance Europe allows reverse engineering for inter compatibility. Terms of services carry little value here (none for this clause that's already covered by a specific law). However, it would be interesting in the case of bank services. Accessing the account from the customer probably let you initiate a variety of actions like money transfers or loans, it's sensible to argue for limited and controlled inter interoperability. Also, customers are not allowed to share their credentials and he is in breach of his contract. The account should be considered compromised and be locked.
diminoten 8y ago
I don't necessarily agree that the first project was unethical, the banks were saying no because they wanted to develop a product of their own to compete with the company that hired you, not because they had any moral standing to prevent the customer from running analytics on their own data. The company you did the work for is immensely helpful to the consumer, and banks refusing to let consumers interact with their own data is, IMO, more of the moral violation than forcibly scraping that data out.
- mindslight 8y ago
  Even worse, the banks preventing users from straightforwardly accessing their own data is exactly what is pushing users right into the arms of these "fintech" surveillance companies. The technology (OFX direct connect) and software (Quicken, now Moneydance) for users to privately track their finances has been around for quite some time. But the protocol appears to be getting deprecated in favor of this "web download" rigmarole as banks attempt to decommodify. I had thought about futzing with Plaid for my own monitoring purposes, figuring since they were B2B they wouldn't be directly surveilling (leaving that up to customers instead). Apparently I was wrong. Surveillance capitalism, indeed!
qmalzp 8y ago
First project sounds like Coinbase.
- bgnm2000 8y ago
  I was thinking the first is Mint
ptenk 8y ago
Thank you for sharing. This is why I never sign up for any fintech services. The amount of data I hand over is never worth the "benefits" they provide.
- throwawaymath 8y ago
  Most of those sorts of companies (both B2C and B2B) ultimately end up selling the user transaction data (or some derivative analytics thereof) to hedge funds. It's used (for example) to forecast the revenue of B2C companies in advance of equity earnings announcements. NB: I don't have an issue with user data being mined for things like market research if it's a situation where the product is free and users can be easily made aware of it. But I find it dishonest if the company mining that data is doing so without direct user consent, or in a "backdoored" manner using their status as a downstream client's "affiliate" for T&C purposes.
ryanackley 8y ago
I'm guessing the first company is Plaid. I'm in the ACH payments space so we're a potential customer of Plaid and I vaguely remember a presentation or something where they (Dwolla + Plaid) promise merchants the ability to grab 12 months worth of transaction history from a user's bank account. Could be wrong about that. I do know for sure I made a mental note to myself to never use Plaid as a consumer.
- throwawaymath 8y ago
  Well, since you already guessed it: yes, it was Plaid.
  - charlesdm 8y ago
    I would have no issues working on this. Doesn't seem very unethical. (For people who know the space, it's pretty obvious that you're talking about Plaid :))
    - throwawaymath 8y ago
      Well I subscribe to moral relativism, so I don't have an issue with you not finding it to be unethical. But personally, it bothered me that so much user data would be mined from their financial statements. I know B2C companies like Mint do this as well, and I don't have an issue with user data being used for market research. But it seemed underhanded since most customers aren't aware of it. If you use Mint you can read their T&C and understand your data is mined. But Plaid is not sold to users, it's sold to companies. So end users of a completely different company which simply uses Plaid for account verification would be implicitly volunteering their data. The other thing that bothered me is that companies which have valid reasons for not wanting to develop API integrations for Plaid would be strong armed into doing so once Plaid had acquired a critical mass of their users using the methods I helped them develop.
asragab 8y ago
Throwing in a guess for the second company, was it Zest Finance? I know this isn't the point of the thread, but I couldn't help myself. I will ask for forgiveness (as I am sure we have all done in both technical and ethical matters)
joewee 8y ago
Sounds like you would be a valuable member of any company, especially a startup. I’d love to hear your thoughts on more ethical ideas for analyzing difficult to obtain data. Feel free to ping me directly.
- brokenmachine 8y ago
  > hear your thoughts on more ethical ideas for analyzing difficult to obtain data. See, this data is not difficult to obtain if you're operating in an ethical manner. You can't do ethical analysis of data obtained in an unethical fashion. Fruit of the poisoned tree and all that. I guess it worked for radiologists after nazi germany however...
driverdan 8y ago
What's wrong with allowing customers to fetch their own data from their bank? I hold the opposite point of view, that banks withholding information from their customers is unethical.

jscholes 8y ago

> A lot of this work is legally defensible (if you do it correctly) Could you expand on the correctness aspect? I'm currently working in this space for what I believe to be good reasons (to improve the accessibility of a particular service for visually impaired users). But I'm eager not to abuse my position and knowledge.

sjtgraham 8y ago
Depends on jurisdiction OP operates in, but clean room reverse engineering is one technique, i.e. OP reverse engineers a piece of software, documents how it works at a high level, and hands off the documentation to legal counsel who reviews it for IP infringement etc. The doc is then handed off to a completely different team/engineer who implements the API contract documented by OP.
throwawaymath 8y ago
To be clear, I'm talking about scraping. I think the sibling commenter is talking about developing competing products via reverse engineering ("clean room implementation"). I am also not a lawyer, so I can only tell you the guidance I received from one for the projects I worked on. Technically speaking you can scrape data in a legally defensible way if you do not need to accept any terms of service explicitly prohibiting scraping in the course of grabbing the data. The distinction is that browsewrap T&C have plausible deniability, but clickwrap T&C do not. And if you receive a cease and desist order, you abide by it with a mea culpa. This also means you don't scrape so loudly as to be noticed, which has the happy side effect of probably not disrupting the target's service. But again: The grey areas of ethics are a separate question from legality. Please engage a lawyer for your specific work.
- user5994461 8y ago
  Depends on the jurisdiction. For instance Europe allows reverse engineering for inter compatibility. Terms of services carry little value here (none for this clause that's already covered by a specific law). However, it would be interesting in the case of bank services. Accessing the account from the customer probably let you initiate a variety of actions like money transfers or loans, it's sensible to argue for limited and controlled inter interoperability. Also, customers are not allowed to share their credentials and he is in breach of his contract. The account should be considered compromised and be locked.

diminoten 8y ago

I don't necessarily agree that the first project was unethical, the banks were saying no because they wanted to develop a product of their own to compete with the company that hired you, not because they had any moral standing to prevent the customer from running analytics on their own data. The company you did the work for is immensely helpful to the consumer, and banks refusing to let consumers interact with their own data is, IMO, more of the moral violation than forcibly scraping that data out.

mindslight 8y ago
Even worse, the banks preventing users from straightforwardly accessing their own data is exactly what is pushing users right into the arms of these "fintech" surveillance companies. The technology (OFX direct connect) and software (Quicken, now Moneydance) for users to privately track their finances has been around for quite some time. But the protocol appears to be getting deprecated in favor of this "web download" rigmarole as banks attempt to decommodify. I had thought about futzing with Plaid for my own monitoring purposes, figuring since they were B2B they wouldn't be directly surveilling (leaving that up to customers instead). Apparently I was wrong. Surveillance capitalism, indeed!

qmalzp 8y ago

First project sounds like Coinbase.

bgnm2000 8y ago
I was thinking the first is Mint

ptenk 8y ago

Thank you for sharing. This is why I never sign up for any fintech services. The amount of data I hand over is never worth the "benefits" they provide.

throwawaymath 8y ago
Most of those sorts of companies (both B2C and B2B) ultimately end up selling the user transaction data (or some derivative analytics thereof) to hedge funds. It's used (for example) to forecast the revenue of B2C companies in advance of equity earnings announcements. NB: I don't have an issue with user data being mined for things like market research if it's a situation where the product is free and users can be easily made aware of it. But I find it dishonest if the company mining that data is doing so without direct user consent, or in a "backdoored" manner using their status as a downstream client's "affiliate" for T&C purposes.

ryanackley 8y ago

I'm guessing the first company is Plaid. I'm in the ACH payments space so we're a potential customer of Plaid and I vaguely remember a presentation or something where they (Dwolla + Plaid) promise merchants the ability to grab 12 months worth of transaction history from a user's bank account. Could be wrong about that. I do know for sure I made a mental note to myself to never use Plaid as a consumer.

throwawaymath 8y ago
Well, since you already guessed it: yes, it was Plaid.
- charlesdm 8y ago
  I would have no issues working on this. Doesn't seem very unethical. (For people who know the space, it's pretty obvious that you're talking about Plaid :))
  - throwawaymath 8y ago
    Well I subscribe to moral relativism, so I don't have an issue with you not finding it to be unethical. But personally, it bothered me that so much user data would be mined from their financial statements. I know B2C companies like Mint do this as well, and I don't have an issue with user data being used for market research. But it seemed underhanded since most customers aren't aware of it. If you use Mint you can read their T&C and understand your data is mined. But Plaid is not sold to users, it's sold to companies. So end users of a completely different company which simply uses Plaid for account verification would be implicitly volunteering their data. The other thing that bothered me is that companies which have valid reasons for not wanting to develop API integrations for Plaid would be strong armed into doing so once Plaid had acquired a critical mass of their users using the methods I helped them develop.

asragab 8y ago

Throwing in a guess for the second company, was it Zest Finance? I know this isn't the point of the thread, but I couldn't help myself. I will ask for forgiveness (as I am sure we have all done in both technical and ethical matters)

joewee 8y ago

Sounds like you would be a valuable member of any company, especially a startup. I’d love to hear your thoughts on more ethical ideas for analyzing difficult to obtain data. Feel free to ping me directly.

brokenmachine 8y ago
> hear your thoughts on more ethical ideas for analyzing difficult to obtain data. See, this data is not difficult to obtain if you're operating in an ethical manner. You can't do ethical analysis of data obtained in an unethical fashion. Fruit of the poisoned tree and all that. I guess it worked for radiologists after nazi germany however...

driverdan 8y ago

What's wrong with allowing customers to fetch their own data from their bank? I hold the opposite point of view, that banks withholding information from their customers is unethical.