There is a very fine line between authorized data, technically public but implicitly unauthorized data, and illegally obtained, unauthorized data. Here’s an exa

There is a very fine line between authorized data, technically public but implicitly unauthorized data, and illegally obtained, unauthorized data. Here’s an example of each in the financial sector, from my personal experience: 1. Financial account aggregators and “budget apps” like Min monetize their business, in part, by selling huge amounts of data to the financial sector. Sometimes companies like Second Measure take raw data from companies like Yodlee and clean it, then resell it. Nowadays there is an entire industry of alterative market research that has had all sorts of participants, from Foursquare (locations) to Spark (email enhancement). This is technically authorized, because it’s in the TOS. The users effectively contribute their own data. 2. I developed an extremely accurate, reasonably generalizable method of forecasting vehicle production at several companies that relies on implementing a VIN searching algorithm in conjunction with legally required NHTSA recall lookup portals hosted by each manufacturer. This data is what you’d call unauthorized, because no entity explicitly endorses your use of it. For example, several colleagues and I knew well ahead of time that Tesla would miss on production of the Model 3s because they were utterly unrepresented in our data. But this data is public, so it’s fine to use from a legal and compliance standpoint. It was lucrative data specifically because it had a high signal for revenue, yet was hitherto unused and unidentified. 3. I once found, in the course of looking for legally usable data, an actual security vulnerability disclosing all users of a publicly traded QSR’s online delivery service, along with their phone numbers, email addresses and last four digits of credit cards. This is both unauthorized and illegal, because the data is contaminated with personally identifiable information and it clearly requires a vulnerability (not just scraping) to acquire. I’ve seen overzealous data vendors accidentally slip from #2 into #3, which is really bad for all concerned. It’s not a great look for the vendor, who will likely be fired, and it represents a breach for the company who owns the data and its users. Any firm that has purchased the data will likely be contamined and be forced into a trading lockdown of that security for a period of time by compliance. My real concern is that illicit data like this is used in machine learning research. Machine learning is already pretty frustrating - it’s common for me to find research from a conference that I’m simply unable to replicate because the training or experiment data is not available (this is annoyingly the case with A/B experiment optimization research put out by giant companies in particular). I worry that this trend of accepting machine learning research without any requirement for total data transparency will incentivize researchers to conduct their experiments using illicit data that doesn’t need to be sourced.