5 ms·

Because there is a real moat with data ownership and pipelines. If you want to do any analysis you quickly find that learning to properly use scikit-learn and

Because there is a real moat with data ownership and pipelines. If you want to do any analysis you quickly find that learning to properly use scikit-learn and tensorflow (or your machine learning algorithm of your choice) is atleast an order of magnitude lower of work than getting the data. For instance, I wanted to build a machine learning algorithm which took simple data from the SEC filed 10-Q and 10-K, which are freely available online and predict whether stocks were likely to outperform the market average for over the next 3 years. Time to setup up scikit learn and tensorflow algorithms to make predictions: 4 hours. Time to setup python scripts which could parse through the excel spreadsheets, figure out which row corresponded to gross profit margin, and a few other "standard" metrics: ??? unknown because I gave up at about 80 hours trying to figure out rules to process all the different spreadsheets and how names were determined. I had a professor who was doing machine learning + chemistry. He was building up his own personal database for machine learning. He spent ~5 years with about 500 computers building the database so that he would be able to do the actual the machine learning.

axegon_ 7y ago
So much this!!! In all fairness, it doesn't matter what you pick up, you'll spend north of 80% of your time preparing and pre-processing data, which is almost always catastrophically tedious and boring. Annoyingly that also applies to publicly available datasets - pre-processing is still most of the work. For instance, the tensorflow team invested a lot of time and effort into tf.data for that reason but imo it doesn't make things a whole lot better.
- deleted 7y ago
  [deleted]
- geebee 7y ago
  I agree with you, though I actually personally don't find pre-processing data tedious and boring. I kind of like knitting it all together. On a side note... data pre-processing is often viewed a side job that needs to get done before the real work can begin. I don't think I've ever been able to prepare a data pipeline without making decisions about the data that will impact the outcome. For example How do you deal with missing data? Interpolate, ignore, use averages, use a machine learning algorithm to plug the gaps? How do you decide what data sources to include in the pipeline. What if one data source seems more reliable, but another has far more data, too much to use. Should you amplify it, sample from the other data sets to make the volumes equal, keep things proportional? What if one data set changes more rapidly than another, how should you update the data set used to populate the ML model? These are just a few that jump into my head, and while there are techniques to deal with them, ultimately, there isn't a correct answer. And honestly, even these examples make it all seem more glamorous than it is, a lot of this is just figuring out why various encoding and formatting errors are breaking the feed, why column headers mysteriously change, why handwriting on form scans gets properly translated into text some of the time and completely garbled in others. The funny thing is, I do see people fine tuning ML algorithms (kaggle style, seeing if they can wring a bit more predictiveness out of a model), when in the real world projects, decisions upstream about the data pipeline will have an impact perhaps 5-10 times greater than any tweak to the ML parameters (or even which general algorithm to choose). And yet, it's hard to get people to even pay attention to these decisions, probably - as you said - because people find it catastrophically tedious and boring.
  - emmanuel_1234 7y ago
    As a corollary, "Data Scientists" who can't program their way out of a paper bag (e.g.: write simple SQL, or a scraper in Python) is near useless, and a stress on their peer who can. I'd much rather hire a good programmer with some statistical knowledge than the other way around.
    - dawg- 7y ago
      Either way, you are gonna be paying someone to fill in gaps in their knowledge. Anyone with a laptop and an internet connection can learn to "program their way out of a paper bag" in a weekend.
      - geebee 7y ago
        I kinda disagree, though I suppose it depends on the paper bag.
    - omar_a1 7y ago
      I've seen so much bad math and misunderstood statistics hard coded into widely-implemented software that that's not ideal either. Ultimately, you need a team with a combination of strengths if your product requires multidisciplinary work. Otherwise you end up with hilariously wrong equations/assumptions in your code base. But, full disclosure, my background is math/science.
- dsaavy 7y ago
  Yep, I actually started a company that cleans data reliably because of this. It’s usually a huge waste of resources for a highly paid person to clean data (like ML specialists) and most data cleaning companies just outsource to overseas resources that aren’t reliable or knowledgeable beyond an Excel skill set.
papln 7y ago
> data ownership and pipelines not ML. By your argument, the money is in data hoarding and brokering, and renting that data (with DRM) to ML outfits, not doing ML. Anyway, cleaning dirty data isn't execeptionally hard, it's just boring work; getting the raw data is the hard part.
- killjoywashere 7y ago
  Um, yes and no. Some of the data annotation problems get really hard and really expensive. Look at radiology or genomics. You literally need N people to die for more knowledge in some of these situations.
  - jnwatson 7y ago
    Data cleaning and data labeling are two separate tasks, with vastly different workflows. For example, getting structured data out of Whois requires a lot of cleanup. Labeling domains as “bad” might take forever.
- nostrademons 7y ago
  This is an interesting observation that deserves to be addressed. I think in practice, what happens is that there's no easy transactional boundary that can keep the data ownership and ML in separate firms. The theory of the firm [1] states that firms arise when transaction costs are less than the economic inefficiencies of centralized resource allocation. There are some pretty heavy transaction costs between doing data cleaning and data science in separate organizations: 1.) DRM for datasets isn't really a thing, since to explore, visualize, and train on them, you need access to the raw data, and then instead of your machine-learning function you can just pass the identity function to get the raw data. DRM for consumers always relies on a publisher whose incentive is to stay in business (by not breaking any laws) rather than to obtain the raw data. 2.) You don't know whether a cleaned data set will be useful for your ML application until you've inspected it, visualized it, run some statistics over it, etc. at which point you've done most of the work for setting up your models. That means there's a big risk premium for buying data, and potential buyers usually want samples & statistics before committing a lot of money. 3.) At the price that good datasets go for, you're usually dealing with enterprise sales, which involves commissioned salespeople, face-to-face meetings, expensive dinners out, etc. 4.) The type of labeling you need to do is often intimately connected with the usage of the ML model. What constitutes spam? What constitutes abuse? These are questions for your policy team, which the data-collection organization would have no visibility into and no way to set up a one-size-fits-all policy that all potential customers would be okay with. 5.) Software markets tend to be winner-take-all, which means you're dealing with a handful of customers, and one will likely become dominant and able to acquire you. Instead of separate firms for data ownership, ML, and user interface, usually the service that makes the final product will end up collecting or buying the data outright, hire data-scientists to do the ML, and then offer a product. The end result, economically, is what you say: the money is in data hoarding. But it isn't apparent in prices of a sustainable customer ecosystem, it's apparent in acquisition prices for startups with data vs. salaries of data scientists. [1] https://en.wikipedia.org/wiki/The_Nature_of_the_Firm https://en.wikipedia.org/wiki/The_Nature_of_the_Firm
variaga 7y ago
Agree 100% - the single largest problem with applying ML to anything is getting (1) enough (2) accurate (3) representative (4) correctly labeled data to train and validate the models. Everything else is noise. For any proposed ML project, the first question you should ask is "Where are you going to get the training data?" If someone tells you "Oh, getting data will be the easy part", back away very slowly :)
killjoywashere 7y ago
So this. The data problem is being controlled, fought, and negotiated at the level of nation-states and indeed whole meta-national cultures.
H8crilA 7y ago
You should've bought the processed 10q and 10k data ranging many years for all US stocks. I bet you can get that for under $1000. This is the most basic securities data that exists in finance, perhaps you can even get it for free from some Yahoo or other retail data source.
- peterwoerner 7y ago
  Yes, but I was in graduate school making $1500/mo and was doing it mostly for fun as a class project. I was also very naive about how much effort it would take to process that data. Both google and yahoo have current data which is easy to get but not historical. Google didn't really help me find what I was looking for.
ayw 7y ago
Strongly agree with this. One thing I’ll mention is that this is true both at the very early stages of a ML project, and even when an ML project is scaled up and in production. Oftentimes, the data pipeline is the true way in which a model will improve versus anything else, so it’s pretty critical that these data pipelines are setup to get an initial dataset but also to scale properly. It’s one reason I started Scale (scale.com). It was viscerally clear that the real bottleneck to ML was getting the needed data, and in our case, annotating that data appropriately. It is very heartening to hear it echoed in this whole thread that data is very clearly what “matters” for ML.
emmamm05 7y ago
Strongly agreed. In my daily job, the exciting, shiny machine learning work is superfluous compared to the strong keyboard smashing of having to parse millions of rows of excel sheets with incomprehensible organization and column naming.
foldingmoney 7y ago
>Time to setup python scripts which could parse through the excel spreadsheets, figure out which row corresponded to gross profit margin, and a few other "standard" metrics: ??? If you just want basic financial data there are easier ways, e.g. the Quandl API (not affiliated). But the bigger issue is that you can't accurately label your training data. A subsequent stock price move could have been due by something revealed in the 10-Q that the market somehow missed, or it could have been caused by a forest fire in Bolivia, or a competitor getting acquired, or any combination of near infinite external events that couldn't have been predicted from the 10-Q.

axegon_ 7y ago

So much this!!! In all fairness, it doesn't matter what you pick up, you'll spend north of 80% of your time preparing and pre-processing data, which is almost always catastrophically tedious and boring. Annoyingly that also applies to publicly available datasets - pre-processing is still most of the work. For instance, the tensorflow team invested a lot of time and effort into tf.data for that reason but imo it doesn't make things a whole lot better.

deleted 7y ago
[deleted]
geebee 7y ago
I agree with you, though I actually personally don't find pre-processing data tedious and boring. I kind of like knitting it all together. On a side note... data pre-processing is often viewed a side job that needs to get done before the real work can begin. I don't think I've ever been able to prepare a data pipeline without making decisions about the data that will impact the outcome. For example How do you deal with missing data? Interpolate, ignore, use averages, use a machine learning algorithm to plug the gaps? How do you decide what data sources to include in the pipeline. What if one data source seems more reliable, but another has far more data, too much to use. Should you amplify it, sample from the other data sets to make the volumes equal, keep things proportional? What if one data set changes more rapidly than another, how should you update the data set used to populate the ML model? These are just a few that jump into my head, and while there are techniques to deal with them, ultimately, there isn't a correct answer. And honestly, even these examples make it all seem more glamorous than it is, a lot of this is just figuring out why various encoding and formatting errors are breaking the feed, why column headers mysteriously change, why handwriting on form scans gets properly translated into text some of the time and completely garbled in others. The funny thing is, I do see people fine tuning ML algorithms (kaggle style, seeing if they can wring a bit more predictiveness out of a model), when in the real world projects, decisions upstream about the data pipeline will have an impact perhaps 5-10 times greater than any tweak to the ML parameters (or even which general algorithm to choose). And yet, it's hard to get people to even pay attention to these decisions, probably - as you said - because people find it catastrophically tedious and boring.
- emmanuel_1234 7y ago
  As a corollary, "Data Scientists" who can't program their way out of a paper bag (e.g.: write simple SQL, or a scraper in Python) is near useless, and a stress on their peer who can. I'd much rather hire a good programmer with some statistical knowledge than the other way around.
  - dawg- 7y ago
    Either way, you are gonna be paying someone to fill in gaps in their knowledge. Anyone with a laptop and an internet connection can learn to "program their way out of a paper bag" in a weekend.
    - geebee 7y ago
      I kinda disagree, though I suppose it depends on the paper bag.
  - omar_a1 7y ago
    I've seen so much bad math and misunderstood statistics hard coded into widely-implemented software that that's not ideal either. Ultimately, you need a team with a combination of strengths if your product requires multidisciplinary work. Otherwise you end up with hilariously wrong equations/assumptions in your code base. But, full disclosure, my background is math/science.
dsaavy 7y ago
Yep, I actually started a company that cleans data reliably because of this. It’s usually a huge waste of resources for a highly paid person to clean data (like ML specialists) and most data cleaning companies just outsource to overseas resources that aren’t reliable or knowledgeable beyond an Excel skill set.

papln 7y ago

> data ownership and pipelines not ML. By your argument, the money is in data hoarding and brokering, and renting that data (with DRM) to ML outfits, not doing ML. Anyway, cleaning dirty data isn't execeptionally hard, it's just boring work; getting the raw data is the hard part.

killjoywashere 7y ago
Um, yes and no. Some of the data annotation problems get really hard and really expensive. Look at radiology or genomics. You literally need N people to die for more knowledge in some of these situations.
- jnwatson 7y ago
  Data cleaning and data labeling are two separate tasks, with vastly different workflows. For example, getting structured data out of Whois requires a lot of cleanup. Labeling domains as “bad” might take forever.
nostrademons 7y ago
This is an interesting observation that deserves to be addressed. I think in practice, what happens is that there's no easy transactional boundary that can keep the data ownership and ML in separate firms. The theory of the firm [1] states that firms arise when transaction costs are less than the economic inefficiencies of centralized resource allocation. There are some pretty heavy transaction costs between doing data cleaning and data science in separate organizations: 1.) DRM for datasets isn't really a thing, since to explore, visualize, and train on them, you need access to the raw data, and then instead of your machine-learning function you can just pass the identity function to get the raw data. DRM for consumers always relies on a publisher whose incentive is to stay in business (by not breaking any laws) rather than to obtain the raw data. 2.) You don't know whether a cleaned data set will be useful for your ML application until you've inspected it, visualized it, run some statistics over it, etc. at which point you've done most of the work for setting up your models. That means there's a big risk premium for buying data, and potential buyers usually want samples & statistics before committing a lot of money. 3.) At the price that good datasets go for, you're usually dealing with enterprise sales, which involves commissioned salespeople, face-to-face meetings, expensive dinners out, etc. 4.) The type of labeling you need to do is often intimately connected with the usage of the ML model. What constitutes spam? What constitutes abuse? These are questions for your policy team, which the data-collection organization would have no visibility into and no way to set up a one-size-fits-all policy that all potential customers would be okay with. 5.) Software markets tend to be winner-take-all, which means you're dealing with a handful of customers, and one will likely become dominant and able to acquire you. Instead of separate firms for data ownership, ML, and user interface, usually the service that makes the final product will end up collecting or buying the data outright, hire data-scientists to do the ML, and then offer a product. The end result, economically, is what you say: the money is in data hoarding. But it isn't apparent in prices of a sustainable customer ecosystem, it's apparent in acquisition prices for startups with data vs. salaries of data scientists. [1] https://en.wikipedia.org/wiki/The_Nature_of_the_Firm https://en.wikipedia.org/wiki/The_Nature_of_the_Firm

variaga 7y ago

Agree 100% - the single largest problem with applying ML to anything is getting (1) enough (2) accurate (3) representative (4) correctly labeled data to train and validate the models. Everything else is noise. For any proposed ML project, the first question you should ask is "Where are you going to get the training data?" If someone tells you "Oh, getting data will be the easy part", back away very slowly :)

killjoywashere 7y ago

So this. The data problem is being controlled, fought, and negotiated at the level of nation-states and indeed whole meta-national cultures.

H8crilA 7y ago

You should've bought the processed 10q and 10k data ranging many years for all US stocks. I bet you can get that for under $1000. This is the most basic securities data that exists in finance, perhaps you can even get it for free from some Yahoo or other retail data source.

peterwoerner 7y ago
Yes, but I was in graduate school making $1500/mo and was doing it mostly for fun as a class project. I was also very naive about how much effort it would take to process that data. Both google and yahoo have current data which is easy to get but not historical. Google didn't really help me find what I was looking for.

ayw 7y ago

Strongly agree with this. One thing I’ll mention is that this is true both at the very early stages of a ML project, and even when an ML project is scaled up and in production. Oftentimes, the data pipeline is the true way in which a model will improve versus anything else, so it’s pretty critical that these data pipelines are setup to get an initial dataset but also to scale properly. It’s one reason I started Scale (scale.com). It was viscerally clear that the real bottleneck to ML was getting the needed data, and in our case, annotating that data appropriately. It is very heartening to hear it echoed in this whole thread that data is very clearly what “matters” for ML.

emmamm05 7y ago

Strongly agreed. In my daily job, the exciting, shiny machine learning work is superfluous compared to the strong keyboard smashing of having to parse millions of rows of excel sheets with incomprehensible organization and column naming.

foldingmoney 7y ago

>Time to setup python scripts which could parse through the excel spreadsheets, figure out which row corresponded to gross profit margin, and a few other "standard" metrics: ??? If you just want basic financial data there are easier ways, e.g. the Quandl API (not affiliated). But the bigger issue is that you can't accurately label your training data. A subsequent stock price move could have been due by something revealed in the 10-Q that the market somehow missed, or it could have been caused by a forest fire in Bolivia, or a competitor getting acquired, or any combination of near infinite external events that couldn't have been predicted from the 10-Q.