5 ms·

> they paid more to get those insights than they were worth! This understates how awful ML is at many of these companies. I've seen quite a few companies that

> they paid more to get those insights than they were worth! This understates how awful ML is at many of these companies. I've seen quite a few companies that rushed to hire teams of people with a PhD in anything that barely made it through a DS/ML boot camp. To prove that they're super smart ML researchers without fail these hires rush to deploy a 3+ layer MLP to solve a problem that need at most a simple regression. They have no understanding of how this model works, and have zero engineering sense so they don't care if it's a nightmare of complexity to maintain. Then to make sure their work is 'valuable' management tries to get as many teams as possible to make use of the questionable outputs of these models. The end is a nightmare of tightly coupled models that nobody can debug, trouble shoot or understand. And because the people building them don't really understand how they work the results are always very noisy. So you end up with this mess of expensive to build and run models talking noise to each other. When I saw this I realized data science was doomed in the next recession, since the only solution to this mess is to just remove it all. There is some really valuable DS work out there, but it requires real understanding of either modeling or statistics. That work will probably stick around, but these giant farms of boot camp grads churning out keras models will disappear soon.

disgruntledphd2 6y ago
And this is a good thing! To be fair, I started to understand why developers gave out about bootcamp grads lacking a foundation when the bootcamps came for my discipline (data science). The PhD fetish is pretty mental (even though I have one), as it's really not necessary. Additionally, everyone thinks they need researchers, when they really, really don't. Having worked with researchy vs more product/business driven teams, I found that the best results came when a researchy person took the time to understand the product domain, but many of them believe they're too good for business (in which case you should head back to academia). What you actually need from an ML/Data Science person: - Experience with data cleaning (this is most of the gig) - A solid understanding of linear and logistic regression, along with cross-validation - Some reasonable coding skills (in both R and Python, with a side of SQL). That's it. Pretty much everything else can be taught, given the above prerequisites. But it's tricky for hiring managers/companies as they don't know who to hire, so they end up over-indexing on bullshitters, due to the confidence, leading to lots of nonsese. And finally, deep learning is good in some scenarios and not in others, so anyone who's just a deep learning developer is not going to be useful to most companies.
- hogFeast 6y ago
  Just an anecdote but if you go to most baseball data departments, where there is real competition between teams, you don't just have PHds. You have people with undergrads/domain knowledge, and people with PHds. This isn't to say that PHd knowledge isn't valuable but if you look at firms in finance that have had success with data i.e. RenTech, they hire very smart people with PHds but it isn't only the PHd. You need someone who has the knowledge AND someone who has common sense/can get results. That is very hard to do correctly (and yes, some people who come from academia literally do not want anything to do with business...it is like the devs who come from a CS PHd and insist on using complicated algo and data structure everywhere, optimising every line, etc.).
  - mkl 6y ago
    *PhD or Ph.D. It's an abbreviation of the Latin "philosophiae doctor" == "doctor of philosophy". Getting one is as much about persistence as intelligence, and makes you very knowledgeable about one narrow area, so what you say makes sense. Academic researchers usually branch out into other fields and subfields as well, but straight out of a PhD, narrow and deep is what you tend to get.
tachyonbeam 6y ago
I worked in a place full of deep learning PhDs, and you'd have people trying to apply reinforcement learning to problems that had known mathematical solutions, and integer programming problems. I don't think the issue is just that companies hire people who are awful at ML, it's also that people are trying to shoehorn deep learning into everything, even when it currently has nothing to offer and we have better solutions already. IMHO, we're producing too many deep learning PhDs.
- hogFeast 6y ago
  This is just my general sense, as a very non-expert with more experience of doing than theory...but the benefit is someone knowing the theory AND being able to translate that into revenue. I think most people view the hard part as doing the PHd, and so lots of people value that experience, and because they have that experience you have this endowment effect: wow, that PHd was hard, I must do very hard and complex things. To give you an example: Man Group. They are a huge quant hedge fund, in fact they were one of the first big quant funds. Now, they even have their own program at Oxford University that they hire out of...have you heard of them? Most people haven't. Their performance is mostly terrible, and despite being decades ahead of everyone their returns were never very good (they did well at the start because they had a few exceptional employees, who then went elsewhere...David Harding was one). The issue isn't PHds, they have many of them, the issue is having that knowledge AND being able to convert it. I think this is really hard to grasp because most people expect problems to yield instantly to ML but, in most cases, they don't and other people have done valuable work with non-ML stuff that should be built on but isn't because domain knowledge or common sense is often lacking. A similar thing is people who come out of CS, and don't know how to program. They know a bit but they don't know how to use Git, they don't know how to write code others can read, etc.
  - smabie 6y ago
    The Man Group has had respectable returns, especially during Coronavirus. Nothing amazing, but certainly not terrible. Regardless, there's more to the picture: Sharpe ratio, vol, correlation to the market, etc
mumblemumble 6y ago
My sense is that the original sin here is conflating data science with machine learning. A good data scientist might choose to use machine learning to accomplish their job. Or they might find that classical statistical inference is the better tool for the task at hand. A good data scientist, having built this model, might choose to put it into production. Or they might find that a simple if-statement could do the job almost as effectively but not nearly as expensively. A good data scientist, having decided to productionize a model, will also provide some information about how it might break down - for example, describing shifts in customer behavior, or changes in how some input signal is generated, or feedback effects that might invalidate the model. OTOH, if your job has been framed in terms of cutting-edge machine learning, then you may well know - at a gut level, if not consciously - that your job is basically just a pissing match to see who can deploy the most bleeding-edge or expensive technology the fastest. It's like the modern hospital childbirth scene in Monty Python's The Meaning of Life, where the doctor is more interested in showing off the machine that goes, "ping!" in order to impress the other doctors than he is in paying attention to the mother.
- aspaceman 6y ago
  There’s people who consider classical inference and the like to be machine learning just as much as neural nets are. I like that perspective.
  - mumblemumble 6y ago
    There are some things, like OLS and logistic regression, that are commonly used for both purposes. But there's a sort of moral distinction between machine learning and statistical inference, driven by whether you consider your key deliverable to be y-hat or beta-hat, that ends up having implications. For example, I can get pretty preoccupied with multicollinearity or heteroskedasticity when I'm wearing my statistician hat, while they barely qualify as passing diversions when I'm wearing my machine learning engineer hat. If I'm doing ML, I'll happily deliberately bias the model. That would be anathema if I were doing statistical inference.
    - 6y ago
visarga 6y ago
It's just a process of exploration, people trying out ideas to see what works. Over time, with sharing of results, we will gradually discover more nuanced approaches, but the exploration phase is necessary in order to map a path forward and train a generation of ML engineers and PM's who don't have seniors to learn from. Of course it sucks on the short term, but there is zero chance the field will be abandoned. It has enough uses already.
- ric2b 6y ago
  But the exploration should be led by the data scientists, not by the suits who think they need a specific type of model.
emmap21 6y ago
It sounds so painful as someone all-in this area. But I have to agree on a task of overdoing with fancy models. Nevertheless, the most common ML algo in industry is still linear regression along with boostraping.

disgruntledphd2 6y ago

And this is a good thing! To be fair, I started to understand why developers gave out about bootcamp grads lacking a foundation when the bootcamps came for my discipline (data science). The PhD fetish is pretty mental (even though I have one), as it's really not necessary. Additionally, everyone thinks they need researchers, when they really, really don't. Having worked with researchy vs more product/business driven teams, I found that the best results came when a researchy person took the time to understand the product domain, but many of them believe they're too good for business (in which case you should head back to academia). What you actually need from an ML/Data Science person: - Experience with data cleaning (this is most of the gig) - A solid understanding of linear and logistic regression, along with cross-validation - Some reasonable coding skills (in both R and Python, with a side of SQL). That's it. Pretty much everything else can be taught, given the above prerequisites. But it's tricky for hiring managers/companies as they don't know who to hire, so they end up over-indexing on bullshitters, due to the confidence, leading to lots of nonsese. And finally, deep learning is good in some scenarios and not in others, so anyone who's just a deep learning developer is not going to be useful to most companies.

hogFeast 6y ago
Just an anecdote but if you go to most baseball data departments, where there is real competition between teams, you don't just have PHds. You have people with undergrads/domain knowledge, and people with PHds. This isn't to say that PHd knowledge isn't valuable but if you look at firms in finance that have had success with data i.e. RenTech, they hire very smart people with PHds but it isn't only the PHd. You need someone who has the knowledge AND someone who has common sense/can get results. That is very hard to do correctly (and yes, some people who come from academia literally do not want anything to do with business...it is like the devs who come from a CS PHd and insist on using complicated algo and data structure everywhere, optimising every line, etc.).
- mkl 6y ago
  *PhD or Ph.D. It's an abbreviation of the Latin "philosophiae doctor" == "doctor of philosophy". Getting one is as much about persistence as intelligence, and makes you very knowledgeable about one narrow area, so what you say makes sense. Academic researchers usually branch out into other fields and subfields as well, but straight out of a PhD, narrow and deep is what you tend to get.

tachyonbeam 6y ago

I worked in a place full of deep learning PhDs, and you'd have people trying to apply reinforcement learning to problems that had known mathematical solutions, and integer programming problems. I don't think the issue is just that companies hire people who are awful at ML, it's also that people are trying to shoehorn deep learning into everything, even when it currently has nothing to offer and we have better solutions already. IMHO, we're producing too many deep learning PhDs.

hogFeast 6y ago
This is just my general sense, as a very non-expert with more experience of doing than theory...but the benefit is someone knowing the theory AND being able to translate that into revenue. I think most people view the hard part as doing the PHd, and so lots of people value that experience, and because they have that experience you have this endowment effect: wow, that PHd was hard, I must do very hard and complex things. To give you an example: Man Group. They are a huge quant hedge fund, in fact they were one of the first big quant funds. Now, they even have their own program at Oxford University that they hire out of...have you heard of them? Most people haven't. Their performance is mostly terrible, and despite being decades ahead of everyone their returns were never very good (they did well at the start because they had a few exceptional employees, who then went elsewhere...David Harding was one). The issue isn't PHds, they have many of them, the issue is having that knowledge AND being able to convert it. I think this is really hard to grasp because most people expect problems to yield instantly to ML but, in most cases, they don't and other people have done valuable work with non-ML stuff that should be built on but isn't because domain knowledge or common sense is often lacking. A similar thing is people who come out of CS, and don't know how to program. They know a bit but they don't know how to use Git, they don't know how to write code others can read, etc.
- smabie 6y ago
  The Man Group has had respectable returns, especially during Coronavirus. Nothing amazing, but certainly not terrible. Regardless, there's more to the picture: Sharpe ratio, vol, correlation to the market, etc

mumblemumble 6y ago

My sense is that the original sin here is conflating data science with machine learning. A good data scientist might choose to use machine learning to accomplish their job. Or they might find that classical statistical inference is the better tool for the task at hand. A good data scientist, having built this model, might choose to put it into production. Or they might find that a simple if-statement could do the job almost as effectively but not nearly as expensively. A good data scientist, having decided to productionize a model, will also provide some information about how it might break down - for example, describing shifts in customer behavior, or changes in how some input signal is generated, or feedback effects that might invalidate the model. OTOH, if your job has been framed in terms of cutting-edge machine learning, then you may well know - at a gut level, if not consciously - that your job is basically just a pissing match to see who can deploy the most bleeding-edge or expensive technology the fastest. It's like the modern hospital childbirth scene in Monty Python's The Meaning of Life, where the doctor is more interested in showing off the machine that goes, "ping!" in order to impress the other doctors than he is in paying attention to the mother.

aspaceman 6y ago
There’s people who consider classical inference and the like to be machine learning just as much as neural nets are. I like that perspective.
- mumblemumble 6y ago
  There are some things, like OLS and logistic regression, that are commonly used for both purposes. But there's a sort of moral distinction between machine learning and statistical inference, driven by whether you consider your key deliverable to be y-hat or beta-hat, that ends up having implications. For example, I can get pretty preoccupied with multicollinearity or heteroskedasticity when I'm wearing my statistician hat, while they barely qualify as passing diversions when I'm wearing my machine learning engineer hat. If I'm doing ML, I'll happily deliberately bias the model. That would be anathema if I were doing statistical inference.
  - 6y ago

visarga 6y ago

It's just a process of exploration, people trying out ideas to see what works. Over time, with sharing of results, we will gradually discover more nuanced approaches, but the exploration phase is necessary in order to map a path forward and train a generation of ML engineers and PM's who don't have seniors to learn from. Of course it sucks on the short term, but there is zero chance the field will be abandoned. It has enough uses already.

ric2b 6y ago
But the exploration should be led by the data scientists, not by the suits who think they need a specific type of model.

emmap21 6y ago

It sounds so painful as someone all-in this area. But I have to agree on a task of overdoing with fancy models. Nevertheless, the most common ML algo in industry is still linear regression along with boostraping.