11 ms·

I've worked in lots of big corps as a consultant. Every one raced to harness the power of "big data" ~7 years ago. They couldn't hire or spend money fast enou

by eric_b 6y ago

I've worked in lots of big corps as a consultant. Every one raced to harness the power of "big data" ~7 years ago. They couldn't hire or spend money fast enough. And for their investment they (mostly) got nothing. The few that managed to bludgeon their map/reduce clusters in to submission and get actionable insights discovered... they paid more to get those insights than they were worth! I think this same thing is happening with ML. It was a hiring bonanza. Every big corp wanted to get an ML/AI strategy in place. They were forcing ML in to places it didn't (and may never) belong. This "recession" is mostly COVID related I think - but companies will discover that ML is (for the vast majority) a shiny object with no discernible ROI. Like Big Data, I think we'll see a few companies execute well and actually get some value, while most will just jump to the next shiny thing in a year or two.

plants 6y ago
This is sadly so consistent with what I'm seeing at a big corporation. We are working so hard to make a centralized ML platform, get our data up to par, etc. but so many ML projects either have no chance of succeeding or have so little business value that they're not worth pursuing. Everyone on the development team for the project I'm working on is silently in agreement that our model would be better off being replaced by a well-managed rules engine, but every time we bring up these concerns, they're effectively disregarded. There are obviously places in my company where ML is making an enormous impact, it's just not something that's fit for every single place where decisions need to be made. Sometimes doing some analysis to inform blunt rules works just as well - without the overhead of ML model management.
- CuriouslyC 6y ago
  Being mostly disconnected from the fruits of your labor while being incentivized to turn your resume into buzzword bingo causes bad technology choices that hurt the organization, what a surprise.
- hectormalot 6y ago
  > Everyone on the development team for the project I'm working on is silently in agreement that our model would be better off being replaced by a well-managed rules engine That was one of the better insights with our team. We should measure the value-add of ML against a baseline that is e.g. a simple rules engine, not against 0. In some cases that looked appealing (‘lots of value by predicting Y better’) it turned out that a simple Excel sort would get us 90-98% of the value starting tomorrow. Investing an ML team for a few weeks/months then only makes sense if the business case on getting from 95% to 98% is big enough in itself. Hint: in many cases it isn’t.
- Balgair 6y ago
  > or have so little business value that they're not worth pursuing It seems that I'm inverted from you. The Machine part of Machine Learning is likely of high business value, but the Learning part is the easier and better solution. We do a lot of hardware stuff and our customers are, well let's just say they could use some re-training. Think not putting ink in the printer and then complaining about it. Only much more expensive. Because the details get murky (and legal-y and regulation-y) very quickly, we're forced to do ML on the products to 'assist' our users [0]. But in the end, the easiest solution is to have better users. [0] Yes, UX, training, education, etc. We've tried, spent a lot of money on it. It doesn't help.
- lumost 6y ago
  I think part of the problem here is that ML development is extraordinarily more expensive then traditional dev. I don't generally need to develop my own deployment infrastructure for every new project. However I've yet to see an ml team or company consistently use the same toolchain between 2 projects. The same pattern repeats across data processing, model development, and inference. Oddly, adding more scientists appears to have a super-linear increase in cost - with the net effect being either duplicated effort or exhaustive search across possible solutions.
apohn 6y ago
"Like Big Data, I think we'll see a few companies execute well and actually get some value, while most will just jump to the next shiny thing in a year or two." Here's another aspect - in many places nobody listens to the actual people doing the work. In my last job I was hired to lead a Data Science team and to help the company get value of Stats/ML/AI/DL/Buzzword. And I (and my team) were promptly overridden on every decision of what projects an expectations were realistic and what were not. I left, as did everybody else that reported to me, and we were replaced by people who would make really good BS slides that showed what upper management wanted to see. A year after that the whole initiative was cancelled. Back in 2000 I was in a similar position with a small company jumping on the internet as their next business model. Lots of nonsense and one horrible web based business later, the company failed. It's the same story over and over again. Some winners, lot of losers, many by self-inflicted wounds.
- mrtksn 6y ago
  If you think about it, that's the natural outcome. Why? Because people in corporations don't have the incentive to benefit the business but to progress their careers and that's done through meeting the goals for their position and make their upper ups progress with their careers too. So essentially, you have a system where people spend other people's resources for living and their success is judged by making the chain link above happy. In especially large companies it's easy to have a disconnect from the product because people in the top specialise in topics that have nothing to do with the product. If the people at the top want to have this shiny new thing that the press and everyone else is saying that it's the next big thing, you better give them the new shiny thing if you want to have a smooth career. In publicly traded companies, this is even more prevalent because people who buy and sell the stocks would be even more disconnected from the product and tied to the buzzwords. The more technical minded people who have the hunch on tech miss the point of the organisation that they are in and get very frustrated. It's probably the reason why startups can be much more fulfilling for deeply technical people.
  - apohn 6y ago
    >If you think about it, that's the natural outcome. Why? Because people in corporations don't have the incentive to benefit the business but to progress their careers and that's done through meeting the goals for their position and make their upper ups progress with their careers too. This is one of the reasons I roll my eyes whenever I read something like "McKinsey says 75% of Big Data/AI/Buzzword projects do not deliver any value." What's the baseline for failing and/or delivering zero value because those projects were destined to fail?
twelfthnight 6y ago
I've seen similar patterns with clients and companies I've worked at as well. My experience was less that ML wasn't useful, it's just that no organization I worked with could really break down the silos in order for it to work. Especially in ML, the entire process from data collection to the final product and feedback loop needs to be integrated. This is _really_ difficult for most companies. Many data scientists I knew were either sitting on their hands waiting for data or working on problems that the downstream teams had no intention of implementing (even if they were improvements). I still really believe that ML (be it fancy deep learning or just evidence driven rules-based models) will effectively be table stakes for most industries in the upcoming decade. However, it'll take more leadership than just hiring a bunch of smart folks out of a PhD program.
PragmaticPulp 6y ago
Ironically, I worked on a product that had a classic use case for machine learning during this time period and still had great difficulty getting results. It was difficult to attract top ML talent no matter how much we offered. Everyone wanted to work for one of the big, recognizable names in the industry for the resume name recognition and a chance to pivot their way into a top role at a leading company later. Meanwhile, we were flooded with applicants who exaggerated their ML knowledge and experience to an extreme, hoping to land high paying ML jobs through hiring managers who couldn’t understand what they were looking for. It was easy to spot most of these candidates after going through some ML courses online and creating a very basic interview problem, but I could see many of these candidates successfully getting ML jobs at companies that didn’t know any better. Maybe they were going to fake it until they made it, or maybe they were counting on ML job performance being notoriously difficult to quantify on big data sets. Dealing with 3rd party vendors and consulting shops wasn’t much better. A lot of the bigger shops were too busy with never ending lucrative contracts to take on new work. A lot of the smaller shops were too new to be able to show us much of a track record. Their proposals often boiled down to just implementing some famous open source solution on our product and letting us handle the training. Thanks, but we can do that ourselves. I get the impression that it is (or was) more lucrative to start your own ML company and hope for an acquisition than to do the work for other companies. We tried to engage with several small ML vendors in our space and more than half of them came back with suggestions that we simply acquire them for large sums of money. Meanwhile, one of the vendors we engaged with was acquired by someone else and, of course, their support dried up completely. Ultimately we found a solution from a vendor that had prepared a nice solution for our exact problem.the contracts were drawn up in a way that wouldn’t be too disastrous if (when?) they were acquired. I have to wonder if an industry-wide slowdown to the ML frenzy is exactly what we need to give people and companies time to focus on solving real problems instead of just chasing easy money.
- bluetwo 6y ago
  I find your post kind of interesting. I develop software in a non-AI field and have been following and experimenting with AI on the side for a long time. Academics seem intent on publishing papers, not finding solutions to creating value. Corporate AI seems focused on sizzle not substance. It is so frustrating to see the potential in the AI world and realize almost no one is really interested in building it.
baron_harkonnen 6y ago
> they paid more to get those insights than they were worth! This understates how awful ML is at many of these companies. I've seen quite a few companies that rushed to hire teams of people with a PhD in anything that barely made it through a DS/ML boot camp. To prove that they're super smart ML researchers without fail these hires rush to deploy a 3+ layer MLP to solve a problem that need at most a simple regression. They have no understanding of how this model works, and have zero engineering sense so they don't care if it's a nightmare of complexity to maintain. Then to make sure their work is 'valuable' management tries to get as many teams as possible to make use of the questionable outputs of these models. The end is a nightmare of tightly coupled models that nobody can debug, trouble shoot or understand. And because the people building them don't really understand how they work the results are always very noisy. So you end up with this mess of expensive to build and run models talking noise to each other. When I saw this I realized data science was doomed in the next recession, since the only solution to this mess is to just remove it all. There is some really valuable DS work out there, but it requires real understanding of either modeling or statistics. That work will probably stick around, but these giant farms of boot camp grads churning out keras models will disappear soon.
- disgruntledphd2 6y ago
  And this is a good thing! To be fair, I started to understand why developers gave out about bootcamp grads lacking a foundation when the bootcamps came for my discipline (data science). The PhD fetish is pretty mental (even though I have one), as it's really not necessary. Additionally, everyone thinks they need researchers, when they really, really don't. Having worked with researchy vs more product/business driven teams, I found that the best results came when a researchy person took the time to understand the product domain, but many of them believe they're too good for business (in which case you should head back to academia). What you actually need from an ML/Data Science person: - Experience with data cleaning (this is most of the gig) - A solid understanding of linear and logistic regression, along with cross-validation - Some reasonable coding skills (in both R and Python, with a side of SQL). That's it. Pretty much everything else can be taught, given the above prerequisites. But it's tricky for hiring managers/companies as they don't know who to hire, so they end up over-indexing on bullshitters, due to the confidence, leading to lots of nonsese. And finally, deep learning is good in some scenarios and not in others, so anyone who's just a deep learning developer is not going to be useful to most companies.
blaird 6y ago
Curious if there is a correlation with companies that failed to capitalize with the ones who relied on consultants versus really reshaping their own people. I worked for a financial services co that saw massive gains from big data/ML/AWS. Given, we were already using statistical models for everything, we just now could build more powerful features, more complex models, and move many things to more-real time, with more frequent retrains/deploys bc of cloud. I do agree that companies who don't already recognize the value of their data and maybe rely on a consultant to tell them what to do might not be in the position to really capitalize on it and would just be throwing money after the shiny object. It really does take a huge overhaul sometimes. We retooled all of our job families from analysts/statisticians to data engineers and scientists and hired a ton of new people
- apohn 6y ago
  >Curious if there is a correlation with companies that failed to capitalize with the ones who relied on consultants versus really reshaping their own people. I've worked in Data Science customers facing roles for 2 companies, and one anecdotal correlation between success with Stats/ML/AI I've seen is how "Data Driven" people really are for their daily decision making. The more data driven you are, the more likely you are to identify a problem that can actually be improved by an Stat/ML/AI algorithm. This is because you really understand your data and the value you can get from it. Everybody has metrics, KPIs, OKS, etc, but the reality is that there's a spectrum from 100% gut to 100% data driven. And a lot of people are on the gut side of things while thinking (or claiming they are) they are on the data side. I'll provide an example. I currently work for a company that sells to (among others) companies working with industrial machinery. If your industrial machine runs in a remote area (e.g. an Oil Field), then any question about that machine starts with pulling up data. Being data driven is the only way to figure out what's going on. These folks have a good sense for identifying the value they can get from their data and they usually understand when you say dealing with their data is a engineering task in itself. The other side of this is a factory filled with people. Since somebody is always operating and watching the machine, the "data driven" part is mainly alarms (e.g. is my temp over 100C) and some external KPI (e.g. a quality measurement). They are much less data driven than they think they are, and a lot of them don't understand what value they could get out of their data beyond some simple stuff you don't really need ML/AI for. I mention industrial equipment because I think a lot of people (even me) are really surprised when they hear about people working in factories not being super data driven. You think of factories, engineering, and data as being very lumped together. It's amazing how many areas (sales, marketing, HR, are other great examples) exist where people aren't as data driven as they think they are.
cashsterling 6y ago
I also witnesses this first hand at a Biotech company I worked at... we were using many variants of machine learning algorithms to develop predictive models of cell culture and separation processes. Problem is... the models have so many parameters in order to get a useful fit that the same model can also fit a carrot or an elephant. We found that dynamic parameter estimation on ODE/DAE/PDE system models, while harder to develop, actually worked much better and gave us real insight into the processes. So now my advice is others is "if you can start with some first principles equation or system of equations... start there and use optimization/regression to fit the model to the data." AND: "if you don't think such equations exist for your problem... read/research more, because some useful equations probably do exist." This is usually pretty straightforward for engineering and science applications... equations exist or can be derived for the system under study. In my very limited exposure to other areas of machine learning application... I have found quite a bit of mathematical science related to marketing, human behavior, etc.
- x86_64Ubuntu 6y ago
  Kind of weird that they would use ML/AI for a separations process. Separations and chemical engineering in general absolutely LOVES parameters and systems of equations. And don't go anywhere near colloids, those have so many empirically sourced parameters it will make your head spin.
  - xkcd-sucks 6y ago
    You do this when you're a vendor working for big pharma and the people there can't/won't give you/don't understand the relevant quantities and aren't even familiar with standard models of the process despite their being formally trained chemical engineers and the models being 50 years old. Speaking from direct experience with several companies that are trying to bring us covid vaccines
- alephu5 6y ago
  I completely agree with this sentiment, I've seen a lot of people throw ML at problems because they don't know much mathematics. Especially when you have a lot of data, I can understand the allure of just wiring up the input & output to generate the model.
insomniacity 6y ago
My employer is big enough that I know we're doing a bunch of ML/AI and probably getting some value out of it somewhere. However someone is trying to make robotic process automation the Next Big Thing - which I think is hysterically funny.
OscarTheGrinch 6y ago
Yeah the big data comparison is apt, and a few years ago was The Block-Chain that got middle managers frothing like Pavlov's dog. It is clear that for most of the companies who are investing in deep learning are tangible results are always around the corner, and maybe 1 in 100 will build something worthwhile. But here is the carrot driving them all on, it's like the lottery: you have to be in to win. The stick is the fear that their competitors will do so. This field is more art than science, give talented people incentive to play and don't expect too much for the next decade.
ellisv 6y ago
> they paid more to get those insights than they were worth! > They were forcing ML in to places it didn't (and may never) belong. I find that I spend a lot of time as a senior MLE telling someone why they don’t need ML
monksy 6y ago
> big data That's because it didn't get a chance to mature and to show how it could be powerful. People kept trying to force hadoop into it and call themselves "big data experts" We've gotten a bit more clarity in this world with streaming technologies. However, there hasn't been a good and clear voice to say "hey .. this is how it fits in with your web app and this is what you expect of it". (I'm thinking about developing a talk on this.. how it fits in [hint.. your microservice app shouldn't do any heavy lifting of processing data])
- synthc 6y ago
  These days it's people trying to force Kafka into it and call themselves "streaming experts"
  - monksy 6y ago
    Kafka is good.. but it requires a lot of good work to get it working well for a pipeline.
- lumost 6y ago
  Most buzzwords exist for consultants to sell their services. Successful buzzwords turn engineers into "Big Data Engineers" who only want to work on something called big data, and convince management that they need more "big data expertise". In practice most of these technologies and their peers exist to support real applications, and it would be almost immediately recognizable that they are the appropriate choice when working on a similar application. You don't need a streaming engineer/big data consultant to cram them in.
bishalb 6y ago
Or like conversion rate optimization tools.
Abishek_Muthian 6y ago
Could it be also because for most companies after large investment in DS/ML/DL, they couldn't create a promising solution because they don't have as much access to the data/hardware/talent as Google/Amazon/MS does? And at the end of the day using just an API from the former gives better ROI? (or) In simple terms, is profitable commercial Deep Learning just for oligarchies?
visarga 6y ago
I don't agree, most of the low hanging fruit in ML engineering hasn't been picked yet. ML is like electricity 100 years ago, it will only expand and eat the world. And the research is not slowing down, on the contrary, it advances by leaps and bounds. The problem is that we don't have enough ML engineers and many who go by this title are not really capable of doing the job. We're just coming into decent tools and hardware, and many applications are still limited by hardware which itself is being reinvented every 2 years. Take just one single subfield - CV - it has applications in manufacturing, health, education, commerce, photography, agriculture, robotics, assisting blind persons, ... basically everywhere. It empowers new projects and amplifies automation. With the advent of pre-trained neural nets every new task can be 10x or 100x easier. We don't need as many labels anymore, it works much better now.
toomanybeersies 6y ago
That happened/is happening at my job. There's been a push to implement features that utilise AI/ML. Not because it would be a good use case (although there are some for our product), or because it would be of any practical benefit, but because it makes for good marketing copy. Never mind the fact that nobody on the team has any experience with machine learning (I actually failed the paper at university).
erichocean 6y ago
Without ML, our business today is literally impossible (from a financial perspective). I work in 2D animation and we were able to design our current pipeline around adopting ML at specific steps to remove massive amounts of manual labor. I know this doesn't disprove your anecdote, I just wanted to point out that real businesses are using ML effectively to deliver real value that's not possible without it.
Accujack 6y ago
This has happened since the dawn of the computer age, and probably before. Any technology too complex for the managers who purchase for it to understand fully can be sold and oversold by marketing people as "the next big thing". Managers may or may not see through that, but if their superiors want them to pursue it or if they need to pursue something in order to show they're doing something of value, then they're happy to follow where the marketers lead. Java everywhere, set top TV boxes, IOT devices, transitioning mainframes to minis, you name it... the marketers have made a mint selling it, usually for little benefit to the companies that bought into it.
tarsinge 6y ago
The problem I see is that in most non tech businesses they are not at the stage where they need ML, they are simply struggling with the basics: being able to seamlessly query or have consolidated up to date metrics and dashboards of the data scattered in all their databases. Of course the Big Data/AI “we’ll transform your data into insights” appealed to them, but that’s not what they need (also see the comments on the Palantir thread the other day).
jacobsenscott 6y ago
People have been trying to used algorithms of various sorts to increase sales (actionable insights) forever. The buzzwords change, but the results are always the same. No permutation of CPU instructions will turn a product people don't want to pay for into a product people want to pay for.
at-fates-hands 6y ago
The company I work for is a large health care company. I started in robotic automation about a year ago. The company said its next three huge initiatives would be: 1) AI 2) Machine Learning 3) Robotic Process Automation They felt RPA would help them stay more competitive since there are tons of smaller health care companies who are moving faster and innovating faster because they're not buying up companies and having to integrate all their technology at a sloth's pace. They thought RPA would be a way to mitigate these issues. 18 months later and the one manager, director and VP in my org has all but said they don't care about RPA, all their money is going into ML and AI. Even though in all the presentations I've seen them put on, its all blue skies and BS about "IF we had this, we COULD do this." Nothing concrete at all about how the plan to use ML to increase profit margins or reduce overhead. Right now, our team is basically an afterthought in the company and I'm already starting to interview elsewhere with the knowledge at some point, they're going to kill my team and cut everybody loose.
deleted 6y ago
[deleted]
Lxr 6y ago
ML is a shiny object with often no discernible ROI but occasionally very large ROI, and companies are understandably nervous about missing out. Spending a small amount to hedge their bets isn't necessarily irrational.
cnst 6y ago
There's also a lot of deception going on. The easiest way to solve many problems is through lexers, regular expressions and plain-old pattern matching. But that doesn't sell, so, they call it AI anyways.
RandoHolmes 6y ago
When I hear ML, deep learning, etc, I consider it a red flag for exactly the reasons you state. It's kind of batty actually, people looking for ideas to make money just been taking old ideas and attaching ML to the side of it as if that automatically made it better. And then not educating their customers on the limitations of ML both generally and with respect to their data size. I personally think the companies that make and sell the software that the police used to make incorrect arrests should be legally liable. Yes, the police shouldn't have blindly trusted the software, but I guaran-fucking-tee you part of why they did is the marketing from the company themselves.

by eric_b 6y ago

plants 6y ago

This is sadly so consistent with what I'm seeing at a big corporation. We are working so hard to make a centralized ML platform, get our data up to par, etc. but so many ML projects either have no chance of succeeding or have so little business value that they're not worth pursuing. Everyone on the development team for the project I'm working on is silently in agreement that our model would be better off being replaced by a well-managed rules engine, but every time we bring up these concerns, they're effectively disregarded. There are obviously places in my company where ML is making an enormous impact, it's just not something that's fit for every single place where decisions need to be made. Sometimes doing some analysis to inform blunt rules works just as well - without the overhead of ML model management.

CuriouslyC 6y ago
Being mostly disconnected from the fruits of your labor while being incentivized to turn your resume into buzzword bingo causes bad technology choices that hurt the organization, what a surprise.
hectormalot 6y ago
> Everyone on the development team for the project I'm working on is silently in agreement that our model would be better off being replaced by a well-managed rules engine That was one of the better insights with our team. We should measure the value-add of ML against a baseline that is e.g. a simple rules engine, not against 0. In some cases that looked appealing (‘lots of value by predicting Y better’) it turned out that a simple Excel sort would get us 90-98% of the value starting tomorrow. Investing an ML team for a few weeks/months then only makes sense if the business case on getting from 95% to 98% is big enough in itself. Hint: in many cases it isn’t.
Balgair 6y ago
> or have so little business value that they're not worth pursuing It seems that I'm inverted from you. The Machine part of Machine Learning is likely of high business value, but the Learning part is the easier and better solution. We do a lot of hardware stuff and our customers are, well let's just say they could use some re-training. Think not putting ink in the printer and then complaining about it. Only much more expensive. Because the details get murky (and legal-y and regulation-y) very quickly, we're forced to do ML on the products to 'assist' our users [0]. But in the end, the easiest solution is to have better users. [0] Yes, UX, training, education, etc. We've tried, spent a lot of money on it. It doesn't help.
lumost 6y ago
I think part of the problem here is that ML development is extraordinarily more expensive then traditional dev. I don't generally need to develop my own deployment infrastructure for every new project. However I've yet to see an ml team or company consistently use the same toolchain between 2 projects. The same pattern repeats across data processing, model development, and inference. Oddly, adding more scientists appears to have a super-linear increase in cost - with the net effect being either duplicated effort or exhaustive search across possible solutions.

apohn 6y ago

"Like Big Data, I think we'll see a few companies execute well and actually get some value, while most will just jump to the next shiny thing in a year or two." Here's another aspect - in many places nobody listens to the actual people doing the work. In my last job I was hired to lead a Data Science team and to help the company get value of Stats/ML/AI/DL/Buzzword. And I (and my team) were promptly overridden on every decision of what projects an expectations were realistic and what were not. I left, as did everybody else that reported to me, and we were replaced by people who would make really good BS slides that showed what upper management wanted to see. A year after that the whole initiative was cancelled. Back in 2000 I was in a similar position with a small company jumping on the internet as their next business model. Lots of nonsense and one horrible web based business later, the company failed. It's the same story over and over again. Some winners, lot of losers, many by self-inflicted wounds.

mrtksn 6y ago
If you think about it, that's the natural outcome. Why? Because people in corporations don't have the incentive to benefit the business but to progress their careers and that's done through meeting the goals for their position and make their upper ups progress with their careers too. So essentially, you have a system where people spend other people's resources for living and their success is judged by making the chain link above happy. In especially large companies it's easy to have a disconnect from the product because people in the top specialise in topics that have nothing to do with the product. If the people at the top want to have this shiny new thing that the press and everyone else is saying that it's the next big thing, you better give them the new shiny thing if you want to have a smooth career. In publicly traded companies, this is even more prevalent because people who buy and sell the stocks would be even more disconnected from the product and tied to the buzzwords. The more technical minded people who have the hunch on tech miss the point of the organisation that they are in and get very frustrated. It's probably the reason why startups can be much more fulfilling for deeply technical people.
- apohn 6y ago
  >If you think about it, that's the natural outcome. Why? Because people in corporations don't have the incentive to benefit the business but to progress their careers and that's done through meeting the goals for their position and make their upper ups progress with their careers too. This is one of the reasons I roll my eyes whenever I read something like "McKinsey says 75% of Big Data/AI/Buzzword projects do not deliver any value." What's the baseline for failing and/or delivering zero value because those projects were destined to fail?

twelfthnight 6y ago

I've seen similar patterns with clients and companies I've worked at as well. My experience was less that ML wasn't useful, it's just that no organization I worked with could really break down the silos in order for it to work. Especially in ML, the entire process from data collection to the final product and feedback loop needs to be integrated. This is _really_ difficult for most companies. Many data scientists I knew were either sitting on their hands waiting for data or working on problems that the downstream teams had no intention of implementing (even if they were improvements). I still really believe that ML (be it fancy deep learning or just evidence driven rules-based models) will effectively be table stakes for most industries in the upcoming decade. However, it'll take more leadership than just hiring a bunch of smart folks out of a PhD program.

PragmaticPulp 6y ago

Ironically, I worked on a product that had a classic use case for machine learning during this time period and still had great difficulty getting results. It was difficult to attract top ML talent no matter how much we offered. Everyone wanted to work for one of the big, recognizable names in the industry for the resume name recognition and a chance to pivot their way into a top role at a leading company later. Meanwhile, we were flooded with applicants who exaggerated their ML knowledge and experience to an extreme, hoping to land high paying ML jobs through hiring managers who couldn’t understand what they were looking for. It was easy to spot most of these candidates after going through some ML courses online and creating a very basic interview problem, but I could see many of these candidates successfully getting ML jobs at companies that didn’t know any better. Maybe they were going to fake it until they made it, or maybe they were counting on ML job performance being notoriously difficult to quantify on big data sets. Dealing with 3rd party vendors and consulting shops wasn’t much better. A lot of the bigger shops were too busy with never ending lucrative contracts to take on new work. A lot of the smaller shops were too new to be able to show us much of a track record. Their proposals often boiled down to just implementing some famous open source solution on our product and letting us handle the training. Thanks, but we can do that ourselves. I get the impression that it is (or was) more lucrative to start your own ML company and hope for an acquisition than to do the work for other companies. We tried to engage with several small ML vendors in our space and more than half of them came back with suggestions that we simply acquire them for large sums of money. Meanwhile, one of the vendors we engaged with was acquired by someone else and, of course, their support dried up completely. Ultimately we found a solution from a vendor that had prepared a nice solution for our exact problem.the contracts were drawn up in a way that wouldn’t be too disastrous if (when?) they were acquired. I have to wonder if an industry-wide slowdown to the ML frenzy is exactly what we need to give people and companies time to focus on solving real problems instead of just chasing easy money.

bluetwo 6y ago
I find your post kind of interesting. I develop software in a non-AI field and have been following and experimenting with AI on the side for a long time. Academics seem intent on publishing papers, not finding solutions to creating value. Corporate AI seems focused on sizzle not substance. It is so frustrating to see the potential in the AI world and realize almost no one is really interested in building it.

baron_harkonnen 6y ago

> they paid more to get those insights than they were worth! This understates how awful ML is at many of these companies. I've seen quite a few companies that rushed to hire teams of people with a PhD in anything that barely made it through a DS/ML boot camp. To prove that they're super smart ML researchers without fail these hires rush to deploy a 3+ layer MLP to solve a problem that need at most a simple regression. They have no understanding of how this model works, and have zero engineering sense so they don't care if it's a nightmare of complexity to maintain. Then to make sure their work is 'valuable' management tries to get as many teams as possible to make use of the questionable outputs of these models. The end is a nightmare of tightly coupled models that nobody can debug, trouble shoot or understand. And because the people building them don't really understand how they work the results are always very noisy. So you end up with this mess of expensive to build and run models talking noise to each other. When I saw this I realized data science was doomed in the next recession, since the only solution to this mess is to just remove it all. There is some really valuable DS work out there, but it requires real understanding of either modeling or statistics. That work will probably stick around, but these giant farms of boot camp grads churning out keras models will disappear soon.

disgruntledphd2 6y ago
And this is a good thing! To be fair, I started to understand why developers gave out about bootcamp grads lacking a foundation when the bootcamps came for my discipline (data science). The PhD fetish is pretty mental (even though I have one), as it's really not necessary. Additionally, everyone thinks they need researchers, when they really, really don't. Having worked with researchy vs more product/business driven teams, I found that the best results came when a researchy person took the time to understand the product domain, but many of them believe they're too good for business (in which case you should head back to academia). What you actually need from an ML/Data Science person: - Experience with data cleaning (this is most of the gig) - A solid understanding of linear and logistic regression, along with cross-validation - Some reasonable coding skills (in both R and Python, with a side of SQL). That's it. Pretty much everything else can be taught, given the above prerequisites. But it's tricky for hiring managers/companies as they don't know who to hire, so they end up over-indexing on bullshitters, due to the confidence, leading to lots of nonsese. And finally, deep learning is good in some scenarios and not in others, so anyone who's just a deep learning developer is not going to be useful to most companies.

blaird 6y ago

Curious if there is a correlation with companies that failed to capitalize with the ones who relied on consultants versus really reshaping their own people. I worked for a financial services co that saw massive gains from big data/ML/AWS. Given, we were already using statistical models for everything, we just now could build more powerful features, more complex models, and move many things to more-real time, with more frequent retrains/deploys bc of cloud. I do agree that companies who don't already recognize the value of their data and maybe rely on a consultant to tell them what to do might not be in the position to really capitalize on it and would just be throwing money after the shiny object. It really does take a huge overhaul sometimes. We retooled all of our job families from analysts/statisticians to data engineers and scientists and hired a ton of new people

apohn 6y ago
>Curious if there is a correlation with companies that failed to capitalize with the ones who relied on consultants versus really reshaping their own people. I've worked in Data Science customers facing roles for 2 companies, and one anecdotal correlation between success with Stats/ML/AI I've seen is how "Data Driven" people really are for their daily decision making. The more data driven you are, the more likely you are to identify a problem that can actually be improved by an Stat/ML/AI algorithm. This is because you really understand your data and the value you can get from it. Everybody has metrics, KPIs, OKS, etc, but the reality is that there's a spectrum from 100% gut to 100% data driven. And a lot of people are on the gut side of things while thinking (or claiming they are) they are on the data side. I'll provide an example. I currently work for a company that sells to (among others) companies working with industrial machinery. If your industrial machine runs in a remote area (e.g. an Oil Field), then any question about that machine starts with pulling up data. Being data driven is the only way to figure out what's going on. These folks have a good sense for identifying the value they can get from their data and they usually understand when you say dealing with their data is a engineering task in itself. The other side of this is a factory filled with people. Since somebody is always operating and watching the machine, the "data driven" part is mainly alarms (e.g. is my temp over 100C) and some external KPI (e.g. a quality measurement). They are much less data driven than they think they are, and a lot of them don't understand what value they could get out of their data beyond some simple stuff you don't really need ML/AI for. I mention industrial equipment because I think a lot of people (even me) are really surprised when they hear about people working in factories not being super data driven. You think of factories, engineering, and data as being very lumped together. It's amazing how many areas (sales, marketing, HR, are other great examples) exist where people aren't as data driven as they think they are.

cashsterling 6y ago

I also witnesses this first hand at a Biotech company I worked at... we were using many variants of machine learning algorithms to develop predictive models of cell culture and separation processes. Problem is... the models have so many parameters in order to get a useful fit that the same model can also fit a carrot or an elephant. We found that dynamic parameter estimation on ODE/DAE/PDE system models, while harder to develop, actually worked much better and gave us real insight into the processes. So now my advice is others is "if you can start with some first principles equation or system of equations... start there and use optimization/regression to fit the model to the data." AND: "if you don't think such equations exist for your problem... read/research more, because some useful equations probably do exist." This is usually pretty straightforward for engineering and science applications... equations exist or can be derived for the system under study. In my very limited exposure to other areas of machine learning application... I have found quite a bit of mathematical science related to marketing, human behavior, etc.

x86_64Ubuntu 6y ago
Kind of weird that they would use ML/AI for a separations process. Separations and chemical engineering in general absolutely LOVES parameters and systems of equations. And don't go anywhere near colloids, those have so many empirically sourced parameters it will make your head spin.
- xkcd-sucks 6y ago
  You do this when you're a vendor working for big pharma and the people there can't/won't give you/don't understand the relevant quantities and aren't even familiar with standard models of the process despite their being formally trained chemical engineers and the models being 50 years old. Speaking from direct experience with several companies that are trying to bring us covid vaccines
alephu5 6y ago
I completely agree with this sentiment, I've seen a lot of people throw ML at problems because they don't know much mathematics. Especially when you have a lot of data, I can understand the allure of just wiring up the input & output to generate the model.

insomniacity 6y ago

My employer is big enough that I know we're doing a bunch of ML/AI and probably getting some value out of it somewhere. However someone is trying to make robotic process automation the Next Big Thing - which I think is hysterically funny.

OscarTheGrinch 6y ago

Yeah the big data comparison is apt, and a few years ago was The Block-Chain that got middle managers frothing like Pavlov's dog. It is clear that for most of the companies who are investing in deep learning are tangible results are always around the corner, and maybe 1 in 100 will build something worthwhile. But here is the carrot driving them all on, it's like the lottery: you have to be in to win. The stick is the fear that their competitors will do so. This field is more art than science, give talented people incentive to play and don't expect too much for the next decade.

ellisv 6y ago

> they paid more to get those insights than they were worth! > They were forcing ML in to places it didn't (and may never) belong. I find that I spend a lot of time as a senior MLE telling someone why they don’t need ML

monksy 6y ago

> big data That's because it didn't get a chance to mature and to show how it could be powerful. People kept trying to force hadoop into it and call themselves "big data experts" We've gotten a bit more clarity in this world with streaming technologies. However, there hasn't been a good and clear voice to say "hey .. this is how it fits in with your web app and this is what you expect of it". (I'm thinking about developing a talk on this.. how it fits in [hint.. your microservice app shouldn't do any heavy lifting of processing data])

synthc 6y ago
These days it's people trying to force Kafka into it and call themselves "streaming experts"
- monksy 6y ago
  Kafka is good.. but it requires a lot of good work to get it working well for a pipeline.
lumost 6y ago
Most buzzwords exist for consultants to sell their services. Successful buzzwords turn engineers into "Big Data Engineers" who only want to work on something called big data, and convince management that they need more "big data expertise". In practice most of these technologies and their peers exist to support real applications, and it would be almost immediately recognizable that they are the appropriate choice when working on a similar application. You don't need a streaming engineer/big data consultant to cram them in.

bishalb 6y ago

Or like conversion rate optimization tools.

Abishek_Muthian 6y ago

Could it be also because for most companies after large investment in DS/ML/DL, they couldn't create a promising solution because they don't have as much access to the data/hardware/talent as Google/Amazon/MS does? And at the end of the day using just an API from the former gives better ROI? (or) In simple terms, is profitable commercial Deep Learning just for oligarchies?

visarga 6y ago

I don't agree, most of the low hanging fruit in ML engineering hasn't been picked yet. ML is like electricity 100 years ago, it will only expand and eat the world. And the research is not slowing down, on the contrary, it advances by leaps and bounds. The problem is that we don't have enough ML engineers and many who go by this title are not really capable of doing the job. We're just coming into decent tools and hardware, and many applications are still limited by hardware which itself is being reinvented every 2 years. Take just one single subfield - CV - it has applications in manufacturing, health, education, commerce, photography, agriculture, robotics, assisting blind persons, ... basically everywhere. It empowers new projects and amplifies automation. With the advent of pre-trained neural nets every new task can be 10x or 100x easier. We don't need as many labels anymore, it works much better now.

toomanybeersies 6y ago

That happened/is happening at my job. There's been a push to implement features that utilise AI/ML. Not because it would be a good use case (although there are some for our product), or because it would be of any practical benefit, but because it makes for good marketing copy. Never mind the fact that nobody on the team has any experience with machine learning (I actually failed the paper at university).

erichocean 6y ago

Without ML, our business today is literally impossible (from a financial perspective). I work in 2D animation and we were able to design our current pipeline around adopting ML at specific steps to remove massive amounts of manual labor. I know this doesn't disprove your anecdote, I just wanted to point out that real businesses are using ML effectively to deliver real value that's not possible without it.

Accujack 6y ago

This has happened since the dawn of the computer age, and probably before. Any technology too complex for the managers who purchase for it to understand fully can be sold and oversold by marketing people as "the next big thing". Managers may or may not see through that, but if their superiors want them to pursue it or if they need to pursue something in order to show they're doing something of value, then they're happy to follow where the marketers lead. Java everywhere, set top TV boxes, IOT devices, transitioning mainframes to minis, you name it... the marketers have made a mint selling it, usually for little benefit to the companies that bought into it.

tarsinge 6y ago

The problem I see is that in most non tech businesses they are not at the stage where they need ML, they are simply struggling with the basics: being able to seamlessly query or have consolidated up to date metrics and dashboards of the data scattered in all their databases. Of course the Big Data/AI “we’ll transform your data into insights” appealed to them, but that’s not what they need (also see the comments on the Palantir thread the other day).

jacobsenscott 6y ago

People have been trying to used algorithms of various sorts to increase sales (actionable insights) forever. The buzzwords change, but the results are always the same. No permutation of CPU instructions will turn a product people don't want to pay for into a product people want to pay for.

at-fates-hands 6y ago

The company I work for is a large health care company. I started in robotic automation about a year ago. The company said its next three huge initiatives would be: 1) AI 2) Machine Learning 3) Robotic Process Automation They felt RPA would help them stay more competitive since there are tons of smaller health care companies who are moving faster and innovating faster because they're not buying up companies and having to integrate all their technology at a sloth's pace. They thought RPA would be a way to mitigate these issues. 18 months later and the one manager, director and VP in my org has all but said they don't care about RPA, all their money is going into ML and AI. Even though in all the presentations I've seen them put on, its all blue skies and BS about "IF we had this, we COULD do this." Nothing concrete at all about how the plan to use ML to increase profit margins or reduce overhead. Right now, our team is basically an afterthought in the company and I'm already starting to interview elsewhere with the knowledge at some point, they're going to kill my team and cut everybody loose.

deleted 6y ago

[deleted]

Lxr 6y ago

ML is a shiny object with often no discernible ROI but occasionally very large ROI, and companies are understandably nervous about missing out. Spending a small amount to hedge their bets isn't necessarily irrational.

cnst 6y ago

There's also a lot of deception going on. The easiest way to solve many problems is through lexers, regular expressions and plain-old pattern matching. But that doesn't sell, so, they call it AI anyways.

RandoHolmes 6y ago

When I hear ML, deep learning, etc, I consider it a red flag for exactly the reasons you state. It's kind of batty actually, people looking for ideas to make money just been taking old ideas and attaching ML to the side of it as if that automatically made it better. And then not educating their customers on the limitations of ML both generally and with respect to their data size. I personally think the companies that make and sell the software that the police used to make incorrect arrests should be legally liable. Yes, the police shouldn't have blindly trusted the software, but I guaran-fucking-tee you part of why they did is the marketing from the company themselves.