6 ms·

"Like Big Data, I think we'll see a few companies execute well and actually get some value, while most will just jump to the next shiny thing in a year or two."

by apohn 6y ago

"Like Big Data, I think we'll see a few companies execute well and actually get some value, while most will just jump to the next shiny thing in a year or two." Here's another aspect - in many places nobody listens to the actual people doing the work. In my last job I was hired to lead a Data Science team and to help the company get value of Stats/ML/AI/DL/Buzzword. And I (and my team) were promptly overridden on every decision of what projects an expectations were realistic and what were not. I left, as did everybody else that reported to me, and we were replaced by people who would make really good BS slides that showed what upper management wanted to see. A year after that the whole initiative was cancelled. Back in 2000 I was in a similar position with a small company jumping on the internet as their next business model. Lots of nonsense and one horrible web based business later, the company failed. It's the same story over and over again. Some winners, lot of losers, many by self-inflicted wounds.

mrtksn 6y ago
If you think about it, that's the natural outcome. Why? Because people in corporations don't have the incentive to benefit the business but to progress their careers and that's done through meeting the goals for their position and make their upper ups progress with their careers too. So essentially, you have a system where people spend other people's resources for living and their success is judged by making the chain link above happy. In especially large companies it's easy to have a disconnect from the product because people in the top specialise in topics that have nothing to do with the product. If the people at the top want to have this shiny new thing that the press and everyone else is saying that it's the next big thing, you better give them the new shiny thing if you want to have a smooth career. In publicly traded companies, this is even more prevalent because people who buy and sell the stocks would be even more disconnected from the product and tied to the buzzwords. The more technical minded people who have the hunch on tech miss the point of the organisation that they are in and get very frustrated. It's probably the reason why startups can be much more fulfilling for deeply technical people.
- apohn 6y ago
  >If you think about it, that's the natural outcome. Why? Because people in corporations don't have the incentive to benefit the business but to progress their careers and that's done through meeting the goals for their position and make their upper ups progress with their careers too. This is one of the reasons I roll my eyes whenever I read something like "McKinsey says 75% of Big Data/AI/Buzzword projects do not deliver any value." What's the baseline for failing and/or delivering zero value because those projects were destined to fail?
  - bonoboTP 6y ago
    > because of silly management decisions? The whole point is, from their point of view those decisions are rational. It's much more lucrative from their (managers') personal point of view to develop a smokes-and-mirrors looks-good-on-ppt AI project. To be safe from risk, don't give the AI people too much responsibility, let them "do stuff", who cares, the point is we can now say we are an AI-driven company on the brochures, and we have something to report up to upper management. When they ask "are we also doing this deep learning thing? It's important nowadays!" we say "Of course, we have a team working on it, here's a PPT!". An actual AI project would have much bigger risks and uncertainty. I as a manager may be blamed for messing up real company processes if we actually rely on the AI. If it's just there but doesn't actually do anything, it's a net win for me. Note how this is not how things run when there are real goals that can be immediately improved through ML/AI and it shows up immediately on the bottom line, like ad and recommendation optimizations in Youtube or Netflix or core product value like at Tesla etc. The bullshit powerpoint AI with frustrated and confused engineers happens in companies where the connection is less direct and everyone only has a nebulous idea of what they would even want out of the AI system (extract valuable business knowledge!).
    - huffmsa 6y ago
      I think the problem a lot of places has been wanting "appealing" ML/AI solutions. The kind you write papers about and put on Powerpoints. The useful AI/ML isn't glamorous, it's quite boring and ugly. Things like spam detection, image labeling, event parsing, text classification. It's hard to get a big, shiny model into direct user facing systems.
austinl 6y ago
I've heard this happen in a lot of places — companies want to be "data-driven", but then leadership simply ignores the data. I think being data-driven is something that is built into company culture, or otherwise it's too easy to just ignore the results and ship. The place I currently work is data-driven (perhaps to a fault). Every change is wrapped behind an experiment and analyzed. Engineers play a major role in this process (responsible for analysis of simple experiments), whereas the data org owns more thorough, long-term analysis. This means there are a significant number of people invested in making numbers go up. It also means we're very good at finding local maxima, but struggle greatly shipping larger changes that land somewhere else on the graph. Some of the best advice I've heard related to this is for leadership to be honest about the "why". Sometimes we just want to ship a redesign to eventually find a new maximum, even through we know it will hurt metrics for a while.
- mumblemumble 6y ago
  Imagine what it must be like for the senior leadership of an established company to actually become data-driven. All of a sudden the leadership is going to consent to having all of their strategic and tactical decision-making be questioned by a bunch of relatively new hires from way down the org chart, whose entire basis for questioning all that expertise and business acumen is that they know how to fiddle around with numbers in some program called R? And all the while, they're constantly whining that this same data is junk and unreliable and we need to upend a whole bunch of IT systems just so they can rock the boat even harder? Pffft.
  - lallysingh 6y ago
    I expect data driven leaders to be good at analyzing data. The rest are bullshitters.
  - wokwokwok 6y ago
    That just reads as leaders who are high on cognitive bias. ...probably true to some extent, but not all leaders are self important ass hats who refuse to acknowledge they are simply “making decisions” not “making good decisions”. Most leaders are doing the best they can (often even very well) with the insights available to them. I don’t think most data teams are really at fault; they’re just doing what they’re told. The problem imo lies with the analysts who fail to do anything useful with the data they’re given, and demand constant changes from the data team because they want to deliver silver bullet results to the leadership level. That’s the problem layer; people who want to be important but have nothing to offer, whipping their data team to produce rubbish and then blaming them for either a) not producing anything fast enough or b) not making the numbers big enough.
poorman 6y ago
I think if a business is set up to scale by volume they can see gains from it. For example, say a business is already doing well at 100k conversions a day. They manage to apply "big data/ML" to optimize those conversions and gain a 3% lift, they are now making over a 1,095,000 extra conversions a year they would not have otherwise made.
- chrisandchris 6y ago
  So they need to make $1 profit for each of those conversions just to make it worth if they hire 1 ML scientist for 95k/year. Or $10 if they hire 10 for 950k/year in total. And so on... And there‘s the point where - IMHO - 3% gain may not be profitable enough.
  - tomrod 6y ago
    Extra conversions/year, so 1 DS at 95k means 1mm net profit
itronitron 6y ago
I think the only places where it yields consistent results is organizations that have at least 80% of their staff doing the ML/DS work and less than 20% managing the people doing the work (up and down in the organization.)
barkingcat 6y ago
This can be applied as "nobody listens to the people who actually do the work" as in company hires ML/AI experts to analyze purchase records and service records, and spits back out trends that the service front line workers (tier 1) already knew dead solid. Then the company doesn't listen to either group of people (neither tier 1 sales/support people, nor the ML people) and then fires / shuts down the entire division because "upper management didn't find value"
- closeparen 6y ago
  Contempt for this kind of knowledge is almost a religion in Silicon Valley.
- stjohnswarts 6y ago
  Some of the better historic manufacturers that "made it" were known to have good managers go and visit the filthy masses on the factory floor and get a feel for what's going on. It was very valuable for me when I used to help with manufacturing testing. I always spent some time with the techs and the people on the floor assembling stuff. A lot of it was useless but a lot of it was worthwhile and we learned to trust each other better instead of the "eggheads upstairs" and the "jarheads downstairs" that seemed to be most prevalent there.
  - mindentropy 6y ago
    I think what you have done is similar to what is depicted in "The Goal" by Eliyahu M. Goldratt. The graphic novel depicts it in a more succinct manner.
- alexslobodnik 6y ago
  Or it could be that a lot of data is wrong. It may be "technically" correct, ie the table in a database produces X. It is no surprise that executives would ignore what the "data" says because they don't trust it. A lot of time they are right to ignore it. I've seen tables say X, but there was some flaw up the capture stack. Very few data analyst have the broad based knowledge and dedication needed to trace the data stack to establish the needed trust with the executive team.
proverbialbunny 6y ago
Ditto. The same thing happened to me a few companies back. I lead a data science team of two solving difficult problems that would determine the company's success. However, management was the type to be uncomfortable with ignorance so they had to pretend to know data science and demand tasks be solved a certain way, which for anyone who has any familiar experience has already guessed it: what they were pushing made no sense. So, I switched from predictive analytics and put on my prescriptive analytics hat. Over the time I was there I created several presentations containing multiple paths forward letting management feel like they were deciding the path forward. This continued until I was fired. The board didn't like that I wasn't using a neural nets to solve the companies problems. Startups often do not have enough labeled data, so DNNs were not considered. Oddly, I didn't get a warning or a request about this before being let go. I suspect management got tired of me managing upward. In response my coworker quit right then and there and took me out to lunch. ^_^
mattferderer 6y ago
I did data a decade ago at a school district. A more experienced mentor at another school district let me in on a secret that is true across all industries no matter the size or type. If you give executives data and they don't like the results, they will ask you to tweak parameters until the data represents what they want.
wsgreen 6y ago
Currently experiencing the exact same thing. Roughly 15 machine learning engineers through hiring and acquisitions and all of them have no product or management power. Everyone with any management or product power has zero ML/Data Science experience. I spend half my day explaining to managers what ML is.

by apohn 6y ago

mrtksn 6y ago

If you think about it, that's the natural outcome. Why? Because people in corporations don't have the incentive to benefit the business but to progress their careers and that's done through meeting the goals for their position and make their upper ups progress with their careers too. So essentially, you have a system where people spend other people's resources for living and their success is judged by making the chain link above happy. In especially large companies it's easy to have a disconnect from the product because people in the top specialise in topics that have nothing to do with the product. If the people at the top want to have this shiny new thing that the press and everyone else is saying that it's the next big thing, you better give them the new shiny thing if you want to have a smooth career. In publicly traded companies, this is even more prevalent because people who buy and sell the stocks would be even more disconnected from the product and tied to the buzzwords. The more technical minded people who have the hunch on tech miss the point of the organisation that they are in and get very frustrated. It's probably the reason why startups can be much more fulfilling for deeply technical people.

apohn 6y ago
>If you think about it, that's the natural outcome. Why? Because people in corporations don't have the incentive to benefit the business but to progress their careers and that's done through meeting the goals for their position and make their upper ups progress with their careers too. This is one of the reasons I roll my eyes whenever I read something like "McKinsey says 75% of Big Data/AI/Buzzword projects do not deliver any value." What's the baseline for failing and/or delivering zero value because those projects were destined to fail?
- bonoboTP 6y ago
  > because of silly management decisions? The whole point is, from their point of view those decisions are rational. It's much more lucrative from their (managers') personal point of view to develop a smokes-and-mirrors looks-good-on-ppt AI project. To be safe from risk, don't give the AI people too much responsibility, let them "do stuff", who cares, the point is we can now say we are an AI-driven company on the brochures, and we have something to report up to upper management. When they ask "are we also doing this deep learning thing? It's important nowadays!" we say "Of course, we have a team working on it, here's a PPT!". An actual AI project would have much bigger risks and uncertainty. I as a manager may be blamed for messing up real company processes if we actually rely on the AI. If it's just there but doesn't actually do anything, it's a net win for me. Note how this is not how things run when there are real goals that can be immediately improved through ML/AI and it shows up immediately on the bottom line, like ad and recommendation optimizations in Youtube or Netflix or core product value like at Tesla etc. The bullshit powerpoint AI with frustrated and confused engineers happens in companies where the connection is less direct and everyone only has a nebulous idea of what they would even want out of the AI system (extract valuable business knowledge!).
  - huffmsa 6y ago
    I think the problem a lot of places has been wanting "appealing" ML/AI solutions. The kind you write papers about and put on Powerpoints. The useful AI/ML isn't glamorous, it's quite boring and ugly. Things like spam detection, image labeling, event parsing, text classification. It's hard to get a big, shiny model into direct user facing systems.

austinl 6y ago

I've heard this happen in a lot of places — companies want to be "data-driven", but then leadership simply ignores the data. I think being data-driven is something that is built into company culture, or otherwise it's too easy to just ignore the results and ship. The place I currently work is data-driven (perhaps to a fault). Every change is wrapped behind an experiment and analyzed. Engineers play a major role in this process (responsible for analysis of simple experiments), whereas the data org owns more thorough, long-term analysis. This means there are a significant number of people invested in making numbers go up. It also means we're very good at finding local maxima, but struggle greatly shipping larger changes that land somewhere else on the graph. Some of the best advice I've heard related to this is for leadership to be honest about the "why". Sometimes we just want to ship a redesign to eventually find a new maximum, even through we know it will hurt metrics for a while.

mumblemumble 6y ago
Imagine what it must be like for the senior leadership of an established company to actually become data-driven. All of a sudden the leadership is going to consent to having all of their strategic and tactical decision-making be questioned by a bunch of relatively new hires from way down the org chart, whose entire basis for questioning all that expertise and business acumen is that they know how to fiddle around with numbers in some program called R? And all the while, they're constantly whining that this same data is junk and unreliable and we need to upend a whole bunch of IT systems just so they can rock the boat even harder? Pffft.
- lallysingh 6y ago
  I expect data driven leaders to be good at analyzing data. The rest are bullshitters.
- wokwokwok 6y ago
  That just reads as leaders who are high on cognitive bias. ...probably true to some extent, but not all leaders are self important ass hats who refuse to acknowledge they are simply “making decisions” not “making good decisions”. Most leaders are doing the best they can (often even very well) with the insights available to them. I don’t think most data teams are really at fault; they’re just doing what they’re told. The problem imo lies with the analysts who fail to do anything useful with the data they’re given, and demand constant changes from the data team because they want to deliver silver bullet results to the leadership level. That’s the problem layer; people who want to be important but have nothing to offer, whipping their data team to produce rubbish and then blaming them for either a) not producing anything fast enough or b) not making the numbers big enough.

poorman 6y ago

I think if a business is set up to scale by volume they can see gains from it. For example, say a business is already doing well at 100k conversions a day. They manage to apply "big data/ML" to optimize those conversions and gain a 3% lift, they are now making over a 1,095,000 extra conversions a year they would not have otherwise made.

chrisandchris 6y ago
So they need to make $1 profit for each of those conversions just to make it worth if they hire 1 ML scientist for 95k/year. Or $10 if they hire 10 for 950k/year in total. And so on... And there‘s the point where - IMHO - 3% gain may not be profitable enough.
- tomrod 6y ago
  Extra conversions/year, so 1 DS at 95k means 1mm net profit

itronitron 6y ago

I think the only places where it yields consistent results is organizations that have at least 80% of their staff doing the ML/DS work and less than 20% managing the people doing the work (up and down in the organization.)

barkingcat 6y ago

This can be applied as "nobody listens to the people who actually do the work" as in company hires ML/AI experts to analyze purchase records and service records, and spits back out trends that the service front line workers (tier 1) already knew dead solid. Then the company doesn't listen to either group of people (neither tier 1 sales/support people, nor the ML people) and then fires / shuts down the entire division because "upper management didn't find value"

closeparen 6y ago
Contempt for this kind of knowledge is almost a religion in Silicon Valley.
stjohnswarts 6y ago
Some of the better historic manufacturers that "made it" were known to have good managers go and visit the filthy masses on the factory floor and get a feel for what's going on. It was very valuable for me when I used to help with manufacturing testing. I always spent some time with the techs and the people on the floor assembling stuff. A lot of it was useless but a lot of it was worthwhile and we learned to trust each other better instead of the "eggheads upstairs" and the "jarheads downstairs" that seemed to be most prevalent there.
- mindentropy 6y ago
  I think what you have done is similar to what is depicted in "The Goal" by Eliyahu M. Goldratt. The graphic novel depicts it in a more succinct manner.
alexslobodnik 6y ago
Or it could be that a lot of data is wrong. It may be "technically" correct, ie the table in a database produces X. It is no surprise that executives would ignore what the "data" says because they don't trust it. A lot of time they are right to ignore it. I've seen tables say X, but there was some flaw up the capture stack. Very few data analyst have the broad based knowledge and dedication needed to trace the data stack to establish the needed trust with the executive team.

proverbialbunny 6y ago

Ditto. The same thing happened to me a few companies back. I lead a data science team of two solving difficult problems that would determine the company's success. However, management was the type to be uncomfortable with ignorance so they had to pretend to know data science and demand tasks be solved a certain way, which for anyone who has any familiar experience has already guessed it: what they were pushing made no sense. So, I switched from predictive analytics and put on my prescriptive analytics hat. Over the time I was there I created several presentations containing multiple paths forward letting management feel like they were deciding the path forward. This continued until I was fired. The board didn't like that I wasn't using a neural nets to solve the companies problems. Startups often do not have enough labeled data, so DNNs were not considered. Oddly, I didn't get a warning or a request about this before being let go. I suspect management got tired of me managing upward. In response my coworker quit right then and there and took me out to lunch. ^_^

mattferderer 6y ago

I did data a decade ago at a school district. A more experienced mentor at another school district let me in on a secret that is true across all industries no matter the size or type. If you give executives data and they don't like the results, they will ask you to tweak parameters until the data represents what they want.

wsgreen 6y ago

Currently experiencing the exact same thing. Roughly 15 machine learning engineers through hiring and acquisitions and all of them have no product or management power. Everyone with any management or product power has zero ML/Data Science experience. I spend half my day explaining to managers what ML is.