3 ms·

This is some of the reason, but I've noticed the same behavior in data scientists who are "vertically integrated" and who are only measured on the quality of th

This is some of the reason, but I've noticed the same behavior in data scientists who are "vertically integrated" and who are only measured on the quality of the final model. In this case, their incentives are aligned with good data work, but they still suck at it and neglect it. I believe there's a few psychological reasons: - People aren't trained to do it in college, so it's undervalued. - It requires domain knowledge, which is hard. - The work is quite gruelling and annoying. It's very detail oriented. - You don't feel like you're making progress while you're doing data work. Modelling work feels closer to the final output. - Self-delusion during modelling is easy since you can overfit on the holdout data which gives you those dopamine hits. - Modelling satisfies curiosity. During data work, the data scientist can't wait to hurry up to the modelling stage and see what the results look like. - Many people are just bad with data work, they don't have the attention to detail required and produce data with glaring errors.

disgruntledphd2 6y ago
Again, this boggles my mind. Data cleaning is 90% of the job. If you don't like this, then you should not be working in ML/DS. I actually enjoy data cleaning, sure it's annoying sometimes but the insight you gain actually allows you to build a much better model. And, to counterpoint one of your points, I would note that part of my (psychology) degree was collecting data from our friends for various surveys and experiments. We'd then add everyone's data together and run analyses on this. It's really crazy to me that stats/ML/everyone else didn't do any of this, as it certainly lead to me being a much better data scientist.
- bonoboTP 6y ago
  Incentives. Do you have skin in the game? Are you building your cv and career and brand or are you building the product? Do you personally benefit more from 10% improvement due to data cleaning or 1% improvement from some fancy tweak of the model which you can hold talks on and write papers about, with math formulas etc?
  - disgruntledphd2 6y ago
    No, but I like making/saving money, doing analysis, writing code and talking about how to do all of those things better. Especially the making money part, as that's ultimately why my employer/clients pay me.
- hntrader 6y ago
  You have the right practical mindset, and it might have something to do with your degree choice. People that pick psychology aren't going to have their head in their clouds and aren't doing DS for prestige reasons - DS is more a means to an end instead of an end itself. You're also trained only in fairly simple, mostly linear models, which helps. There's an even more extreme type to the one I outlined. This type of DS dislikes data engineering and dislikes applied modelling. They prefer to sit in an armchair and think about modelling, writing up a LaTeX document with abstract and overcomplicated math formula. I met one DS like this who was from a theoretical physics background and his years of operating in this mode couldn't be shaken off.
  - disgruntledphd2 6y ago
    Just as an FYI, DS didn't exist when I did my undergrad, and Facebook were the only people with DS during my PhD. I got into DS because I really enjoy analysing data and running experiments, and as a backup to academia.

disgruntledphd2 6y ago

Again, this boggles my mind. Data cleaning is 90% of the job. If you don't like this, then you should not be working in ML/DS. I actually enjoy data cleaning, sure it's annoying sometimes but the insight you gain actually allows you to build a much better model. And, to counterpoint one of your points, I would note that part of my (psychology) degree was collecting data from our friends for various surveys and experiments. We'd then add everyone's data together and run analyses on this. It's really crazy to me that stats/ML/everyone else didn't do any of this, as it certainly lead to me being a much better data scientist.

bonoboTP 6y ago
Incentives. Do you have skin in the game? Are you building your cv and career and brand or are you building the product? Do you personally benefit more from 10% improvement due to data cleaning or 1% improvement from some fancy tweak of the model which you can hold talks on and write papers about, with math formulas etc?
- disgruntledphd2 6y ago
  No, but I like making/saving money, doing analysis, writing code and talking about how to do all of those things better. Especially the making money part, as that's ultimately why my employer/clients pay me.
hntrader 6y ago
You have the right practical mindset, and it might have something to do with your degree choice. People that pick psychology aren't going to have their head in their clouds and aren't doing DS for prestige reasons - DS is more a means to an end instead of an end itself. You're also trained only in fairly simple, mostly linear models, which helps. There's an even more extreme type to the one I outlined. This type of DS dislikes data engineering and dislikes applied modelling. They prefer to sit in an armchair and think about modelling, writing up a LaTeX document with abstract and overcomplicated math formula. I met one DS like this who was from a theoretical physics background and his years of operating in this mode couldn't be shaken off.
- disgruntledphd2 6y ago
  Just as an FYI, DS didn't exist when I did my undergrad, and Facebook were the only people with DS during my PhD. I got into DS because I really enjoy analysing data and running experiments, and as a backup to academia.