5 ms·

Yes, this is extremely common and extremely frustrating. Companies, governments, individuals come to me all the time asking how to "Do AI" without knowing anyt

Yes, this is extremely common and extremely frustrating. Companies, governments, individuals come to me all the time asking how to "Do AI" without knowing anything about their product, data provenance, testing plans, state observability etc... Sure, you can do a toy problem with some interesting vision or NLP framework and some off the shelf data that isn't ready for production or anything serious. Who cares? That doesn't do anything novel, it doesn't actually change your product function. Chomsky said the most important thing to do as a young person is to follow the phrase "Know Thyself." People who want to do AI need to "Know Thy-data and product architecture" first. I did a talk last year on exactly this: https://www.youtube.com/watch?v=jK-SBu1iiHo https://www.youtube.com/watch?v=jK-SBu1iiHo

lumost 6y ago
My 2 cents is that ML oriented job titles are too fine grained with everyone in theory serving a mythic modeler who just needs to be fead data. Everyone finds the 1% of the time they spend building models the rewarding part - however many of the job titles like Data engineer, or ML engineer do not include modeling in the job description. There are companies which differentiate scientists into multiple categoies based on whether they do ML, can code, or just statistics. Now the problem is that any of these folks can get into trouble for doing the others job, and the pay scales are often radically different. Relegating core responsibilities such as data cleaning/collection to support roles. I'd bet that flattening the title structure in these groups would yield better results, and allow team members to more seamlessly move across Backend, Data, ML, and science oriented tasks.
- ethbr0 6y ago
  I think it's a problem of visibility up the management ladder. (A) If management understands ML enough to launch a feasible project, (B) if management understands that models are a component of ML, (C) if management understands models are created from data, (D) if management understands that production models must continually be fed data. Typically, understanding seems to stop somewhere between A & B.
- lostdog 6y ago
  Some companies don't separate ML roles by title, but that still leads to problems. Without management-driven separation, too many people try to work on just modeling. You wind up with nobody labeling and cleaning the data. Everyone complains that it's hard to get their models to work on the limited data available, but no one wants to be relegated to the data cleaning role. I have yet to see a good company layout that makes everyone happy. Too many people want to work on modeling, and the worst part is that individuals are right to fight for it. The more practice you get, the more you improve at the work. Plus, companies tend to value good modelers, so your salary goes up too. It's a very frustrating feedback loop. Separating modeling work by title or by team is one of the few ways companies can keep from having a run on the modeling projects. There are a few other ways, but they are all pretty bad too. You can have a "keep what you kill" structure, where everyone gets first dibs on doing modeling on the data they've collected and cleaned. I've also seen a company that makes the modeling job as miserable as possible with slow tooling and high expectations, so lots of people wash out and only a few modelers are left. I wish I knew a good solution to this. My only idea is to play-up the importance (and salary!) of the support roles, but I'm not satisfied yet.
  - disgruntledphd2 6y ago
    Like, this blows my mind. I can't even imagine a world where one would model without data cleaning. In the olden times, your model would normally error out if you hadn't done any data cleaning (length 1 factors, constant columns etc) but i suppose with NN's they just ignore stuff like that because of the regularisation, I assume. Every good DS I know regards the data as much more important than the models, normally from bitter, bitter experience. Models are "sexy" though, so people love them. Additionally you can do the XKCD compiling thing x 10 with models, as they take forever to fit on any non-trivial datasets. I actually think that the cause of this is poor management. ML/DS//statistics are tools, they do not supply any business value by themselves. If people focus too much on the tools, then their manager should refocus them on the actual problem. Mind you, if you don't have the background in this area as a manager, you're less likely to be able to spot these pathologies. Personally, as an experienced DS, I know that my time is best spent understanding the data and the business problem, as that will drive better results. It's sad that people don't realise this (including some of my current management chain, unfortunately).
- beforeolives 6y ago
  This is interesting because one of the criticisms towards data science have been that it's a term so broad that two people can have the title data scientist and do completely different jobs. Hence the differentiation over time into research scientists, ML engineers etc. Personally I might prefer the generalist hat but I see things moving in the opposite direction. There is also the problem of hiring - if 1-10% of the job requires a lot of statistical theory/research and the rest doesn't, do you hire PhDs and ask them to mostly do the engineering tasks that they aren't necessarily prepared to do? What about the people who are perfectly capable of doing 99% of the job and picking up the extra 1% of theory - do you exclude them because they don't have the academic credentials?
  - lumost 6y ago
    I would take the approach that there is a spectrum of knowledge and hire the quote "T" shaped individual. Strong in at least one area, but broad enough to contribute anywhere. People change faster than orgs/titles, it would be better to keep this part flexible to maximize their growth. This can be further extended to the whole engineering culture. It's not uncommon that a frontend engineer shares a title with a backend software engineer - these skills are different enough that the individuals are no longer fungible for the most part. Why not give DS's engineering titles and treat it as a management objective to ensure you have the right people on any given project? If scientist is the preferred title, why not give anyone the science title?
rodgerd 6y ago
> Yes, this is extremely common and extremely frustrating. Particularly when you can raise billions by being confidently wrong.
neltnerb 6y ago
Maybe they could try actually hiring those of us who have PhDs in other fields who are actually enthusiastic about generating large quantities of reproducible and transferable data. I hear people say all the time how this is what they want, yet for some reason if you aren't already published in machine learning no one wants to take a risk. Instead it's easier to complain about poorly formatted datasets.
- disgruntledphd2 6y ago
  Don't go for a research job. Your actual data skills will be super valuable to a company that's trying to increase revenue/decrease costs. Bonus if you did an individual PhD, as you'll likely be able to organise your workload better.
  - bosie 6y ago
    what's an individual PhD?
    - disgruntledphd2 6y ago
      One where you don't get handed a topic and a lab by your advisor.
  - neltnerb 6y ago
    My skills are strongly in the direction of "design, build, and operate a test lab that will be used to automate collection of high quality datasets relevant to the problem of interest". Whenever I've tried to actually use existing data to attack the problems I'm most interested in the fundamental problem is that data that is comparable between different labs doesn't exist. And to avoid any confusion, I'm not saying it doesn't replicate, just that it's not easily comparable, like if you want to know what platinum catalysts will do over an alkene but the data you can find was done in a batch reactor with 22nm platinum and you're using a continuous flow reactor with 10nm platinum. And it's usually worse than that once you include how it's supported, what flow rates were, heating profiles, etc. It just goes on and on if you don't do it yourself.

lumost 6y ago

My 2 cents is that ML oriented job titles are too fine grained with everyone in theory serving a mythic modeler who just needs to be fead data. Everyone finds the 1% of the time they spend building models the rewarding part - however many of the job titles like Data engineer, or ML engineer do not include modeling in the job description. There are companies which differentiate scientists into multiple categoies based on whether they do ML, can code, or just statistics. Now the problem is that any of these folks can get into trouble for doing the others job, and the pay scales are often radically different. Relegating core responsibilities such as data cleaning/collection to support roles. I'd bet that flattening the title structure in these groups would yield better results, and allow team members to more seamlessly move across Backend, Data, ML, and science oriented tasks.

ethbr0 6y ago
I think it's a problem of visibility up the management ladder. (A) If management understands ML enough to launch a feasible project, (B) if management understands that models are a component of ML, (C) if management understands models are created from data, (D) if management understands that production models must continually be fed data. Typically, understanding seems to stop somewhere between A & B.
lostdog 6y ago
Some companies don't separate ML roles by title, but that still leads to problems. Without management-driven separation, too many people try to work on just modeling. You wind up with nobody labeling and cleaning the data. Everyone complains that it's hard to get their models to work on the limited data available, but no one wants to be relegated to the data cleaning role. I have yet to see a good company layout that makes everyone happy. Too many people want to work on modeling, and the worst part is that individuals are right to fight for it. The more practice you get, the more you improve at the work. Plus, companies tend to value good modelers, so your salary goes up too. It's a very frustrating feedback loop. Separating modeling work by title or by team is one of the few ways companies can keep from having a run on the modeling projects. There are a few other ways, but they are all pretty bad too. You can have a "keep what you kill" structure, where everyone gets first dibs on doing modeling on the data they've collected and cleaned. I've also seen a company that makes the modeling job as miserable as possible with slow tooling and high expectations, so lots of people wash out and only a few modelers are left. I wish I knew a good solution to this. My only idea is to play-up the importance (and salary!) of the support roles, but I'm not satisfied yet.
- disgruntledphd2 6y ago
  Like, this blows my mind. I can't even imagine a world where one would model without data cleaning. In the olden times, your model would normally error out if you hadn't done any data cleaning (length 1 factors, constant columns etc) but i suppose with NN's they just ignore stuff like that because of the regularisation, I assume. Every good DS I know regards the data as much more important than the models, normally from bitter, bitter experience. Models are "sexy" though, so people love them. Additionally you can do the XKCD compiling thing x 10 with models, as they take forever to fit on any non-trivial datasets. I actually think that the cause of this is poor management. ML/DS//statistics are tools, they do not supply any business value by themselves. If people focus too much on the tools, then their manager should refocus them on the actual problem. Mind you, if you don't have the background in this area as a manager, you're less likely to be able to spot these pathologies. Personally, as an experienced DS, I know that my time is best spent understanding the data and the business problem, as that will drive better results. It's sad that people don't realise this (including some of my current management chain, unfortunately).
beforeolives 6y ago
This is interesting because one of the criticisms towards data science have been that it's a term so broad that two people can have the title data scientist and do completely different jobs. Hence the differentiation over time into research scientists, ML engineers etc. Personally I might prefer the generalist hat but I see things moving in the opposite direction. There is also the problem of hiring - if 1-10% of the job requires a lot of statistical theory/research and the rest doesn't, do you hire PhDs and ask them to mostly do the engineering tasks that they aren't necessarily prepared to do? What about the people who are perfectly capable of doing 99% of the job and picking up the extra 1% of theory - do you exclude them because they don't have the academic credentials?
- lumost 6y ago
  I would take the approach that there is a spectrum of knowledge and hire the quote "T" shaped individual. Strong in at least one area, but broad enough to contribute anywhere. People change faster than orgs/titles, it would be better to keep this part flexible to maximize their growth. This can be further extended to the whole engineering culture. It's not uncommon that a frontend engineer shares a title with a backend software engineer - these skills are different enough that the individuals are no longer fungible for the most part. Why not give DS's engineering titles and treat it as a management objective to ensure you have the right people on any given project? If scientist is the preferred title, why not give anyone the science title?

rodgerd 6y ago

> Yes, this is extremely common and extremely frustrating. Particularly when you can raise billions by being confidently wrong.

neltnerb 6y ago

Maybe they could try actually hiring those of us who have PhDs in other fields who are actually enthusiastic about generating large quantities of reproducible and transferable data. I hear people say all the time how this is what they want, yet for some reason if you aren't already published in machine learning no one wants to take a risk. Instead it's easier to complain about poorly formatted datasets.

disgruntledphd2 6y ago
Don't go for a research job. Your actual data skills will be super valuable to a company that's trying to increase revenue/decrease costs. Bonus if you did an individual PhD, as you'll likely be able to organise your workload better.
- bosie 6y ago
  what's an individual PhD?
  - disgruntledphd2 6y ago
    One where you don't get handed a topic and a lab by your advisor.
- neltnerb 6y ago
  My skills are strongly in the direction of "design, build, and operate a test lab that will be used to automate collection of high quality datasets relevant to the problem of interest". Whenever I've tried to actually use existing data to attack the problems I'm most interested in the fundamental problem is that data that is comparable between different labs doesn't exist. And to avoid any confusion, I'm not saying it doesn't replicate, just that it's not easily comparable, like if you want to know what platinum catalysts will do over an alkene but the data you can find was done in a batch reactor with 22nm platinum and you're using a continuous flow reactor with 10nm platinum. And it's usually worse than that once you include how it's supported, what flow rates were, heating profiles, etc. It just goes on and on if you don't do it yourself.