5 ms·

Little offtopic about the curse of knowledge. I recently started interviewing ML Engineers for my company. In general I'm quite surprised by the lack of knowle

Little offtopic about the curse of knowledge. I recently started interviewing ML Engineers for my company. In general I'm quite surprised by the lack of knowledge of people applying for the job. People seem to have several misconceptions, very surface knowledge, and lack even the fundamentals. That made me question if I my expectations are set right. Is it possible that working in the field every day I expect candidates to know way more than it's reasonable? I'm not sure, and I don't feel we have a solid way to deal with that.

tehsauce 4y ago
Curious, any specific examples of missing knowledge or misconceptions?
- angarg12 4y ago
  Misconceptions: not knowing regression vs classification, supervised vs unsupervised, or thinking that ML is just neural networks.
- deepsquirrelnet 4y ago
  I tend to notice that model evaluation is lacking, perhaps because it’s not especially interesting. But to me, most business applied ML falls under the optimization umbrella. For some reason it’s never portrayed this way, but perhaps if it were, junior practitioners would more commonly pay attention to learning to thoroughly examine how their trained models will perform.
  - mrslave 4y ago
    Last year I nerded out on model accuracy and it was great fun. I would love to do this more at work but unless I fall into it in an existing role I usually am not considered for data science roles because I don't have 7 Ph.Ds. Simple accuracy, R2 for classification, ROC curves, calibration, Brier Skill scores. Good times. The best book I found at the time was Kuhn & Johnson. If anyone can recommend a better book I'd love to hear it? (Examples in R or Python, it doesn't matter.)
  - locuscoeruleus 4y ago
    I don't know what type of ML you do, but my experience is that it is hard to pin point poor model evaluation as being the reason for poor performance in production (especially if you have an interest in protecting your ego). It can be hard enough to figure out if you even have poor performance in production as you often don't know how well you could/should be doing. I have recently started thinking about what part of the performance issues I have are due to poor evaluation vs under-specification. I'm not sure how I'm supposed to tease the two apart.
cjbgkagh 4y ago
I consider myself a decent ML expert, but the field is so vast that I think I could be tripped up in an interview rather easily. Plus I tend to get a bit rusty with the basics that I never use for anything.
Silica6149 4y ago
Were they mostly new graduates, or did they have some years of experience already?
- angarg12 4y ago
  Both juniors and mid levels.
nl 4y ago
I once interviewed a ML engineer candidate who had two PhDs in ML who couldn't properly identify regression vs classification problems. > and lack even the fundamentals I don't think there is a strong consensus on what the fundamentals are. I've also noticed that the fundamentals differ remarkably between people who think of themselves as "data scientists" vs those who think of themselves as "machine learning practitioners".
- neodypsis 4y ago
  In your opinion, what are some examples of ML fundamentals? Would you include intermediate linear algebra?
  - qqqwwweeerrr 4y ago
    In my opinion (MSc math, PhD in applied statistics, currently postdoc at epidemiology department), the fundamentals are books like The Elements of Statistical Learning: https://hastie.su.domains/Papers/ESLII.pdf https://hastie.su.domains/Papers/ESLII.pdf I feel like the term data science is completely useless. Machine learning is an approach to "do AI" through statistics. Specifically, it is a branch of statistics where the sole focus is on prediction, compared to e.g. inference.
    - vadertemp 4y ago
      This is exactly how I tend to explain ML/ AI simply to people. Its mostly stats paired up with Linear Algebra, Calculus and Comp Science.
    - yobbo 4y ago
      The topics in that book are good typical "study material" but; 1) we can easily make anyone fail an interview by asking them to derive algorithms or formulas from there, and 2) all these topics predate deep learning, and everything that comes with it. I agree these are good "filtering" topics, among many others.
      - 4y ago
anton_ai 4y ago
Same experience hiring Junior Data Scientist, it's a shitshow out there. The level is so low that I had to hire a "old school" statistician with a senior position
- melling 4y ago
  These fields are relatively new. There doesn’t seem to be a clear path to break into them. Personally, I believe Kaggle is one of the ways to slowly gain some practical experience: https://www.kaggle.com/ https://www.kaggle.com/ However, I’m not sure if it’s sufficient. Recently, I’ve been taking a deeper dive into studying various types of competitions. For example, I’ve created a repo where I’m organizing notebooks, etc for a regression competition: https://github.com/melling/ml-regression https://github.com/melling/ml-regression I’m creating others for classification, nlp, vision, etc Of course, the self-study method means people have knowledge gaps because there’s no syllabus tailored for an interview
  - anton_ai 4y ago
    The field is new yes but before covid the recent graduate with even 1 year of experience were so good compared to now. The real problem is that with remote work you can't really learn something if you're a fresh graduate. I had my worst interns during the last two years, and I don't think that they were stupid. The real problem is that they can't really learn in remote when I don't have the time to teach them. When we were in the office they would jsut follow in every meeting, pair coding and so on. I love remote work, for me it's a bless, I exploit wvery single minute of my day working, but for interns and juniors is a nightmare they are not going to learn anything from me or from others.
antman 4y ago
What exactly do you ask an ML emgineer? In my experience if we ask 10 people of the scope of work pf an ML engineer we'll get 10 different answers 9 of which will be all inclusive aka Data Scientist that can also do robust production systems. I ask for the actual fundamental skills in the job ad. Say 5 skills for junior, 15 skills for senior, organizing other people (incl clients) for a manager
- angarg12 4y ago
  This is a fairly new role in our company. Officially it's described as 50% data scientist and 50% developer, but the loop composition is the same as developer but replacing one of the coding rounds for an ML round. I'm still experimenting with the format, but I do a mix of asking theoretical ML questions, prior experience with ML, and designing a system to solve a business problem using ML.
strikelaserclaw 4y ago
ML is the "hot" field right now, like "big data" was a couple years ago, it attracts a vast amount of people who are interested in only money (and a huge subset of those people aren't really that interested in "knowing")
arolihas 4y ago
Do you have any examples? What are your expectations?
- angarg12 4y ago
  The most baffling example is a candidate who admitted they didn't expect ML specific questions and hence hadn't prepared. I spent a minute figuring out if we were interviewing the right candidate. My expectations are always evolving since this is a new role for us. The current guidelines are that candidates should have broad knowledge of ML fundamentals. We also work through a design challenge together where a candidate solves a business problem using ML. I'm still figuring out the best ways to evaluate these.
  - arolihas 4y ago
    What kind of preparation do you expect, when you say ML specific questions or broad knowledge of ML fundamentals, is it doing linear algebra on the fly or just knowing high level topics like bias-variance, regularization, neural nets, k-means clustering?

tehsauce 4y ago

Curious, any specific examples of missing knowledge or misconceptions?

angarg12 4y ago
Misconceptions: not knowing regression vs classification, supervised vs unsupervised, or thinking that ML is just neural networks.
deepsquirrelnet 4y ago
I tend to notice that model evaluation is lacking, perhaps because it’s not especially interesting. But to me, most business applied ML falls under the optimization umbrella. For some reason it’s never portrayed this way, but perhaps if it were, junior practitioners would more commonly pay attention to learning to thoroughly examine how their trained models will perform.
- mrslave 4y ago
  Last year I nerded out on model accuracy and it was great fun. I would love to do this more at work but unless I fall into it in an existing role I usually am not considered for data science roles because I don't have 7 Ph.Ds. Simple accuracy, R2 for classification, ROC curves, calibration, Brier Skill scores. Good times. The best book I found at the time was Kuhn & Johnson. If anyone can recommend a better book I'd love to hear it? (Examples in R or Python, it doesn't matter.)
- locuscoeruleus 4y ago
  I don't know what type of ML you do, but my experience is that it is hard to pin point poor model evaluation as being the reason for poor performance in production (especially if you have an interest in protecting your ego). It can be hard enough to figure out if you even have poor performance in production as you often don't know how well you could/should be doing. I have recently started thinking about what part of the performance issues I have are due to poor evaluation vs under-specification. I'm not sure how I'm supposed to tease the two apart.

cjbgkagh 4y ago

I consider myself a decent ML expert, but the field is so vast that I think I could be tripped up in an interview rather easily. Plus I tend to get a bit rusty with the basics that I never use for anything.

Silica6149 4y ago

Were they mostly new graduates, or did they have some years of experience already?

angarg12 4y ago
Both juniors and mid levels.

nl 4y ago

I once interviewed a ML engineer candidate who had two PhDs in ML who couldn't properly identify regression vs classification problems. > and lack even the fundamentals I don't think there is a strong consensus on what the fundamentals are. I've also noticed that the fundamentals differ remarkably between people who think of themselves as "data scientists" vs those who think of themselves as "machine learning practitioners".

neodypsis 4y ago
In your opinion, what are some examples of ML fundamentals? Would you include intermediate linear algebra?
- qqqwwweeerrr 4y ago
  In my opinion (MSc math, PhD in applied statistics, currently postdoc at epidemiology department), the fundamentals are books like The Elements of Statistical Learning: https://hastie.su.domains/Papers/ESLII.pdf https://hastie.su.domains/Papers/ESLII.pdf I feel like the term data science is completely useless. Machine learning is an approach to "do AI" through statistics. Specifically, it is a branch of statistics where the sole focus is on prediction, compared to e.g. inference.
  - vadertemp 4y ago
    This is exactly how I tend to explain ML/ AI simply to people. Its mostly stats paired up with Linear Algebra, Calculus and Comp Science.
  - yobbo 4y ago
    The topics in that book are good typical "study material" but; 1) we can easily make anyone fail an interview by asking them to derive algorithms or formulas from there, and 2) all these topics predate deep learning, and everything that comes with it. I agree these are good "filtering" topics, among many others.
    - 4y ago

anton_ai 4y ago

Same experience hiring Junior Data Scientist, it's a shitshow out there. The level is so low that I had to hire a "old school" statistician with a senior position

melling 4y ago
These fields are relatively new. There doesn’t seem to be a clear path to break into them. Personally, I believe Kaggle is one of the ways to slowly gain some practical experience: https://www.kaggle.com/ https://www.kaggle.com/ However, I’m not sure if it’s sufficient. Recently, I’ve been taking a deeper dive into studying various types of competitions. For example, I’ve created a repo where I’m organizing notebooks, etc for a regression competition: https://github.com/melling/ml-regression https://github.com/melling/ml-regression I’m creating others for classification, nlp, vision, etc Of course, the self-study method means people have knowledge gaps because there’s no syllabus tailored for an interview
- anton_ai 4y ago
  The field is new yes but before covid the recent graduate with even 1 year of experience were so good compared to now. The real problem is that with remote work you can't really learn something if you're a fresh graduate. I had my worst interns during the last two years, and I don't think that they were stupid. The real problem is that they can't really learn in remote when I don't have the time to teach them. When we were in the office they would jsut follow in every meeting, pair coding and so on. I love remote work, for me it's a bless, I exploit wvery single minute of my day working, but for interns and juniors is a nightmare they are not going to learn anything from me or from others.

antman 4y ago

What exactly do you ask an ML emgineer? In my experience if we ask 10 people of the scope of work pf an ML engineer we'll get 10 different answers 9 of which will be all inclusive aka Data Scientist that can also do robust production systems. I ask for the actual fundamental skills in the job ad. Say 5 skills for junior, 15 skills for senior, organizing other people (incl clients) for a manager

angarg12 4y ago
This is a fairly new role in our company. Officially it's described as 50% data scientist and 50% developer, but the loop composition is the same as developer but replacing one of the coding rounds for an ML round. I'm still experimenting with the format, but I do a mix of asking theoretical ML questions, prior experience with ML, and designing a system to solve a business problem using ML.

strikelaserclaw 4y ago

ML is the "hot" field right now, like "big data" was a couple years ago, it attracts a vast amount of people who are interested in only money (and a huge subset of those people aren't really that interested in "knowing")

arolihas 4y ago

Do you have any examples? What are your expectations?

angarg12 4y ago
The most baffling example is a candidate who admitted they didn't expect ML specific questions and hence hadn't prepared. I spent a minute figuring out if we were interviewing the right candidate. My expectations are always evolving since this is a new role for us. The current guidelines are that candidates should have broad knowledge of ML fundamentals. We also work through a design challenge together where a candidate solves a business problem using ML. I'm still figuring out the best ways to evaluate these.
- arolihas 4y ago
  What kind of preparation do you expect, when you say ML specific questions or broad knowledge of ML fundamentals, is it doing linear algebra on the fly or just knowing high level topics like bias-variance, regularization, neural nets, k-means clustering?