4 ms·

> In the context of the modeling onsite, we ask open-ended problems with sufficient business and problem context such that the candidate can clearly identify an

> In the context of the modeling onsite, we ask open-ended problems with sufficient business and problem context such that the candidate can clearly identify an ML-based approach to solve it. I'm disappointed the author wasn't more specific about where the line is drawn between "ML SWE" and "Research Scientist"/"Data Scientist" when it comes to the core ML competencies like model selection, evaluation, and design. Having worked in teams with Data Scientists and "ML Engineers" it's been murky whether me and others in my team that were not the former were Data Engineers, Backend Software Engineers, "ML SWE", or "Software Engineer - Machine Learning". This area of software is rapidly developing and there's no semblance of the bright line that tends to exist between Backend Engineer and Frontend Engineer for ML-involved engineers. I'm personally interested in moving into the "Software Engineer - ML" space (or is it "ML SWE"?) and thus I have to find out what the right balance between Software Eng skills and researcher skills is. I think I have a decent sense of real-world ML basics but would quickly flounder when pressed for details on mathematical technicalities of different modelling approaches. Lyft's job listings have these items for "Software Engineer - Machine Learning": > 5+ years (or Ph.D. with 2+ years) of industry or research experience developing ML models This really seems like the purview of a research scientist or a data scientist, if I understand the meaning of "developing" correctly. > Proven ability to quickly and effectively turn research ML papers into working code This seems fair enough, but my experience has been that the data scientists are doing this mostly, while the "Software Engineer - ML" people are preparing data pipelines or batch training systems. > Deep knowledge of ML libraries like scikit-learn, Tensorflow, PyTorch, Keras, MXNet, etc I know it's a job ad and thus it is a 'wish list', but this seems unreasonable. Guess I'll just keeping studying 'all the things'. Brb in 5 years.

mlthoughts2018 7y ago
I think you underestimate the level of software architecture and engineering skill that people with formal training in graduate level statistics bring to these jobs. I manage a team of machine learning engineers in a mid-size ecommerce company and I can tell you that the same person who is optimizing Dockerfiles for better layer reuse & figuring out how our CI pipeline will safely get secrets needed to retrieve model files for integration tests is the same person researching new variations of triplet loss in a paper from arxiv they present in our team’s journal club & developing Bayesian hierarchical regression models and explaining why partial pooling using an industry-specialized prior distribution actually produces coefficients in the model fit that have a meaningful improvement over OLS for some business outcome. The same people that are refactoring a collection of unit tests to be parallelizable and save us 45 seconds on every test run are also running huge hyperparameter tuning experiments to get a learning rate scheduler that allows us to reduce training time for an in-house deep GRU neural network from 24 hours to 15 hours, and they are defining infrastructure as code tooling to even create the very GPU environments where this training is taking place. The skill set really is extremely different from general backend engineer with a proclivity for hacking around ML models and also is really different from “research developer” who rarely deals with end to end systems or necessary concerns of production or quality code factoring, and also is super different from data science which is effectively just ad hoc business analytics but with more impressive pedigree on the resume. I’d define a machine learning engineer as taking someone with several years of graduate experience in statistics / machine learning inclusive of formal probability theory, analysis, topology, Bayesian stats (or work + research experience that is equivalent, though it’s super rare for this to be able to replace formal graduate math training), and then adding senior level skill set in high-performance computing, generalist backend engineering, system architecture, and full management of the lifecycle of complex production systems. The only piece of common modern engineering that I would say ML engineers typically don’t have a senior-ish level of command over is frontend development (though some do out of just hobbyist interest).
- thundergolfer 7y ago
  I don't think I'm underestimating "people with formal training in graduate level statistics", but quite possibly the particular people you think of when you say the quoted phrase. I've worked with stats/comsci/bio-informatic/ML PhDs at multiple multi-billion-dollar software companies now and it's certainly not true that even the majority of people with graduate stats training are excellent software engineers. Would love to work at a company where that's true, I just don't think that's the common case at all. > The skill set really is extremely different from general backend engineer with a proclivity for hacking around ML models and also is really different from “research developer” who rarely deals with end to end systems or necessary concerns of production or quality code factoring, and also is super different from data science which is effectively just ad hoc business analytics but with more impressive pedigree on the resume. Totally agree. The terms are very messy though. The "ML Engineers" with PhDs in ML are still called "Data Scientists" at Zendesk, for example. At other companies "Data Scientist" just means 'Data Analyst that knows SQL'. > I’d define a machine learning engineer... Would be happy with this definition. It's much clearer than what you can gather from Lyft's post. It's close to the definition I currently hold to. On this definition I've enrolled in graduate Maths+Stats as I'm still almost entirely a Data Engineer / Backend Engineer in terms of skillset.
- laichzeit0 7y ago
  Your definition of ML engineer comes off as being a high-risk individual to have in an organization. I would rather split those into two separate orthogonal roles and have redundancy in my resource pool. It seems like a very difficult and scare resource to hire. How can Average Corp even think of hiring someone like that? Dead no from me.
  - mlthoughts2018 7y ago
    The problem is you can’t really split them into two. You need the same person designing models to also be making tactical decisions about software design and production lifecycle because those things are almost always highly dependent on nuanced specifics of the model itself. There is not a division of labor where one person makes the model and then throws it over the fence to a team that manages its production usage and lifecycle. That team over the fence would not be able to do it, and I’ve seen this attempted org structure fail hard everywhere its been tried for ML services. What’s telling is that you bring up the personnel risk but you don’t consider the value add. When Average Corp hires someone, it’s because that person brings more value than they cost, period. It’s not because the person is “not risky” in some vacuum of decision making like you’re painting it out to be. > Dead no from me. That’s fine and all, but it usually indicates tech death of a company, and likely you have brain drain in more area than just machine learning. If you aren’t willing to take the risk and do what’s needed to structure the work and job to extract value from high performers, that’s a sign of corporate mediocrity and I think anyone with the skill set to be an ML engineer like this would already not even be applying to work in a place like that. Honestly the risks are even worse than your comment says. There are also big risks around keeping this person intellectually engaged, giving them valuable job experience. You pretty much have to pay them a lot, give them good work life balance, give them budget for conference travel & continued learning, give them meaningful upward career & compensation growth, and give them meaningful projects. I look at this and think, yes, if the business can’t give all those things and still be coming out ahead on the person’s productivity, then you don’t want an ML engineer. But more often the company needs an ML engineer and absolutely would gain more from their productivity than they lose on supporting all those job quality aspects — yet managers just take superficial offense at these demands and balk at the idea that you have to provide meaningful projects and career growth instead of just barking orders and expecting them to put up with work that does not help them grow.
applecrazy 7y ago
Undergrad here. This is kind of disappointing. I really want to work in this field, but it seems I must have a PhD at minimum and not sure if I want to take that step yet.
- bart_spoon 7y ago
  You don't, though a MS may be borderline necessary. Without a PhD you will likely only lose out on some jobs at companies that are doing hardcore ML research or are simply more invested in the DS hype than actually looking to solve problems with DS. There are plenty of companies out there with data science needs that can be filled by non-PhD data scientists.

mlthoughts2018 7y ago

I think you underestimate the level of software architecture and engineering skill that people with formal training in graduate level statistics bring to these jobs. I manage a team of machine learning engineers in a mid-size ecommerce company and I can tell you that the same person who is optimizing Dockerfiles for better layer reuse & figuring out how our CI pipeline will safely get secrets needed to retrieve model files for integration tests is the same person researching new variations of triplet loss in a paper from arxiv they present in our team’s journal club & developing Bayesian hierarchical regression models and explaining why partial pooling using an industry-specialized prior distribution actually produces coefficients in the model fit that have a meaningful improvement over OLS for some business outcome. The same people that are refactoring a collection of unit tests to be parallelizable and save us 45 seconds on every test run are also running huge hyperparameter tuning experiments to get a learning rate scheduler that allows us to reduce training time for an in-house deep GRU neural network from 24 hours to 15 hours, and they are defining infrastructure as code tooling to even create the very GPU environments where this training is taking place. The skill set really is extremely different from general backend engineer with a proclivity for hacking around ML models and also is really different from “research developer” who rarely deals with end to end systems or necessary concerns of production or quality code factoring, and also is super different from data science which is effectively just ad hoc business analytics but with more impressive pedigree on the resume. I’d define a machine learning engineer as taking someone with several years of graduate experience in statistics / machine learning inclusive of formal probability theory, analysis, topology, Bayesian stats (or work + research experience that is equivalent, though it’s super rare for this to be able to replace formal graduate math training), and then adding senior level skill set in high-performance computing, generalist backend engineering, system architecture, and full management of the lifecycle of complex production systems. The only piece of common modern engineering that I would say ML engineers typically don’t have a senior-ish level of command over is frontend development (though some do out of just hobbyist interest).

thundergolfer 7y ago
I don't think I'm underestimating "people with formal training in graduate level statistics", but quite possibly the particular people you think of when you say the quoted phrase. I've worked with stats/comsci/bio-informatic/ML PhDs at multiple multi-billion-dollar software companies now and it's certainly not true that even the majority of people with graduate stats training are excellent software engineers. Would love to work at a company where that's true, I just don't think that's the common case at all. > The skill set really is extremely different from general backend engineer with a proclivity for hacking around ML models and also is really different from “research developer” who rarely deals with end to end systems or necessary concerns of production or quality code factoring, and also is super different from data science which is effectively just ad hoc business analytics but with more impressive pedigree on the resume. Totally agree. The terms are very messy though. The "ML Engineers" with PhDs in ML are still called "Data Scientists" at Zendesk, for example. At other companies "Data Scientist" just means 'Data Analyst that knows SQL'. > I’d define a machine learning engineer... Would be happy with this definition. It's much clearer than what you can gather from Lyft's post. It's close to the definition I currently hold to. On this definition I've enrolled in graduate Maths+Stats as I'm still almost entirely a Data Engineer / Backend Engineer in terms of skillset.
laichzeit0 7y ago
Your definition of ML engineer comes off as being a high-risk individual to have in an organization. I would rather split those into two separate orthogonal roles and have redundancy in my resource pool. It seems like a very difficult and scare resource to hire. How can Average Corp even think of hiring someone like that? Dead no from me.
- mlthoughts2018 7y ago
  The problem is you can’t really split them into two. You need the same person designing models to also be making tactical decisions about software design and production lifecycle because those things are almost always highly dependent on nuanced specifics of the model itself. There is not a division of labor where one person makes the model and then throws it over the fence to a team that manages its production usage and lifecycle. That team over the fence would not be able to do it, and I’ve seen this attempted org structure fail hard everywhere its been tried for ML services. What’s telling is that you bring up the personnel risk but you don’t consider the value add. When Average Corp hires someone, it’s because that person brings more value than they cost, period. It’s not because the person is “not risky” in some vacuum of decision making like you’re painting it out to be. > Dead no from me. That’s fine and all, but it usually indicates tech death of a company, and likely you have brain drain in more area than just machine learning. If you aren’t willing to take the risk and do what’s needed to structure the work and job to extract value from high performers, that’s a sign of corporate mediocrity and I think anyone with the skill set to be an ML engineer like this would already not even be applying to work in a place like that. Honestly the risks are even worse than your comment says. There are also big risks around keeping this person intellectually engaged, giving them valuable job experience. You pretty much have to pay them a lot, give them good work life balance, give them budget for conference travel & continued learning, give them meaningful upward career & compensation growth, and give them meaningful projects. I look at this and think, yes, if the business can’t give all those things and still be coming out ahead on the person’s productivity, then you don’t want an ML engineer. But more often the company needs an ML engineer and absolutely would gain more from their productivity than they lose on supporting all those job quality aspects — yet managers just take superficial offense at these demands and balk at the idea that you have to provide meaningful projects and career growth instead of just barking orders and expecting them to put up with work that does not help them grow.

applecrazy 7y ago

Undergrad here. This is kind of disappointing. I really want to work in this field, but it seems I must have a PhD at minimum and not sure if I want to take that step yet.

bart_spoon 7y ago
You don't, though a MS may be borderline necessary. Without a PhD you will likely only lose out on some jobs at companies that are doing hardcore ML research or are simply more invested in the DS hype than actually looking to solve problems with DS. There are plenty of companies out there with data science needs that can be filled by non-PhD data scientists.