7 ms·
Ask HN: What does an ML Engineer do?
I have a few years of experience as an ML Engineer, however I recently changed jobs and I was shocked to learn that my new employer has a very different job definition for ML Engineer than the one I was familiar with. I thought that the standard definition of an ML Engineer is an engineer who specializes in building machine learning models, essentially a subtype of a data scientist. At my new company, an ML Engineer is someone who supports ML models but never works on machine learning directly. At this new company, ML Engineers are Data Engineers who support machine learning applications.
What is the generally accepted definition of an ML Engineer?
- andrewmutz 5y agoAt our company, ML Engineers do both the modeling work and the larger engineering that allows those models to interact with the product. So all the models are built by ML engineers, as well as a lot of the production data engineering that gets any needed data to and from the models in production.
- chroem- 5y agoThat's essentially the job role that I'm familiar with, however this new role is everything except for feature and model engineering.
- thom 5y agoThose kinds of full stack ML engineers feel increasingly rare to me. Partly because people from academic backgrounds sometimes lack software engineering knowledge, and vice versa. Partly because it’s just a lot of stuff to do well.
- tfehring 5y agoThose full stack roles are still relatively common (though decreasingly so, as you pointed out) at companies with small data science orgs, on the order of 10 people. But they're still usually called "data scientist," while I think of more specialized roles like MLE, ML ops, data engineer, data ops, etc. as cropping up in bigger and/or more mature data science teams. Calling it MLE instead of data scientist could be a screening technique though.
- sedeki 5y agoMy subjective view coincides with your new employer. I think that's pretty common. ML Scientist and Data Scientist versus ML Engineer and Data Engineer.
- navbaker 5y agoSame at my fairly large research center. We also use ML researcher to denote folks who are hands on with model building and algorithms.
- toomuchtodo 5y agoSounds like their title for the job should’ve been ML Infra or ML DevOps engineer to indicate it’s a supporting role versus a primary practitioner. Annoying, sorry OP.
- pmalynin 5y agoIts whatever you want it to be, which is the greatest thing. Ex ML Engineer at a company.
- wodenokoto 5y agoI'd say what you've mentioned are the 2 typical definitions. Either its an expert at building and training ml models, or its an expert at putting ml models in production.
- benrawk 5y agoYeah I think in general “ML engineers” are building models (can include feature engineering) and then deploying them into production.
- discordance 5y agoWe call that role ML Ops at our work. It's a mix of data eng and dev ops.
- deleted 5y ago[deleted]
- malux85 5y agoI would say, some combinations of: - Helping with feature engineering at scale - Putting models into production - All of the environmental stuff (Framework versions, security patching, etc) - Monitoring models and model drift - Supporting hot/swapping models with zero downtime - Model persistence, A/B testing and evaluation in production - Distributed hyperparameter tuning
- MattGaiser 5y agoI work for a company where half the company consists of ML Developers. They match your definition. Your job is what we call ML Ops. Kind of interested how you got hired to a completely different role though and how that didn't come up in the interview.
- kajecounterhack 5y agoI think it depends on the company. In my experience most companies have it halfway between the things you described (some data engineering + some modeling). Companies with less established infra generally have more data engineering work. Imo unless you are a rich company or have a well funded research arm, it mostly seems wasteful to make someone's job pure modeling.
- disgruntledphd2 5y agoPure modelling teams are a pointless waste of space. It's entirely analogous to setting up an SQL team and then telling everyone else not to run queries. More generally, in order to provide useful solutions, you need some domain and data expertise, which tends to come from engaging with product, the business and the data. If you don't have that, you may as well replace your modelling team with a giant loop over the sklearn fit and predict API.
- sohei 5y ago> I was shocked to learn that my new employer has a very different job definition for ML Engineer than the one I was familiar with. Job titles do not have consistent meaning across regions or industries. People use buzzwords and hype to recruit funding and talent. You've learned an important lesson: use interviews to gather information about companies and teams. "Can you describe in broad strokes a typical project for this role?" is a perfectly reasonable question to ask a hiring manager.
- alex_g 5y agoThis 1000%. I was an ML engineer, and to me it was far more about data management/cleaning than nitty gritty algorithms improvements, but that's definitely not industry standard. I've also found "Product Engineer" job listings to range from essentially just a backend engineer to a product designer who codes.
- WnZ39p0Dgydaz1 5y agoThis. I would add that role titles also exist for negotiation purposes. For example, a new employee may prefer to be called "research engineer" instead of "ML DevOps" to open up future career opportunities, even if the work done is the same.
- travisjungroth 5y agoHow did you get through a job description and the interview process without this coming up? Was it a bait and switch?
- chroem- 5y agoThe job description was pretty ambiguous, and the interview consisted of pretty standard ML/math questions with some data engineering mixed in. There was one brief offhand comment during the interviews that confused me about the role, but since this was a much larger and more established tech company than my previous job, and since the salary was much higher, I figured that they knew what they were doing and also there was no way they could justify that kind of compensation for someone who wasn't an ML practitioner. What I am finding out is that this is a very large team of data engineers who are attempting to graduate to ML engineers, but I'm not sure how familiar they are with this space. The purpose of their organization seems to be to support an offshore data science team in Singapore.
- iamAy0 5y agoI'm on the same boat, but about SRE. Basically every company have their own definition. For some companies, a SRE is just a traditional software engineer who happens to know how to use Kubernetes, where for others, a SRE is a "modern systems guy", with focus on IaC and automation. Guess you're experiencing the same but regarding ML.
- 2511 5y agoAnd then there are many companies where there are still silos, almost no automation, systems engineers are more focused on machines than the product and the company advertises SRE role- simply because its the trend.
- blihp 5y agoIt may be definition differences between the two companies but here's the first thing that came to mind: by any chance is your new employer a much larger organization? In smaller companies you get to wear many hats and typically have responsibility over a wider stretch of <work/tools/technologies>. In larger companies things tend to be much more siloed. So in a larger organization, the things you did in your previous position would likely be spread across multiple positions and teams.
- siliconc0w 5y agoDesigning the systems that run the 'ML pipeline', or end to end data collection > train -> deploy -> infer -> improve. I don't see 'ML Engineers' as advancing the state of the art but they should be fluent in recent techniques, know how to apply them, optimize them, know hardware is needed to efficiently train, etc.
- PaulHoule 5y agoIf it were my business or my team I would want an ML Engineer to relentlessly knock down any barriers to getting models into production. The deployment of ML is still per-paradigmatic in that people today deploy products based on luck, blood, sweat and tears. Many projects fail and many more never get started. I am sure that somebody somewhere had a team that gets results consistently, I can partially picture what it would take to do so, but that's not the usual condition. (That team is so productive that it doesn't need to hire more people!) For instance, you might not want to do any of these things, but any of them could be essential to putting something in production: * manually classifying 10,000 images or documents * figuring out how to run a program in the Java and C++ debuggers at the same time * be able to convince somebody that their research programme is a dead end * survive a 40 minute build process * push back against a 40 minute build process and reform it * reproduce a result from a scientific paper * get a "data scientist" (who thinks he is too good to listen to you, or even to the CEO) to standardize their python distribution so their work isn't wasted * find the scalability bottleneck in a system, where it is, not where you want it to be. Many teams fail at ML because there was an essential task that nobody wanted to do that didn't get done. People who do these things today will be able to say tomorrow that they were part of a successful team. Other people will expect great praise because they diddled around with Word2Vec for a summer. Some day the social structures will be in place and they might have a role made for you that will make you productive but for now you must ask: who do you want to be?
- acidbaseextract 5y agoI overall agree with the sentiment on delivery and needing to deal with a variety of issues, but one nit: If it were my business or my team I would want an ML Engineer to relentlessly knock down any barriers to getting models into production. My favorite is when they knock down the question of whether a certain project even needs ML to focus on getting ML into production. Many teams fail at ML because there was an essential task that nobody wanted to do that didn't get done. Many teams fail at ML because there was a task that didn't need ML (or at least anything more than a linear model) that is made opaque to anyone but an ML engineer after they implement it.
- craftinator 5y ago
- solidasparagus 5y agoI think it often depends on how large the team is. On a team with many developers per model, ML Engineer to me means someone who does the engineering to support models and is distinct from a researcher or a scientist. ML engineers will not do the science side of the equation (model architectures, optimizing scientific metrics like accuracy) except for when their engineering work impacts the science side (e.g. figuring out how to optimize model throughput without impacting scientific metrics). On smaller teams, the titles mean less - but I rarely see ML engineer positions for small teams. ML involves both science and engineering. ML engineer by definition sits on the engineering side of the fence, but how much of the science they are involved in will vary by company.
- aaccount 5y agoMost of the time they allow the company to say they work on this new flashing buzzword tech while burning through capital doing stuff a few if else statements could easily do
- RhysU 5y ago>... never works on machine learning directly. Meaning, does not push the train button and watch loss curves? What's out of this role's scope at the new company?
- chroem- 5y agoAt my last role I designed the forecasting systems for a major retail chain, which required inventing a new deep learning model architecture for time series problems, due to some peculiar properties of their sales data. I take great pride in my work.
- forgingahead 5y agoDo you know of any publicly available resources that walk through some intuitions in using forecasting systems in retail sales? Basically a dummy's guide that helps non-tech retail leaders better understand how this works, how accurate it can be, and when is appropriate to use it (and when it isn't).
- chroem- 5y agoFor business users, unfortunately none on hand at the moment. That being said, one of the biggest issues companies seem to encounter is that forecasting requires logging of at least several years of business operations, especially if the target variable is highly seasonal. The need for sufficient historical data can make it difficult to commit to a new forecasting initiative at the drop of a hat. There first needs to be a culture of retaining as much data possible, since even external variables from other business areas may be useful in making predictions. You can get around this somewhat with synthetic data and piecing together alternative data sources, but it's not ideal. I would also say the ease of implementing a forecasting solution on a given dataset is inversely proportional to the value it provides. They're mainly useful for situations in which there are either too many independent time series for a group of human business analysts to track, or for situations in which the target variable is unruly for some reason. For forecasting problems that are simple enough to be modeled by state space methods or BSTS, those methods won't be able to provide results that are all that superior to what an experienced analyst could produce, since the series data itself is already quite predictable. Edit: Also, there is no one-size-fits-all approach when it comes to time series, and anyone who tells you differently is trying to sell you something. Time series is weird within ML because each dataset has unique properties that necessitates the use of potentially wildly different modeling techniques.
- reureu 5y agoMy understanding is ML engineer gets machine learning models running in production. They would own the infrastructure that enables teams to deploy ML models, interface with engineering teams to agree on contracts/endpoints/UI/etc, and work on scaling models when/if that time comes. I would view Data Scientist and ML engineer as distinct sister roles, and not as a subset. I've worked in environments where ML engineers were "data science plus", and those were really unhealthy work environments - it ended up feeling like a prestige war with lots of cookie licking and turfing. When you're at the point of needing differentiated roles, you really need someone to focus on infrastructure to deploy ML models and someone else to focus on managing stakeholders, wrangling data, and developing proof of concepts.
- giantg2 5y agoIn my experience, if you have a PhD you can get a job working with the models. It will likely be called something fancier. Without the PhD, I feel like you're just a data jockey. Just clean the data and create APIs for the PhDs and consumers to use.