9 ms·

MLOps is mostly data engineering

169 points by dpbrinkm 4y ago

kfk 4y ago
We are experimenting with workers using the simple python arq library and Redis and I am yet to find a MLOps or Data Engineering use case that is not a good fit for a API+Worker on K8S. For instance, you need to manage ML artifacts? You can just offer an API endpoint so the ML models can automatically update the artifacts. You need data ingestion? You can have a worker running ingestion scripts and kick off the worker via API. We tried pub/sub and Kafka but it can be really wasteful, workers can process work for multiple streams, but Kafka cannot. But of course I wonder if I am missing something, I am not an ML engineer so probably I am?
- alextheparrot 4y ago
  It isn’t particularly clear what technical requirements you were working against, but let me give it a shot: A lot of groups start using Kafka because they have high-throughput event streams they want to aggregate over, and then you just use Kafka for everything because managing a 5 TPS topic alongside a 100k TPS topic is trivial. In terms of why Kafka is a good fit for that workflow — database writes are unnecessary and oddly structured for raw events we want to expire in a few days, and dealing with buffering blob file writes can cause data loss, so Kafka can really simplify the producer architecture where the producer is also a consumer from a producer who wants an ack. Combine this with how trivial it is to have multiple readers on a pub sub system, and it is easy to scale from 1 to N consumers of a data source without duplicating the data everywhere. E.g. you could have three aggregation jobs that use the same data, one job that writes the topic’s data to blob-style storage for batch use-cases, and a low latency inference job all running from the same data stream. More or less, Kafka just simplifies scale-out in some cases, maybe not your case, though. If you’re kicking off workers to do ingestion, you might have a system where you are pulling files down at some infrequent cadence (Let’s say every 10 minutes) —- in that case Kafka is likely going to be overkill and feel like a lot of work for an API call that then becomes a tasked tracked in some database.
  - kfk 4y ago
    interesting, in my case I need the events to be inserted in a SQL db, even the real time ones. For instance, I receive Contacts data from Hubspot in realtime, I send those contacts over to Salesforce and I store them in Postgres. Why Postgres? Because we want to keep a history of the contacts, plus we will need to have 1 source of truth for customers data to fulfill various data privacy requirements. How about Kafka for such a use case? Let's say I receive maybe 10,000 contacts per day.
    - geoduck14 4y ago
      Not your OP, but someone who has been thinking about this. Use Kafka as the super fast layer to connect producers and consumers. Have it write to S3/Blob. Then insert into your dB layer for "cold" access. Inserting into a dB takes longer than pub/sub, and you don't want to mess up the publishing of data.
Longwelwind 4y ago
I've been an MLOps Engineer for around 3 years now and I mostly agree with the article. There is a big overlap between the ML-specific tools that are popping up on the market and traditionnal Data Engineering tools, and I think people are not always realizing that: * Prometheus/Grafana/TSDB/... can be used to setup a model monitoring platform since you're observing metrics whether they are from an ML service or a normal service. * Any service deployment tool can be used to deploy ML models, since they are services. * AirFlow/Dagster/... can be used to orchestrate model training, since training a model is basically a data engineering task. With that said, I still believe that there is space for ML-specific tools to be created. * Model Monitoring tools (ArizeAI is the only one I've used) can be tailored to be easily usable by ML Engineers without requiring DE knowledge. * Deploying models in production has some specifities: things like GPU support, adaptive batching, ... Those specifities can be implemented inside a model deployment tool. * Training orchestration is the only domain where I think there's truly no need for new tools.
- sandkoan 4y ago
  You may want to look at run.house [0] for a pretty powerful solution to many of these problems. [0] https://github.com/run-house/runhouse https://github.com/run-house/runhouse
- mountainriver 4y ago
  Training orchestration does need new tools to spin up GPU instances and make the most of them and then spin them down, we are still struggling in this domain
  - pid-1 4y ago
    Why isn't that solved by k8s + a node autoscaler such as Karpenter?
    - rfoo 4y ago
      That's a viable solution, but since GPU instances are expensive, you really want to make most of it. Ideally the GPU should be busy within 30 seconds of instance launch. Okay, so, where is your training data? Is your training data in the layout which your training code can just linearly scan on S3? Or you have to transform them first? Or provision a dataset cache on-demamd? Is this data engineering or training orchestration?
jstx1 4y ago
This is a strange article. The body of the article correctly talks about all the model work... but data engineers typically don't have to do any of that work. So it's an okay overview of some ML engineering / ops things with a contradictory title which isn't followed up on (and which I'm sure gets more clicks). So no, MLOps isn't just data engineering. For more information read your own article.
- antonvs 4y ago
  Betteridge’s Law of Headlines
- cpard 4y ago
  Hey, thanks for the feedback! There is a reason the article has the structure it does. I'm going through the main Pilars of MLOps and explaining how they overlap with data engineering. Also, the title says, "Mostly" not "just" data engineering. It might be misleading if you just go through the sub-headers but I'm sure if you go through the content you will see that the title and the content are pretty aligned.
steveBK123 4y ago
Data Engineering with a top hat & bow tie, really.
chatmasta 4y ago
If a buzzword is a portmanteau of a previous buzzword (DevOps), combined with a newly hot buzz word (ML), then chances are it's something in disguise. But that doesn't make it any less legitimate - DevOps came from Dev + SysOps, but nobody is arguing DevOps shouldn't be a thing (although you might argue it's no different from SysOps). In general, buzzwords align pretty closely to VC funding cycles.
- pydry 4y ago
  DevOps was associated with a new generation of sysops with stuff like managed infrastructure, IaaC, continuous deploys, etc. It was a whole new generation of that thing. As far as I can see MLOPs is just equivalent generation devops applied to ML.
mountainriver 4y ago
This article feels like it’s written by someone who doesn’t understand the problems faced by ML. These people come through the MLops community every so often, they think no one has realized that dev ops and DE are similar, when in reality they just don’t yet realize how different ML is yet. For one, the customer is entirely different. You are mostly serving data scientists who don’t have strong engineering skills, which dramatically skews the solutions toward things like Python and Jupyter. This is a big reason why the tool space is different and has been successful at what it does. Model training and serving are absolutely nothing like traditional methods. In serving, you are deploying a stateful model, not a stateless backend. That model’s state should ideally be continuously trained, requires different scaling and monitoring capabilities. In training, the GPU problem is far from solved and it is unique to ML with things like how you shard models and fit their weights into memory. There are extremely challenging problems in this space that simply aren’t the same as devops, and this is coming from a former k8s contributor.
- mylons 4y ago
  as someone who’s been in the industry for 20 years, and worked in ML data engineering at twitch, i think you’re just using new words (some are tools but so what) to describe moving data around and making it accessible to people who need it.
  - eropple 4y ago
    Yeah - I've never been a "data engineer", but I've been doing this sort of shoveling for twenty years. We called it "ETL" and felt faintly foolish blogging about it.
    - mylons 4y ago
      i think ETL is the most timeless phrase because it's not tied to a trend. devops, data engineering, ML engineering, MLOps are all just lipstick on the pig.
  - mountainriver 4y ago
    Yes I've been an engineer for 15 years at a bunch of big companies. MLops isn't ML data engineering. I doubt data engineering has many changes outside of active learning.
mmierz 4y ago
I'm currently working in an MLOps engineer role at a mid sized company. I agree with that article that most of what I do is plain old software engineering. I don't think I'm interchangeable with any other backend dev though, because ML expertise really does come in handy here. I think the thing that makes it a bit specialized is that we are providing tools to allow our data scientists to self-serve model deployment and monitoring, but they by and large not expert web programmers. So we need to anticipate the kind of mistakes they're likely to make and provide opinionated tools that guide them into building sane software in the specific context of our company's technology. As well as direct support as needed. We evaluated several commercial MLops tools and ended up going with generic tools that we already use, instead of something new that's branded for MLops. I.e. postgres + snowflake instead of a commercial feature store -- model deployment, monitoring, and alerting on the same platform as the rest of the company's applications -- etc. When we tried "ML" tools, they took so much work to adapt to our use cases that they really added no value.
ritzaco 4y ago
I agree there's a lot of "Agile" like BS around MLOps, but this article doesn't really give the prior art enough attention IMO. Data Engineering is a large part of MLOps, but there are unique parts of production ML engineering so it makes sense that it is (slowly) evolving its own discipline. There are some people who have expertise in building production infrastructure, writing production code, and managing production data, but they are few and far between. So finding a "system" that lets data experts work with code experts work with infrastructure experts is important. Many people say it's "just software engineering" or "just DevOps", but I feel like they are either not respective enough of the challenges of whichever pillar they are ignoring, or they don't even know that those challenges exist. Filtering out the BS and finding the smart people who are writing interesting things about MLOps is difficult as they use the same terminology (and if the smart people switched, the BS people would follow, so they may as well stand their ground) but the BS cover doesn't mean that there's nothing substantial underneath.
- nonethewiser 4y ago
  Couldn’t you say it’s just devops applied to a specific type of app that often isn’t deployed to production? And takes significant understanding of the underlying systems to deploy? It certainly feels like ML discovering devops. That can still mean it requires a unique skillset that devops people dont generally have. It’s just hard to imagine how infrastructure and deployment suddenly isn’t devops because it deals with ML.
  - ritzaco 4y ago
    I mean there has definitely been an explosion of "Ops" terms, and not all of them are justified. I think MLOps makes sense to emphasise the three pieces though. I know people want the "Ops" part of "DevOps" to mean "magic optimization for anything", but it goes back to "ops" as in infrastructure/servers etc. So if DevOps is software engineering + deploying and running software reliably, then it still leaves a big gap on the "data" side - both in terms of orchestrating source data, but also in handling large model files, and tracking things like model decay.
vrglvrglvrgl 4y ago
[dead]
cpard 4y ago
Hey folks I'm the author of the post and happy to see that it gets so much attention on HN. Thank you for the incredible comments! I want to clarify something about my intention with this post. There is a reason I chose "mostly" on the title. I'm not dismissing the different needs of ML. if a category withstands the tests of the market, then there's good reason for it to exist. But, we have ended up creating silos within orgs with fundamentally aligned goals because of the way we build products and companies around them. What I'm advocating for in this article, is the need to think more holistically when we design and build data infra tooling. Yes ML has unique challenges but these challenges won't be addressed by reinventing everything again and again. Tooling should be built having in mind all the practitioners involved in the lifecycle of data. It's harder to do but at least we'll stop wasting our time building one Airflow copy after the other that is doomed to fail.
- jamesblonde 4y ago
  Here's one thing that you missed - transformations. Data engineers hear the word transformations and think they know what they mean - aggregations, binning, data reductions, cleansing, etc. Data scientists, however, think preparing features for models with transformations - encoding categorical variables (one-hot-encoding, LabelEncoding, OrdinalEncoding) and normalizing/standardizing/log-transforms for numerical features. These transformations are so different that some of them are "model-independent" - you can do the transformations and reuse the output feature across many models (aggregations, binning, feature-crosses, embeddings, etc), but the data scientists transformations are model-dependent (and not reusable across different models - e.g., a XGBoost model doesn't want normalized numerical features, but a DNN typically needs normalized or standardized numerical features. These differences are reflected in how we build our "ML pipelines" - we split them into feature/training/inference pipelines, enabling us to localize model-independent transformations to feature pipelines, while doing model-dependent transformations in training/inference pipelines (while ensuring no skew). In summary, the devil's in the details in pipelines for ML. I agree, however, that orchestrators like Airflow are good enough for orchestration. However, the ML Assets (mutable & reusable features, immutable models, immutable training/inference datasets) are different and tooling will ultimately reflect those differences.
nonethewiser 4y ago
Consider that there are different types of developers. This remains true in the context of “Devops”. Devops doesn’t mean web dev ops or something. Given that, how is MLops not a certain type of devops? It’s basically ML engineers figuring out how to deploy their systems to production, no?
- KptMarchewa 4y ago
  "Devops" only means terraform monkey now. Just as "data engineer" is python/sql monkey.
Kalanos 4y ago
https://docs.aiqc.io https://docs.aiqc.io systematically orchestrates the data preprocessing and post-processing of the training loop for multi-dimensional data and various types of analysis
jamesblonde 4y ago
IMO, this article misses the essence of and principles of MLOps. The essence of MLOps is that it is about processes (and tooling/platforms) for creating ML assets - features/labels and models. We call them FTI pipelines. Data engineering has data pipelines that produce datasets for consumption. In MLOps, feature pipelines produce features (from raw data). Training pipelines produce models (from features/labels). Inference pipelines produce predictions (from models + features). There is no such thing as a "ML Pipeline" in a production ML system. There is no ML pipeline that goes from raw data to predictions. We have the above FTI pipelines (feature/training/inference pipelines). The principles of MLOps are around being able to develop faster (shorten the development lifecyle) through automated testing and versioning. You need to validate data to build features. You need tested features to build models. You need to test models for your ML systems. It's a hierarchy: data->features->models-ML Apps. Versioning is needed for features and models in order to safely upgrade systems and enable them to evolve over time. I cover a lot of this in a course i developed called 'serverless ml'.
jldugger 4y ago
Side note: was this a tweetstorm originally? Literally every paragraph is a single sentence, often long ones with no reader afforrdances like bolding key points.
stuartaxelowen 4y ago
Feature stores are essentially materialized views (aside from any realtime feature resolution needed). I think it's a good thing that there is specialized effort being taken here, though: features stores are an abstraction that could be useful in other domains also, and this surge in interest is an opportunity for us to make better tools.
antipaul 4y ago
The overlap in "deployment" between MLOps and Software engineering was hinted at in that well-known 2015 NeuriIPS paper, "Hidden Technical Debt in Machine Learning Systems": "It may be surprising to the academic community to know that only a tiny fraction of the code in many ML systems is actually devoted to learning or prediction – see Figure 1" https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd... PDF: https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd...
PaulHoule 4y ago
(1) The killer product encompasses “all of the above”, if you really are going to buy five of them God help you because with all the mistakes vendors will make and you’ll have to work around plus the overhead of moving data around you’re in for it. (2) A major difference w/ the conventional software development CI/CD pipelines is the sheer size of the data involved. When you are dealing with “tiny data” you can waste resources on Docker, but when your foundation model is 100x the size, when the training process is distributed, and takes a day, the quantity is taking on a quality of its own. (3) The worst performance sin is moving data around although this will be necessary so far as the system is distributed. Avoiding excess data moving can be the difference between training a model and failing to train a model, but when you put together a patchwork of ML ops programs you will fin they are moving data around internally for good reasons sometimes and no reason other times plus the easy (and sometimes only) integration method is moving data around. Don’t be that guy!
danthelion 4y ago
Yes, and Data Engineering is just Software Engineering "in disguise"
- rickette 4y ago
  Exactly my thought.
- marcyb5st 4y ago
  While I agree with the premise, there's the added "complication" that knowledge or experience as a Data Scientist/ML Engineer it's highly beneficial. Pure SWEs, in my experience, struggle a bit in anticipating how Data Scientists will shoot themselves in the feet. Having experienced it yourself before it's a big plus.
- rldjbpin 4y ago
  underrated comment. of course i agree that there are specificities with mlops and data engineering that don't exist for a typical full-stack application. but if you boil it down enough, you just find the same underlying concepts.
- nerdponx 4y ago
  Or database admin.
  - nerdponx 4y ago
    I guess I should clarify because I was downvoted. Data engineering involves a lot of database work. So if ML engineering is largely data engineering, then it's also at least partially database administration, transitively.
o10449366 4y ago
Sure, MLOps is just Data Engineering in disguise when you ignore the complexities of hardware provisioning, GPU optimization, integration tests for model performance and quality, benchmarking, resource constraints (network, disk, memory, GPU memory), etc. Anecdotally, I've worked in high-performance computing and machine learning for years now and the past few months I've seen a huge spike in the number of messages I get for MLOps positions. I think companies are slowly starting to realize that setting up machine learning at scale isn't as simple as deploying poorly written code by research scientists to managed platforms.
- dpbrinkm 4y ago
  >I think companies are slowly starting to realize that setting up machine learning at scale isn't as simple as deploying poorly written code by research scientists to managed platforms. This is some truth right here
- pydry 4y ago
  >complexities of hardware provisioning This is hardly an uncommon problem outside of machine learning.
  - o10449366 4y ago
    You're generally right, but I think the strong dependency on GPUs with specific architectures and capabilities and how scarce they are is a fairly unique problem to ML (outside of crypto mining lol.)
- nonethewiser 4y ago
  > Anecdotally, I've worked in high-performance computing and machine learning for years now and the past few months I've seen a huge spike in the number of messages I get for MLOps positions. I think companies are slowly starting to realize that setting up machine learning at scale isn't as simple as deploying poorly written code by research scientists to managed platforms. This doesn’t bode well for ML in lots of orgs. Obviously machine learning is very powerful and effective in many use cases. But the value proposition already isn’t there for lots of companies. At such places, discovering a hidden requirement for more resources is a great reason to change directions. To be very clear, I’m not talking about the field in general. I’m talking about orgs where management doesn’t see value generated from the ML efforts that suddenly demand more resources to operate.
hgsgm 4y ago
Is ML just statistics (data analysis) in disguise?
- jstx1 4y ago
  They're distinct, there isn't all that much analysis in ML.
- version_five 4y ago
  No. But corporate "ML" has been until recently "dashboards" or "big data" or "data analytics" or "look at your corporate records with a computer" in disguise.
boredumb 4y ago
Learning some pytorch is truly not the bulk of the work to build a model, having to wrangle and mangle a massive amount of data coming from less-than-ideal data sources, orchestrating the jobs and making all of this available for your training routines to slurp up is a lot of work that evolves fairly quickly beneath you.
- PartiallyTyped 4y ago
  I wonder whether you could get a GPT-3 level LLM to extract the data for you.
  - PaulHoule 4y ago
    not fast enough, but maybe it can help write the program that does.
    - PartiallyTyped 4y ago
      Definitely not realtime, but perhaps it could work as a preprocessing step?
      - PaulHoule 4y ago
        I worked for a startup that developed CNN models https://www.davidsbatista.net/blog/2018/03/31/SentenceClassificationConvNets/ https://www.davidsbatista.net/blog/2018/03/31/SentenceClassi... for early processing of data, you can think of it as a profiling tool that works at the level of an individual “cell” in a table, so given a number like 90214 it will guess that it could be a zip code, given “John” it would guess it is a proper name, etc. I’d contrast that to tools that profile a whole column at a time (funny… these are all 5 digit zero padded numbers) but that column is full of numbers consistent with https://en.wikipedia.org/wiki/Benford%27s_law https://en.wikipedia.org/wiki/Benford%27s_law This system was successful for a few customers including a major telecom and ultimately the company got bought by a major shoe and clothing brand. We had an “MLOps” system we developed internally for training those CNN models and we trained some for that task and trained some for other tasks. There were numerous problems with the models that we were concerned about then, most of them have been solved by transformer models which we were just starting to read about then.
noobcoder 4y ago
Before entering the field of ML, I perceived MLOps as a superhero with abilities to handle and deploy ML models. However, it seems that MLOps is more or less a typical engineer who acquired skills to manage and deploy data infrastructure for ML purposes (exclusively) by exposure to data engineering.
- mountainriver 4y ago
  Eh that’s not what I’ve seen, non deterministic models have a whole class of unique challenges
- fest 4y ago
  > However, it seems that XYZ is more or less a typical engineer who acquired skills.. This is true for pretty much every engineering discipline/role.
bobbruno 4y ago
On a first read, I could agree with most, if not all, of the author’s arguments. But there are two aspects that were simply left out that I have to consider when doing MLOps, which can prove too complex for just saying “It’s an extension of tooling that data engineering already has”. First, there’s the matter that ML introduces a significant stack and complexity into what was already a relatively complex framework. I mean, managing storage, quality, data processing, streaming, scheduling/orchestration, transformation logic and SLAs requires a lot of tools, whichever combination pleases you the most. Even full platforms offered by some of the players in this market can get quite complex, and it’s very hard to set everything right and handle all the cases. Specialised tooling or skills is probably a good idea to focus on the things that matter to Ml and that go beyond what DE already covers. Think of all the frameworks, the statistical libraries, the different nature of the logic for ML features when compared to regular reporting needs, the different quality requirements and structure that ML expects, managing versions of raw, labelled and test datasets, etc. (there are many more, the discussion already covered quite a few). Which brings me to the second thing - the knowledge stack required to run ML. Besides some of the usual DE stack (developing, data manipulation, quality, etc), a whole new set of skills, related to several branches of math, parallelism, very complex and costly infrastructure management, research skills, experiment design, specific algorithms and approaches (does the regular DE need to understand neural network patterns, how data and model-parallel training works, the statistics behind setting up and running drift measures, what all the metrics behind model performance mean, etc.?). I find this a better reason for specialisation than any other - there’s just so much one person can hold in their heads, and ML development and operations is just getting more complex by the day. So, my point is, from a very simplified and abstract perspective, the author seems right. But, in practice, you won’t be able to just stack that on top of data engineers and not expect them to become specialized - and that’s where the ML Engineer and MLOps engineer roles are emerging. They’re not completely new, but they’re no longer your regular data engineer or data scientist.
achileas 3y ago
alwayshasbeen.jpg I was doing this work since before MLOps was the new buzzword in town, and it was always under the data engineering job title. It was only in the past few years that data engineering has become more focused, requiring new titles/job descriptions to truly cover the different specializations.

169 points by dpbrinkm 4y ago

kfk 4y ago

We are experimenting with workers using the simple python arq library and Redis and I am yet to find a MLOps or Data Engineering use case that is not a good fit for a API+Worker on K8S. For instance, you need to manage ML artifacts? You can just offer an API endpoint so the ML models can automatically update the artifacts. You need data ingestion? You can have a worker running ingestion scripts and kick off the worker via API. We tried pub/sub and Kafka but it can be really wasteful, workers can process work for multiple streams, but Kafka cannot. But of course I wonder if I am missing something, I am not an ML engineer so probably I am?

alextheparrot 4y ago
It isn’t particularly clear what technical requirements you were working against, but let me give it a shot: A lot of groups start using Kafka because they have high-throughput event streams they want to aggregate over, and then you just use Kafka for everything because managing a 5 TPS topic alongside a 100k TPS topic is trivial. In terms of why Kafka is a good fit for that workflow — database writes are unnecessary and oddly structured for raw events we want to expire in a few days, and dealing with buffering blob file writes can cause data loss, so Kafka can really simplify the producer architecture where the producer is also a consumer from a producer who wants an ack. Combine this with how trivial it is to have multiple readers on a pub sub system, and it is easy to scale from 1 to N consumers of a data source without duplicating the data everywhere. E.g. you could have three aggregation jobs that use the same data, one job that writes the topic’s data to blob-style storage for batch use-cases, and a low latency inference job all running from the same data stream. More or less, Kafka just simplifies scale-out in some cases, maybe not your case, though. If you’re kicking off workers to do ingestion, you might have a system where you are pulling files down at some infrequent cadence (Let’s say every 10 minutes) —- in that case Kafka is likely going to be overkill and feel like a lot of work for an API call that then becomes a tasked tracked in some database.
- kfk 4y ago
  interesting, in my case I need the events to be inserted in a SQL db, even the real time ones. For instance, I receive Contacts data from Hubspot in realtime, I send those contacts over to Salesforce and I store them in Postgres. Why Postgres? Because we want to keep a history of the contacts, plus we will need to have 1 source of truth for customers data to fulfill various data privacy requirements. How about Kafka for such a use case? Let's say I receive maybe 10,000 contacts per day.
  - geoduck14 4y ago
    Not your OP, but someone who has been thinking about this. Use Kafka as the super fast layer to connect producers and consumers. Have it write to S3/Blob. Then insert into your dB layer for "cold" access. Inserting into a dB takes longer than pub/sub, and you don't want to mess up the publishing of data.

Longwelwind 4y ago

I've been an MLOps Engineer for around 3 years now and I mostly agree with the article. There is a big overlap between the ML-specific tools that are popping up on the market and traditionnal Data Engineering tools, and I think people are not always realizing that: * Prometheus/Grafana/TSDB/... can be used to setup a model monitoring platform since you're observing metrics whether they are from an ML service or a normal service. * Any service deployment tool can be used to deploy ML models, since they are services. * AirFlow/Dagster/... can be used to orchestrate model training, since training a model is basically a data engineering task. With that said, I still believe that there is space for ML-specific tools to be created. * Model Monitoring tools (ArizeAI is the only one I've used) can be tailored to be easily usable by ML Engineers without requiring DE knowledge. * Deploying models in production has some specifities: things like GPU support, adaptive batching, ... Those specifities can be implemented inside a model deployment tool. * Training orchestration is the only domain where I think there's truly no need for new tools.

sandkoan 4y ago
You may want to look at run.house [0] for a pretty powerful solution to many of these problems. [0] https://github.com/run-house/runhouse https://github.com/run-house/runhouse
mountainriver 4y ago
Training orchestration does need new tools to spin up GPU instances and make the most of them and then spin them down, we are still struggling in this domain
- pid-1 4y ago
  Why isn't that solved by k8s + a node autoscaler such as Karpenter?
  - rfoo 4y ago
    That's a viable solution, but since GPU instances are expensive, you really want to make most of it. Ideally the GPU should be busy within 30 seconds of instance launch. Okay, so, where is your training data? Is your training data in the layout which your training code can just linearly scan on S3? Or you have to transform them first? Or provision a dataset cache on-demamd? Is this data engineering or training orchestration?

jstx1 4y ago

This is a strange article. The body of the article correctly talks about all the model work... but data engineers typically don't have to do any of that work. So it's an okay overview of some ML engineering / ops things with a contradictory title which isn't followed up on (and which I'm sure gets more clicks). So no, MLOps isn't just data engineering. For more information read your own article.

antonvs 4y ago
Betteridge’s Law of Headlines
cpard 4y ago
Hey, thanks for the feedback! There is a reason the article has the structure it does. I'm going through the main Pilars of MLOps and explaining how they overlap with data engineering. Also, the title says, "Mostly" not "just" data engineering. It might be misleading if you just go through the sub-headers but I'm sure if you go through the content you will see that the title and the content are pretty aligned.

steveBK123 4y ago

Data Engineering with a top hat & bow tie, really.

chatmasta 4y ago

If a buzzword is a portmanteau of a previous buzzword (DevOps), combined with a newly hot buzz word (ML), then chances are it's something in disguise. But that doesn't make it any less legitimate - DevOps came from Dev + SysOps, but nobody is arguing DevOps shouldn't be a thing (although you might argue it's no different from SysOps). In general, buzzwords align pretty closely to VC funding cycles.

pydry 4y ago
DevOps was associated with a new generation of sysops with stuff like managed infrastructure, IaaC, continuous deploys, etc. It was a whole new generation of that thing. As far as I can see MLOPs is just equivalent generation devops applied to ML.

mountainriver 4y ago

This article feels like it’s written by someone who doesn’t understand the problems faced by ML. These people come through the MLops community every so often, they think no one has realized that dev ops and DE are similar, when in reality they just don’t yet realize how different ML is yet. For one, the customer is entirely different. You are mostly serving data scientists who don’t have strong engineering skills, which dramatically skews the solutions toward things like Python and Jupyter. This is a big reason why the tool space is different and has been successful at what it does. Model training and serving are absolutely nothing like traditional methods. In serving, you are deploying a stateful model, not a stateless backend. That model’s state should ideally be continuously trained, requires different scaling and monitoring capabilities. In training, the GPU problem is far from solved and it is unique to ML with things like how you shard models and fit their weights into memory. There are extremely challenging problems in this space that simply aren’t the same as devops, and this is coming from a former k8s contributor.

mylons 4y ago
as someone who’s been in the industry for 20 years, and worked in ML data engineering at twitch, i think you’re just using new words (some are tools but so what) to describe moving data around and making it accessible to people who need it.
- eropple 4y ago
  Yeah - I've never been a "data engineer", but I've been doing this sort of shoveling for twenty years. We called it "ETL" and felt faintly foolish blogging about it.
  - mylons 4y ago
    i think ETL is the most timeless phrase because it's not tied to a trend. devops, data engineering, ML engineering, MLOps are all just lipstick on the pig.
- mountainriver 4y ago
  Yes I've been an engineer for 15 years at a bunch of big companies. MLops isn't ML data engineering. I doubt data engineering has many changes outside of active learning.

mmierz 4y ago

I'm currently working in an MLOps engineer role at a mid sized company. I agree with that article that most of what I do is plain old software engineering. I don't think I'm interchangeable with any other backend dev though, because ML expertise really does come in handy here. I think the thing that makes it a bit specialized is that we are providing tools to allow our data scientists to self-serve model deployment and monitoring, but they by and large not expert web programmers. So we need to anticipate the kind of mistakes they're likely to make and provide opinionated tools that guide them into building sane software in the specific context of our company's technology. As well as direct support as needed. We evaluated several commercial MLops tools and ended up going with generic tools that we already use, instead of something new that's branded for MLops. I.e. postgres + snowflake instead of a commercial feature store -- model deployment, monitoring, and alerting on the same platform as the rest of the company's applications -- etc. When we tried "ML" tools, they took so much work to adapt to our use cases that they really added no value.

ritzaco 4y ago

I agree there's a lot of "Agile" like BS around MLOps, but this article doesn't really give the prior art enough attention IMO. Data Engineering is a large part of MLOps, but there are unique parts of production ML engineering so it makes sense that it is (slowly) evolving its own discipline. There are some people who have expertise in building production infrastructure, writing production code, and managing production data, but they are few and far between. So finding a "system" that lets data experts work with code experts work with infrastructure experts is important. Many people say it's "just software engineering" or "just DevOps", but I feel like they are either not respective enough of the challenges of whichever pillar they are ignoring, or they don't even know that those challenges exist. Filtering out the BS and finding the smart people who are writing interesting things about MLOps is difficult as they use the same terminology (and if the smart people switched, the BS people would follow, so they may as well stand their ground) but the BS cover doesn't mean that there's nothing substantial underneath.

nonethewiser 4y ago
Couldn’t you say it’s just devops applied to a specific type of app that often isn’t deployed to production? And takes significant understanding of the underlying systems to deploy? It certainly feels like ML discovering devops. That can still mean it requires a unique skillset that devops people dont generally have. It’s just hard to imagine how infrastructure and deployment suddenly isn’t devops because it deals with ML.
- ritzaco 4y ago
  I mean there has definitely been an explosion of "Ops" terms, and not all of them are justified. I think MLOps makes sense to emphasise the three pieces though. I know people want the "Ops" part of "DevOps" to mean "magic optimization for anything", but it goes back to "ops" as in infrastructure/servers etc. So if DevOps is software engineering + deploying and running software reliably, then it still leaves a big gap on the "data" side - both in terms of orchestrating source data, but also in handling large model files, and tracking things like model decay.

vrglvrglvrgl 4y ago

[dead]

cpard 4y ago

Hey folks I'm the author of the post and happy to see that it gets so much attention on HN. Thank you for the incredible comments! I want to clarify something about my intention with this post. There is a reason I chose "mostly" on the title. I'm not dismissing the different needs of ML. if a category withstands the tests of the market, then there's good reason for it to exist. But, we have ended up creating silos within orgs with fundamentally aligned goals because of the way we build products and companies around them. What I'm advocating for in this article, is the need to think more holistically when we design and build data infra tooling. Yes ML has unique challenges but these challenges won't be addressed by reinventing everything again and again. Tooling should be built having in mind all the practitioners involved in the lifecycle of data. It's harder to do but at least we'll stop wasting our time building one Airflow copy after the other that is doomed to fail.

jamesblonde 4y ago
Here's one thing that you missed - transformations. Data engineers hear the word transformations and think they know what they mean - aggregations, binning, data reductions, cleansing, etc. Data scientists, however, think preparing features for models with transformations - encoding categorical variables (one-hot-encoding, LabelEncoding, OrdinalEncoding) and normalizing/standardizing/log-transforms for numerical features. These transformations are so different that some of them are "model-independent" - you can do the transformations and reuse the output feature across many models (aggregations, binning, feature-crosses, embeddings, etc), but the data scientists transformations are model-dependent (and not reusable across different models - e.g., a XGBoost model doesn't want normalized numerical features, but a DNN typically needs normalized or standardized numerical features. These differences are reflected in how we build our "ML pipelines" - we split them into feature/training/inference pipelines, enabling us to localize model-independent transformations to feature pipelines, while doing model-dependent transformations in training/inference pipelines (while ensuring no skew). In summary, the devil's in the details in pipelines for ML. I agree, however, that orchestrators like Airflow are good enough for orchestration. However, the ML Assets (mutable & reusable features, immutable models, immutable training/inference datasets) are different and tooling will ultimately reflect those differences.

nonethewiser 4y ago

Consider that there are different types of developers. This remains true in the context of “Devops”. Devops doesn’t mean web dev ops or something. Given that, how is MLops not a certain type of devops? It’s basically ML engineers figuring out how to deploy their systems to production, no?

KptMarchewa 4y ago
"Devops" only means terraform monkey now. Just as "data engineer" is python/sql monkey.

Kalanos 4y ago

https://docs.aiqc.io https://docs.aiqc.io systematically orchestrates the data preprocessing and post-processing of the training loop for multi-dimensional data and various types of analysis

jamesblonde 4y ago

IMO, this article misses the essence of and principles of MLOps. The essence of MLOps is that it is about processes (and tooling/platforms) for creating ML assets - features/labels and models. We call them FTI pipelines. Data engineering has data pipelines that produce datasets for consumption. In MLOps, feature pipelines produce features (from raw data). Training pipelines produce models (from features/labels). Inference pipelines produce predictions (from models + features). There is no such thing as a "ML Pipeline" in a production ML system. There is no ML pipeline that goes from raw data to predictions. We have the above FTI pipelines (feature/training/inference pipelines). The principles of MLOps are around being able to develop faster (shorten the development lifecyle) through automated testing and versioning. You need to validate data to build features. You need tested features to build models. You need to test models for your ML systems. It's a hierarchy: data->features->models-ML Apps. Versioning is needed for features and models in order to safely upgrade systems and enable them to evolve over time. I cover a lot of this in a course i developed called 'serverless ml'.

jldugger 4y ago

Side note: was this a tweetstorm originally? Literally every paragraph is a single sentence, often long ones with no reader afforrdances like bolding key points.

stuartaxelowen 4y ago

Feature stores are essentially materialized views (aside from any realtime feature resolution needed). I think it's a good thing that there is specialized effort being taken here, though: features stores are an abstraction that could be useful in other domains also, and this surge in interest is an opportunity for us to make better tools.

antipaul 4y ago

The overlap in "deployment" between MLOps and Software engineering was hinted at in that well-known 2015 NeuriIPS paper, "Hidden Technical Debt in Machine Learning Systems": "It may be surprising to the academic community to know that only a tiny fraction of the code in many ML systems is actually devoted to learning or prediction – see Figure 1" https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd... PDF: https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd...

PaulHoule 4y ago

(1) The killer product encompasses “all of the above”, if you really are going to buy five of them God help you because with all the mistakes vendors will make and you’ll have to work around plus the overhead of moving data around you’re in for it. (2) A major difference w/ the conventional software development CI/CD pipelines is the sheer size of the data involved. When you are dealing with “tiny data” you can waste resources on Docker, but when your foundation model is 100x the size, when the training process is distributed, and takes a day, the quantity is taking on a quality of its own. (3) The worst performance sin is moving data around although this will be necessary so far as the system is distributed. Avoiding excess data moving can be the difference between training a model and failing to train a model, but when you put together a patchwork of ML ops programs you will fin they are moving data around internally for good reasons sometimes and no reason other times plus the easy (and sometimes only) integration method is moving data around. Don’t be that guy!

danthelion 4y ago

Yes, and Data Engineering is just Software Engineering "in disguise"

rickette 4y ago
Exactly my thought.
marcyb5st 4y ago
While I agree with the premise, there's the added "complication" that knowledge or experience as a Data Scientist/ML Engineer it's highly beneficial. Pure SWEs, in my experience, struggle a bit in anticipating how Data Scientists will shoot themselves in the feet. Having experienced it yourself before it's a big plus.
rldjbpin 4y ago
underrated comment. of course i agree that there are specificities with mlops and data engineering that don't exist for a typical full-stack application. but if you boil it down enough, you just find the same underlying concepts.
nerdponx 4y ago
Or database admin.
- nerdponx 4y ago
  I guess I should clarify because I was downvoted. Data engineering involves a lot of database work. So if ML engineering is largely data engineering, then it's also at least partially database administration, transitively.

o10449366 4y ago

Sure, MLOps is just Data Engineering in disguise when you ignore the complexities of hardware provisioning, GPU optimization, integration tests for model performance and quality, benchmarking, resource constraints (network, disk, memory, GPU memory), etc. Anecdotally, I've worked in high-performance computing and machine learning for years now and the past few months I've seen a huge spike in the number of messages I get for MLOps positions. I think companies are slowly starting to realize that setting up machine learning at scale isn't as simple as deploying poorly written code by research scientists to managed platforms.

dpbrinkm 4y ago
>I think companies are slowly starting to realize that setting up machine learning at scale isn't as simple as deploying poorly written code by research scientists to managed platforms. This is some truth right here
pydry 4y ago
>complexities of hardware provisioning This is hardly an uncommon problem outside of machine learning.
- o10449366 4y ago
  You're generally right, but I think the strong dependency on GPUs with specific architectures and capabilities and how scarce they are is a fairly unique problem to ML (outside of crypto mining lol.)
nonethewiser 4y ago
> Anecdotally, I've worked in high-performance computing and machine learning for years now and the past few months I've seen a huge spike in the number of messages I get for MLOps positions. I think companies are slowly starting to realize that setting up machine learning at scale isn't as simple as deploying poorly written code by research scientists to managed platforms. This doesn’t bode well for ML in lots of orgs. Obviously machine learning is very powerful and effective in many use cases. But the value proposition already isn’t there for lots of companies. At such places, discovering a hidden requirement for more resources is a great reason to change directions. To be very clear, I’m not talking about the field in general. I’m talking about orgs where management doesn’t see value generated from the ML efforts that suddenly demand more resources to operate.

hgsgm 4y ago

Is ML just statistics (data analysis) in disguise?

jstx1 4y ago
They're distinct, there isn't all that much analysis in ML.
version_five 4y ago
No. But corporate "ML" has been until recently "dashboards" or "big data" or "data analytics" or "look at your corporate records with a computer" in disguise.

boredumb 4y ago

Learning some pytorch is truly not the bulk of the work to build a model, having to wrangle and mangle a massive amount of data coming from less-than-ideal data sources, orchestrating the jobs and making all of this available for your training routines to slurp up is a lot of work that evolves fairly quickly beneath you.

PartiallyTyped 4y ago
I wonder whether you could get a GPT-3 level LLM to extract the data for you.
- PaulHoule 4y ago
  not fast enough, but maybe it can help write the program that does.
  - PartiallyTyped 4y ago
    Definitely not realtime, but perhaps it could work as a preprocessing step?
    - PaulHoule 4y ago
      I worked for a startup that developed CNN models https://www.davidsbatista.net/blog/2018/03/31/SentenceClassificationConvNets/ https://www.davidsbatista.net/blog/2018/03/31/SentenceClassi... for early processing of data, you can think of it as a profiling tool that works at the level of an individual “cell” in a table, so given a number like 90214 it will guess that it could be a zip code, given “John” it would guess it is a proper name, etc. I’d contrast that to tools that profile a whole column at a time (funny… these are all 5 digit zero padded numbers) but that column is full of numbers consistent with https://en.wikipedia.org/wiki/Benford%27s_law https://en.wikipedia.org/wiki/Benford%27s_law This system was successful for a few customers including a major telecom and ultimately the company got bought by a major shoe and clothing brand. We had an “MLOps” system we developed internally for training those CNN models and we trained some for that task and trained some for other tasks. There were numerous problems with the models that we were concerned about then, most of them have been solved by transformer models which we were just starting to read about then.

noobcoder 4y ago

Before entering the field of ML, I perceived MLOps as a superhero with abilities to handle and deploy ML models. However, it seems that MLOps is more or less a typical engineer who acquired skills to manage and deploy data infrastructure for ML purposes (exclusively) by exposure to data engineering.

mountainriver 4y ago
Eh that’s not what I’ve seen, non deterministic models have a whole class of unique challenges
fest 4y ago
> However, it seems that XYZ is more or less a typical engineer who acquired skills.. This is true for pretty much every engineering discipline/role.

bobbruno 4y ago

On a first read, I could agree with most, if not all, of the author’s arguments. But there are two aspects that were simply left out that I have to consider when doing MLOps, which can prove too complex for just saying “It’s an extension of tooling that data engineering already has”. First, there’s the matter that ML introduces a significant stack and complexity into what was already a relatively complex framework. I mean, managing storage, quality, data processing, streaming, scheduling/orchestration, transformation logic and SLAs requires a lot of tools, whichever combination pleases you the most. Even full platforms offered by some of the players in this market can get quite complex, and it’s very hard to set everything right and handle all the cases. Specialised tooling or skills is probably a good idea to focus on the things that matter to Ml and that go beyond what DE already covers. Think of all the frameworks, the statistical libraries, the different nature of the logic for ML features when compared to regular reporting needs, the different quality requirements and structure that ML expects, managing versions of raw, labelled and test datasets, etc. (there are many more, the discussion already covered quite a few). Which brings me to the second thing - the knowledge stack required to run ML. Besides some of the usual DE stack (developing, data manipulation, quality, etc), a whole new set of skills, related to several branches of math, parallelism, very complex and costly infrastructure management, research skills, experiment design, specific algorithms and approaches (does the regular DE need to understand neural network patterns, how data and model-parallel training works, the statistics behind setting up and running drift measures, what all the metrics behind model performance mean, etc.?). I find this a better reason for specialisation than any other - there’s just so much one person can hold in their heads, and ML development and operations is just getting more complex by the day. So, my point is, from a very simplified and abstract perspective, the author seems right. But, in practice, you won’t be able to just stack that on top of data engineers and not expect them to become specialized - and that’s where the ML Engineer and MLOps engineer roles are emerging. They’re not completely new, but they’re no longer your regular data engineer or data scientist.

achileas 3y ago

alwayshasbeen.jpg I was doing this work since before MLOps was the new buzzword in town, and it was always under the data engineering job title. It was only in the past few years that data engineering has become more focused, requiring new titles/job descriptions to truly cover the different specializations.