3 ms·

Hey folks I'm the author of the post and happy to see that it gets so much attention on HN. Thank you for the incredible comments! I want to clarify something

by cpard 4y ago

Hey folks I'm the author of the post and happy to see that it gets so much attention on HN. Thank you for the incredible comments! I want to clarify something about my intention with this post. There is a reason I chose "mostly" on the title. I'm not dismissing the different needs of ML. if a category withstands the tests of the market, then there's good reason for it to exist. But, we have ended up creating silos within orgs with fundamentally aligned goals because of the way we build products and companies around them. What I'm advocating for in this article, is the need to think more holistically when we design and build data infra tooling. Yes ML has unique challenges but these challenges won't be addressed by reinventing everything again and again. Tooling should be built having in mind all the practitioners involved in the lifecycle of data. It's harder to do but at least we'll stop wasting our time building one Airflow copy after the other that is doomed to fail.

jamesblonde 4y ago
Here's one thing that you missed - transformations. Data engineers hear the word transformations and think they know what they mean - aggregations, binning, data reductions, cleansing, etc. Data scientists, however, think preparing features for models with transformations - encoding categorical variables (one-hot-encoding, LabelEncoding, OrdinalEncoding) and normalizing/standardizing/log-transforms for numerical features. These transformations are so different that some of them are "model-independent" - you can do the transformations and reuse the output feature across many models (aggregations, binning, feature-crosses, embeddings, etc), but the data scientists transformations are model-dependent (and not reusable across different models - e.g., a XGBoost model doesn't want normalized numerical features, but a DNN typically needs normalized or standardized numerical features. These differences are reflected in how we build our "ML pipelines" - we split them into feature/training/inference pipelines, enabling us to localize model-independent transformations to feature pipelines, while doing model-dependent transformations in training/inference pipelines (while ensuring no skew). In summary, the devil's in the details in pipelines for ML. I agree, however, that orchestrators like Airflow are good enough for orchestration. However, the ML Assets (mutable & reusable features, immutable models, immutable training/inference datasets) are different and tooling will ultimately reflect those differences.

by cpard 4y ago

jamesblonde 4y ago

Here's one thing that you missed - transformations. Data engineers hear the word transformations and think they know what they mean - aggregations, binning, data reductions, cleansing, etc. Data scientists, however, think preparing features for models with transformations - encoding categorical variables (one-hot-encoding, LabelEncoding, OrdinalEncoding) and normalizing/standardizing/log-transforms for numerical features. These transformations are so different that some of them are "model-independent" - you can do the transformations and reuse the output feature across many models (aggregations, binning, feature-crosses, embeddings, etc), but the data scientists transformations are model-dependent (and not reusable across different models - e.g., a XGBoost model doesn't want normalized numerical features, but a DNN typically needs normalized or standardized numerical features. These differences are reflected in how we build our "ML pipelines" - we split them into feature/training/inference pipelines, enabling us to localize model-independent transformations to feature pipelines, while doing model-dependent transformations in training/inference pipelines (while ensuring no skew). In summary, the devil's in the details in pipelines for ML. I agree, however, that orchestrators like Airflow are good enough for orchestration. However, the ML Assets (mutable & reusable features, immutable models, immutable training/inference datasets) are different and tooling will ultimately reflect those differences.