3 ms·

As an ML practitioner in a large ecommerce company, I totally agree. When I evaluate third party tools, 99% have no use at all for me. Things like Algolia or Re

by mlthoughts2018 7y ago

As an ML practitioner in a large ecommerce company, I totally agree. When I evaluate third party tools, 99% have no use at all for me. Things like Algolia or Rekognition or Clarifai. Even Sagemaker and Fargate are mostly not useful to me as my company can easily operate its own k8s platform. Open source tools for model building are great and a small team of engineers can write whatever odds & ends that aren’t covered, or make scalable implementations. Notebook platforms like Databricks are similarly useless. A notebook is not a frontend to a cluster and notebooks are universally poor development environments even for the very tasks they are advertised. The main thing that matters for ML development is how quickly & easily you can put an arbitrary container or VM into a deployment environment. You’re always going to need to rapidly change your container or VM, have common ancestors that get specialized to support GPU training or some web server application layer. Making the shortest possible distance between the definition of this environment and deployment of it is the whole game. The type of tools I’m willing to pay for are specialized database engines, rapid data annotation tools, and specialized hardware environments. I have no time for some “platform” tool for data provenance, notebook environments, model tracking, model-as-a-service APIs like Rekognition, or anything where I upload data and get back a model.

ordx 7y ago
How do you train your models? With Databricks (which is just a nice notebook UI for Apache Spark) I don't have to spend time setting up VMs and libraries for the distributed training.
- mlthoughts2018 7y ago
  Your first sentence seems unrelated to your second sentence, but it seems like you think they are related? Any platform that advertises something like you don’t have to spend time defining your training environment, whether it’s Databricks or an out of the box deep learning VM on GCP, is a liability waiting to happen. You always need to define your own training environment, especially because you’ll almost always need specific (likely pinned) versions of all your dependencies, including system dependencies, to manage model training as part of a production life cycle. Very often you also need e.g. custom compiled TensorFlow, custom GPU settings or drivers, etc. It’s very foolish to base that environment on whatever comes out of the box. Spark also is not universally useful. For training small models many times, like a workload that trains hundreds of small models all day (this was a production use case I had before that my company pilot tested with Databricks), the overhead of Py4J connector is insanely bad. It’s a really terrible paradigm for Python software, meanwhile Scala is a miserable ecosystem for production machine learning models. On top of all this, Spark MLib has huge gaps in functionality and whole classes of problems (e.g. large scale MCMC inference) are not solvable in a way in Spark that is seriously comparable to other tools like STAN & pymc running on not-Spark with simple multiprocessing.
  - ordx 7y ago
    Interesting, sounds like you have a very specific usecase. I'm mostly dealing with huge datasets and Spark is a lifesaver.
    - mlthoughts2018 7y ago
      My use case is just general computing on huge datasets for ML and statistics. Spark only serves a few special use cases that usually don’t justify its cost, similar to map reduce on Hadoop. Databricks though (distinct from Spark) serves no use cases.
mistrial9 7y ago
> notebooks are universally poor development environments even for the very tasks they are advertised unnecessarily dismissive, grandstanding.. Bravado of the rejection, not supported at all Rich-content Notebooks tell a story; well-worn Notebooks are a familiar context for experiment and viz.
- FridgeSeal 7y ago
  Notebooks are a maintainability maze waiting to happen. They're incredibly non-portable; they don't play nice with version control; I know how they're supposed to be used, but I'm yet to see them actually used in a way that isn't a breeding ground for the worst software engineering I've seen.
  - un_montagnard 7y ago
    Yet the same could be said of Excel, which is running the whole financial industry.
    - gpderetta 7y ago
      It can be both a bad environment and very successful. Personally I would be happy with just proper version control support.
tixocloud 7y ago
Could you care to elaborate further on specialized database engines and rapid data annotation tools? There are plenty of data annotation tools out there that seem to do the job so not sure what rapid would mean in this instance.
- mlthoughts2018 7y ago
  I mean things like vertica or kdb+ that have specialized performance properties for some use cases. Also to some minor extent managed cluster pipeline tooling like Spark. I don’t mind paying for managed versions of these (not Databricks though). For annotation tools I mean things like Prodigy.
  - tixocloud 7y ago
    Thanks. Why the interest in managed cluster pipeline but not Databricks? They seem to be the big name in the game for Spark. Would some tooling around snapshotting, sequential AB testing, staged rollouts, ghost models be of use?
    - mlthoughts2018 7y ago
      Databricks is a notebook frontend to cluster computing. Cluster computing is useful and I’m willing to pay for it as long as I have total & complete control of the environments and tooling used in the cluster workflow. A notebook frontend however is less than useless, and is actively harmful by propagating poor notebook environments even further into aspects of computing where they cause harm and hurt reproducability and code factoring. Given this, even if Databricks offered perfectly complete features for all aspects of cluster computing, it would still be inferior to just my own managed EC2 or EMR clusters or equivalent with other providers, where there is no “notebook as control plane” garbage. But when you add to that the fact that Databricks lacks full features for me to totally own every customized detail of my cluster environment (e.g. how can I run plain Python multiprocessing tasks with zero Spark in Databricks? How can I bring my own custom defined GPU container with custom compiled Tensorflow?) it makes the deal even worse. Databricks is just another Spark / Hadoop style snake oil seller banking on capturing a bunch of data science teams before people realize that it’s a conceptually junk way to work. As for the other tooling you mention, I’d almost always say to build it in house. For example, I don’t know of a single A/B testing provider that actually uses frequentist sequential testing to correctly avoid early stopping bias. You actually need real statisticians hired in-house to solve these problems, and the engineering work to set up an extensible A/B test as a service internal tool is not bad. (I’ve built Bayesian A/B test frameworks with teams of 2-3 engineers in 3 different companies). It’s just not cost effective to outsource it on the false hopes of not needing to hire your own in-house statistics experts. Just pony up the dough and hire them.

ordx 7y ago

How do you train your models? With Databricks (which is just a nice notebook UI for Apache Spark) I don't have to spend time setting up VMs and libraries for the distributed training.

mlthoughts2018 7y ago
Your first sentence seems unrelated to your second sentence, but it seems like you think they are related? Any platform that advertises something like you don’t have to spend time defining your training environment, whether it’s Databricks or an out of the box deep learning VM on GCP, is a liability waiting to happen. You always need to define your own training environment, especially because you’ll almost always need specific (likely pinned) versions of all your dependencies, including system dependencies, to manage model training as part of a production life cycle. Very often you also need e.g. custom compiled TensorFlow, custom GPU settings or drivers, etc. It’s very foolish to base that environment on whatever comes out of the box. Spark also is not universally useful. For training small models many times, like a workload that trains hundreds of small models all day (this was a production use case I had before that my company pilot tested with Databricks), the overhead of Py4J connector is insanely bad. It’s a really terrible paradigm for Python software, meanwhile Scala is a miserable ecosystem for production machine learning models. On top of all this, Spark MLib has huge gaps in functionality and whole classes of problems (e.g. large scale MCMC inference) are not solvable in a way in Spark that is seriously comparable to other tools like STAN & pymc running on not-Spark with simple multiprocessing.
- ordx 7y ago
  Interesting, sounds like you have a very specific usecase. I'm mostly dealing with huge datasets and Spark is a lifesaver.
  - mlthoughts2018 7y ago
    My use case is just general computing on huge datasets for ML and statistics. Spark only serves a few special use cases that usually don’t justify its cost, similar to map reduce on Hadoop. Databricks though (distinct from Spark) serves no use cases.

mistrial9 7y ago

> notebooks are universally poor development environments even for the very tasks they are advertised unnecessarily dismissive, grandstanding.. Bravado of the rejection, not supported at all Rich-content Notebooks tell a story; well-worn Notebooks are a familiar context for experiment and viz.

FridgeSeal 7y ago
Notebooks are a maintainability maze waiting to happen. They're incredibly non-portable; they don't play nice with version control; I know how they're supposed to be used, but I'm yet to see them actually used in a way that isn't a breeding ground for the worst software engineering I've seen.
- un_montagnard 7y ago
  Yet the same could be said of Excel, which is running the whole financial industry.
  - gpderetta 7y ago
    It can be both a bad environment and very successful. Personally I would be happy with just proper version control support.

tixocloud 7y ago

Could you care to elaborate further on specialized database engines and rapid data annotation tools? There are plenty of data annotation tools out there that seem to do the job so not sure what rapid would mean in this instance.

mlthoughts2018 7y ago
I mean things like vertica or kdb+ that have specialized performance properties for some use cases. Also to some minor extent managed cluster pipeline tooling like Spark. I don’t mind paying for managed versions of these (not Databricks though). For annotation tools I mean things like Prodigy.
- tixocloud 7y ago
  Thanks. Why the interest in managed cluster pipeline but not Databricks? They seem to be the big name in the game for Spark. Would some tooling around snapshotting, sequential AB testing, staged rollouts, ghost models be of use?
  - mlthoughts2018 7y ago
    Databricks is a notebook frontend to cluster computing. Cluster computing is useful and I’m willing to pay for it as long as I have total & complete control of the environments and tooling used in the cluster workflow. A notebook frontend however is less than useless, and is actively harmful by propagating poor notebook environments even further into aspects of computing where they cause harm and hurt reproducability and code factoring. Given this, even if Databricks offered perfectly complete features for all aspects of cluster computing, it would still be inferior to just my own managed EC2 or EMR clusters or equivalent with other providers, where there is no “notebook as control plane” garbage. But when you add to that the fact that Databricks lacks full features for me to totally own every customized detail of my cluster environment (e.g. how can I run plain Python multiprocessing tasks with zero Spark in Databricks? How can I bring my own custom defined GPU container with custom compiled Tensorflow?) it makes the deal even worse. Databricks is just another Spark / Hadoop style snake oil seller banking on capturing a bunch of data science teams before people realize that it’s a conceptually junk way to work. As for the other tooling you mention, I’d almost always say to build it in house. For example, I don’t know of a single A/B testing provider that actually uses frequentist sequential testing to correctly avoid early stopping bias. You actually need real statisticians hired in-house to solve these problems, and the engineering work to set up an extensible A/B test as a service internal tool is not bad. (I’ve built Bayesian A/B test frameworks with teams of 2-3 engineers in 3 different companies). It’s just not cost effective to outsource it on the false hopes of not needing to hire your own in-house statistics experts. Just pony up the dough and hire them.