3 ms·

Sure, MLOps is just Data Engineering in disguise when you ignore the complexities of hardware provisioning, GPU optimization, integration tests for model perfor

Sure, MLOps is just Data Engineering in disguise when you ignore the complexities of hardware provisioning, GPU optimization, integration tests for model performance and quality, benchmarking, resource constraints (network, disk, memory, GPU memory), etc. Anecdotally, I've worked in high-performance computing and machine learning for years now and the past few months I've seen a huge spike in the number of messages I get for MLOps positions. I think companies are slowly starting to realize that setting up machine learning at scale isn't as simple as deploying poorly written code by research scientists to managed platforms.

dpbrinkm 4y ago
>I think companies are slowly starting to realize that setting up machine learning at scale isn't as simple as deploying poorly written code by research scientists to managed platforms. This is some truth right here
pydry 4y ago
>complexities of hardware provisioning This is hardly an uncommon problem outside of machine learning.
- o10449366 4y ago
  You're generally right, but I think the strong dependency on GPUs with specific architectures and capabilities and how scarce they are is a fairly unique problem to ML (outside of crypto mining lol.)
nonethewiser 4y ago
> Anecdotally, I've worked in high-performance computing and machine learning for years now and the past few months I've seen a huge spike in the number of messages I get for MLOps positions. I think companies are slowly starting to realize that setting up machine learning at scale isn't as simple as deploying poorly written code by research scientists to managed platforms. This doesn’t bode well for ML in lots of orgs. Obviously machine learning is very powerful and effective in many use cases. But the value proposition already isn’t there for lots of companies. At such places, discovering a hidden requirement for more resources is a great reason to change directions. To be very clear, I’m not talking about the field in general. I’m talking about orgs where management doesn’t see value generated from the ML efforts that suddenly demand more resources to operate.
- sitkack 4y ago
  Could you explain the statement, "At such places, discovering a hidden requirement for more resources is a great reason to change directions." I don't understand the argument, not feigning confusion. Everyone has to scale at some point and every solution has its limits. If they were successful with Airflow and Pandas/Numpy for a long time and then well, now the fan is spinning. They are going to call for experts that put the pieces in place. Asking for help is a sign of maturity. It really depends on the state of system when the experts arrive. I personally think every org can use ML (it all decays to statistics, then linear algebra).
  - nonethewiser 4y ago
    I’m responding to the idea that MLops infrastructure is heavily underestimated. In which case you need more time, expertise, infrastructure, etc. than previously realized. AKA money I agree with everything you said for whatever it’s worth. I’m mostly just making the observation that there is a lot of machine learning endeavors that aren’t generating much value.
    - sitkack 4y ago
      > a lot of machine learning endeavors that aren’t generating much value. I agree with this. From what I have seen, execs want something fancy, when you could give a boring-ish tool that reduces your OODA loop cycle time to 30% of what it was using unsexy techniques. Much of it having to do with tolerancing of answers, being within 5-10% is more than enough to drive the business, but someone somewhere said it had to be exact and that blows out the latency budget. When engaging in consulting gigs, it is super important to know what kind of org you dealing with before you get involved. The myopic penny pinching orgs should be steered well away from, which I think was your point.

dpbrinkm 4y ago

>I think companies are slowly starting to realize that setting up machine learning at scale isn't as simple as deploying poorly written code by research scientists to managed platforms. This is some truth right here

pydry 4y ago

>complexities of hardware provisioning This is hardly an uncommon problem outside of machine learning.

o10449366 4y ago
You're generally right, but I think the strong dependency on GPUs with specific architectures and capabilities and how scarce they are is a fairly unique problem to ML (outside of crypto mining lol.)

nonethewiser 4y ago

> Anecdotally, I've worked in high-performance computing and machine learning for years now and the past few months I've seen a huge spike in the number of messages I get for MLOps positions. I think companies are slowly starting to realize that setting up machine learning at scale isn't as simple as deploying poorly written code by research scientists to managed platforms. This doesn’t bode well for ML in lots of orgs. Obviously machine learning is very powerful and effective in many use cases. But the value proposition already isn’t there for lots of companies. At such places, discovering a hidden requirement for more resources is a great reason to change directions. To be very clear, I’m not talking about the field in general. I’m talking about orgs where management doesn’t see value generated from the ML efforts that suddenly demand more resources to operate.

sitkack 4y ago
Could you explain the statement, "At such places, discovering a hidden requirement for more resources is a great reason to change directions." I don't understand the argument, not feigning confusion. Everyone has to scale at some point and every solution has its limits. If they were successful with Airflow and Pandas/Numpy for a long time and then well, now the fan is spinning. They are going to call for experts that put the pieces in place. Asking for help is a sign of maturity. It really depends on the state of system when the experts arrive. I personally think every org can use ML (it all decays to statistics, then linear algebra).
- nonethewiser 4y ago
  I’m responding to the idea that MLops infrastructure is heavily underestimated. In which case you need more time, expertise, infrastructure, etc. than previously realized. AKA money I agree with everything you said for whatever it’s worth. I’m mostly just making the observation that there is a lot of machine learning endeavors that aren’t generating much value.
  - sitkack 4y ago
    > a lot of machine learning endeavors that aren’t generating much value. I agree with this. From what I have seen, execs want something fancy, when you could give a boring-ish tool that reduces your OODA loop cycle time to 30% of what it was using unsexy techniques. Much of it having to do with tolerancing of answers, being within 5-10% is more than enough to drive the business, but someone somewhere said it had to be exact and that blows out the latency budget. When engaging in consulting gigs, it is super important to know what kind of org you dealing with before you get involved. The myopic penny pinching orgs should be steered well away from, which I think was your point.