8 ms·

> I think it can yield the highest success rate. At what? If you want to be an SRE for a data platform, sure, but this pretty thankless work: - cleaning up d

> I think it can yield the highest success rate. At what? If you want to be an SRE for a data platform, sure, but this pretty thankless work: - cleaning up dodgy data - cleaning up behind low code data pipelines and other painful integration work with systems that suck and sometimes just don’t work (like PowerBI). - cleaning up behind data scientists that create models in arcane and imaginative ways and expect you to “productionise” them. - cleaning up behind brain dead scheduling systems that fail unexpectedly. - constant churn with partners and cloud products for whatever the latest hotness is. If you want to be solving actual problems with ML, this is a dead end. It’s IT support for the people doing real work. …so, it depends on your goals. Getting a job? Sure! Everyone wants a workhorse who they can dump all the annoying problematic on-call tasks to. Learning ML, contributing to research, building models? This isn’t a path that leads there.

robertlagrant 4y ago
> It’s IT support for the people doing real work. This is an appalling perspective. Good MLE skills seem a lot harder to find that good ML ones.
- wokwokwok 4y ago
  Someone who can engineer infrastructure, pipelines and fire fight production issues is hard to find, but that’s not the point I was making. My apologies; It is real work; the point I was making is it’s not ML work, any more than writing a yaml file is ML work. If you want to write yaml files, any number of possibilities exist. If you want to work with machine learning, then don’t become a data engineer. The skills are, mostly, not related ML, and more closely aligned with SRE / devops. It’s not infrastructure and helping build models as you mature and advance: it’s almost literally just infrastructure and fire fighting… in my, limited, 3 years of experience as such.
  - robertlagrant 4y ago
    Sure - fair clarification. But conversely, you could make some awesome automation that you rule with a light touch as an engineer, or as a DS you could come up with that ML that Amazon has to recommend you endless TVs as soon as you bought a TV :)
  - natch 4y ago
    With the right mindset it can be insanely fun building infrastructure, automating things, and engineering solutions such that improvements ratchet forward and fires get put out before they have a chance to grow. While the ML people, bless them, are chasing an 0.001 improvement in the metric of the day, data engineers can be having huge impact and changing the game. Meantime ML is becoming commoditized in its most common use cases.
  - achileas 4y ago
    I first had the DE title 7 years ago (going into it having never heard of DE), and have been doing MLE/platform work for the past 5. You’re projecting your limited experience onto a poorly defined role that varies wildly from company to company. My experience is much different from yours: little firefighting, lots of actual building. Yes there is infrastructure, but any good programmer these days should be able to stand up some basic infrastructure. Yes, don’t get into it if you want to do ML research or apply ML, but if you are interested a bit in it and find building models the least creative, most boring shit ever like I do, and prefer traditional coding, it’s a nice spot to be in.
    - vasili111 4y ago
      What is the average salary for DE currently in US?
  - lmeyerov 4y ago
    What? A lot of the hard part isn't the model, and especially in a world where bert, xgboost, optuna, pytorch, etc have solved much of the classic problem and forced 'real' DS to specialize on either the business consulting side (not math/engineering) or theory side (barely implemented). The rebrand of 'data analyst' (SQL, powerbi, . ..) to 'data scientist' by even top tech companies underscores this. It's not yet to where web dev has gotten in terms of global $20/hr fiverrr contractors, but already at say $40/hr for someone who can build real production models for more boring scenarios. The result is the vast bulk of data scientists (phd, self-trained, consulting, ...) we interview are weak engineers, so going from a make-believe notebook to a trickier production scenario requires the data engineer / MLOps / etc to solve a lot that a typical DS doesn't really understand in practice. Scale, latency, distributed systems, testing, etc. Likewise, the part the DS solves has little to do with the latest neuroips paper, and more just about lifecycle tasks like getting better data, which the other folks on the team will often be involved with as well. So 2 natural high-paying paths here: data engineer / MLOps -> MLEngineer -> DS data engineer -> all-in-one data analyst/scientist -> ML/AI data scientist
    - i_love_music 4y ago
      Definitely agree with your first two paragraphs, but am confused by the pay paths. Can you expand on what the paths mean?
      - lmeyerov 4y ago
        It's useful to work backwards from the knowledge a DS needs to be worth their weight. Imagine a small team of $400K/yr DS + $400K/yr DE + ... and whatever hw/sw . So say a $2-3M/yr project driving $3M+ of new growing revenue or $6-12M of annual savings. At bigger companies, even more magnitudes & pressure :) The DS will likely: - be close to the business case & business stakeholders to ask questions a normal lead can't - know the relevant math + ML algorithms, and build up specializations pairing DS niches ("time series forecasting") with industry niches ("supply chains in manufacturing") - enough engineering & performance understanding to work with a DE on going from small data sets to big ones - have an intuitive feel for all of the above - how data/usecases/etc. go right/wrong That's a lot!! One path is jumping in as a low-paid intern or new grad and doing your time. But a pivot is different, esp. to get paid along the way. Most CS grads had little math ("intros to stats, combinatorics, & algs; dropped linear algebra"), weak ML ("did algs; intro to ML only covered kmeans & bayes; tried running a BERT model on some data"), and little intuition for how ML typically goes wrong ("what's class imbalance?"). So if they do get hired directly as a mid-level DS, it's probably on a team of the blind-leading-the-blind. Oops. BUT SQL/Spark/K8S/pandas/regex are real skills. Doing the data engineering, ML operations, etc., around making an ML pipeline more than a fanciful notebook that wouldn't last a minute in production is real work. That stuff does pay well, and by working with the ML folks, you'd naturally get pulled into the ML tasks as well. DS write all sorts of bugs that surface as production evolves and the full team works together on, and new features that needs a team to make real. So taking a job that mixes engineering specialties with ML specialties is a smoother pivot path for the typical CS backgrounds I've seen. Over time, drift to more ML-y aspects of the projects happening until you can do the full hop. (Nit: That won't teach the math & deeper intuition, so I'd still do courses + projects on the side.)
  - i_love_music 4y ago
    Great comments. I agree with your take on what being an ML Eng actually means. Of course this will vary to a degree from team to team and company to company, but I think you still capture it well. I absolutely think MLEng is important and much needed, but too often under appreciated. Being this half breed part engineer part ML leaves you on a lonely island often in many orgs. The ML managers don't really understand what you do and neither do the engineering managers. It is kind of thankless unless your management really understands your role and appropriately advocates for you. MLEng is often an engineer who wanted to get into the sexy ML space and since it is in the title it feels cool. Then you realize you're more an Ops engineer who deals with the inane code of many "true" DS/ML scientists. Thankless, indeed.
    - fisf 4y ago
      Especially in the edge / embedded space, MLEng will imply more than just doing ops. Stuff to do could include: - Getting a network architecture to run. - Applying optimization depending on target arch (pruning, quantisation, custom cuda kernels, etc). - Integrating models (rule of thumb: a product is 95% ordinary code, 5% is ML related). - Constructing benchmarks, monitoring
- psi75 4y ago
  I think he meant "real work" from the perspective of the overeducated people who tend to end up in (and resent being in) corporate data science roles. From the boss's perspective, the grungy IT support stuff is closer to real work (although the only thing that's actually respected in management, because it's what they do) than the shiny ML stuff that one star hire is allowed to do because it makes the company look cooler than it actually is.
LeetHacks 4y ago
It highly depends. I was hired for a small research group that didn't have a product in production. Got hired for programming, was on the table discussing and contributing to research within a couple months without any background in ML.
- bitL 4y ago
  In smaller teams/companies one gets to wear multiple hats. However, the term "Data engineer" was specifically created by/for ML folks to get rid of unpleasant repetitive work that has to be done but nobody looks forward to it.
  - natch 4y ago
    Sorry, no. The term "ML scientist" was specifically created by/for data folks to get rid of unpleasant repetitive work with math equations that has to be done but nobody looks forward to it. If you've ever crafted a pipeline and tuned it to hum along, then watched it break with new/more/messier data, then figured out creative ways to fix it or replace parts of it with more robust parts, iterating on that and scaling it up, you would know some of the fantastic pleasure of whatever you call it, data engineering.
    - bsenftner 4y ago
      Sounds like what the media entertainment companies call a "pipeline engineer". Hmmm...
  - hnhg 4y ago
    Data engineers exist at organisations without any ML work.
    - bitL 4y ago
      Yes, but they are basically what DBAs were before with the addition of ETL. OP is asking about data engineers in the context of ML.
  - 4y ago
theteapot 4y ago
SRE = Site Reliability Engineer?
- shoo 4y ago
  yes
psi75 4y ago
This, 100%. That said, most data scientists don't do what you would consider real work (meaning, I assume, interesting work with significant mathematical/analytical meat). There just isn't a lot that's both interesting and useful to private-sector rent-seekers whose opinions of your work determine whether or not you advance. Most of the people doing real ML in industry are prestige hires--they're hired because their names draw people in, but basically get to work on whatever they want--and you need a top-10 PhD at an absolute minimum to be eligible for those. The ugly truth about industry is that 99.9997% of it is flow capture based on power relationships, found artifacts (i.e., corruption opportunities) within the state, and the implementation of very simple processes but in a way such that the threat to executive reputations as a first priority, and profit as an important second one, are minimized. This doesn't exactly make a market for ML innovation, unless your boss for some weird reason still cares about being a co-author on your papers (which his bosses will pressure him not to let you publish, because after all, this publishing is a distraction from your paid work). On the other hand, if you want to be able to afford a house in the Bay Area, and to be tapped for (indeed, most likely forced into, both due to losing interest in and being unhireable for IC work) management in your mid-30s... then go for industry. The poison carrot will make you sick but it will kill you more slowly than poverty, so that ain't so bad, now is it?
- urthor 4y ago
  Strongly disagree. There's a vast amount of work that doesn't involve unethical recommendation systems. Expand your horizon outside the Bay Area. The plurality of work I see is straightforward computer vision/NLP applications.
  - natch 4y ago
    I suspect the work you're talking about could be easily handled by an intern working with Core ML and a MacBook. The landscape is varied. There are companies doing real actual big leading edge stuff, there are companies where ML is sprinkled onto projects as a buzzword but no real interesting work happens, and companies that just need a practical small solution like the ones you mentioned, and could get by with Core ML, but don't because they hire a PhD who isn't aware of Core ML.
deleted 4y ago
[deleted]
urthor 4y ago
Can confirm. While data engineer is an excellent role for a fresh graduate, the data engineering profession shares many similarities with the SRE/IT professonional. The best data engineers are the folk who had job title of DBAs of yonder year. You will always be the supporting cast, rarely the star.
deleted 4y ago
[deleted]

robertlagrant 4y ago

> It’s IT support for the people doing real work. This is an appalling perspective. Good MLE skills seem a lot harder to find that good ML ones.

wokwokwok 4y ago
Someone who can engineer infrastructure, pipelines and fire fight production issues is hard to find, but that’s not the point I was making. My apologies; It is real work; the point I was making is it’s not ML work, any more than writing a yaml file is ML work. If you want to write yaml files, any number of possibilities exist. If you want to work with machine learning, then don’t become a data engineer. The skills are, mostly, not related ML, and more closely aligned with SRE / devops. It’s not infrastructure and helping build models as you mature and advance: it’s almost literally just infrastructure and fire fighting… in my, limited, 3 years of experience as such.
- robertlagrant 4y ago
  Sure - fair clarification. But conversely, you could make some awesome automation that you rule with a light touch as an engineer, or as a DS you could come up with that ML that Amazon has to recommend you endless TVs as soon as you bought a TV :)
- natch 4y ago
  With the right mindset it can be insanely fun building infrastructure, automating things, and engineering solutions such that improvements ratchet forward and fires get put out before they have a chance to grow. While the ML people, bless them, are chasing an 0.001 improvement in the metric of the day, data engineers can be having huge impact and changing the game. Meantime ML is becoming commoditized in its most common use cases.
- achileas 4y ago
  I first had the DE title 7 years ago (going into it having never heard of DE), and have been doing MLE/platform work for the past 5. You’re projecting your limited experience onto a poorly defined role that varies wildly from company to company. My experience is much different from yours: little firefighting, lots of actual building. Yes there is infrastructure, but any good programmer these days should be able to stand up some basic infrastructure. Yes, don’t get into it if you want to do ML research or apply ML, but if you are interested a bit in it and find building models the least creative, most boring shit ever like I do, and prefer traditional coding, it’s a nice spot to be in.
  - vasili111 4y ago
    What is the average salary for DE currently in US?
- lmeyerov 4y ago
  What? A lot of the hard part isn't the model, and especially in a world where bert, xgboost, optuna, pytorch, etc have solved much of the classic problem and forced 'real' DS to specialize on either the business consulting side (not math/engineering) or theory side (barely implemented). The rebrand of 'data analyst' (SQL, powerbi, . ..) to 'data scientist' by even top tech companies underscores this. It's not yet to where web dev has gotten in terms of global $20/hr fiverrr contractors, but already at say $40/hr for someone who can build real production models for more boring scenarios. The result is the vast bulk of data scientists (phd, self-trained, consulting, ...) we interview are weak engineers, so going from a make-believe notebook to a trickier production scenario requires the data engineer / MLOps / etc to solve a lot that a typical DS doesn't really understand in practice. Scale, latency, distributed systems, testing, etc. Likewise, the part the DS solves has little to do with the latest neuroips paper, and more just about lifecycle tasks like getting better data, which the other folks on the team will often be involved with as well. So 2 natural high-paying paths here: data engineer / MLOps -> MLEngineer -> DS data engineer -> all-in-one data analyst/scientist -> ML/AI data scientist
  - i_love_music 4y ago
    Definitely agree with your first two paragraphs, but am confused by the pay paths. Can you expand on what the paths mean?
    - lmeyerov 4y ago
      It's useful to work backwards from the knowledge a DS needs to be worth their weight. Imagine a small team of $400K/yr DS + $400K/yr DE + ... and whatever hw/sw . So say a $2-3M/yr project driving $3M+ of new growing revenue or $6-12M of annual savings. At bigger companies, even more magnitudes & pressure :) The DS will likely: - be close to the business case & business stakeholders to ask questions a normal lead can't - know the relevant math + ML algorithms, and build up specializations pairing DS niches ("time series forecasting") with industry niches ("supply chains in manufacturing") - enough engineering & performance understanding to work with a DE on going from small data sets to big ones - have an intuitive feel for all of the above - how data/usecases/etc. go right/wrong That's a lot!! One path is jumping in as a low-paid intern or new grad and doing your time. But a pivot is different, esp. to get paid along the way. Most CS grads had little math ("intros to stats, combinatorics, & algs; dropped linear algebra"), weak ML ("did algs; intro to ML only covered kmeans & bayes; tried running a BERT model on some data"), and little intuition for how ML typically goes wrong ("what's class imbalance?"). So if they do get hired directly as a mid-level DS, it's probably on a team of the blind-leading-the-blind. Oops. BUT SQL/Spark/K8S/pandas/regex are real skills. Doing the data engineering, ML operations, etc., around making an ML pipeline more than a fanciful notebook that wouldn't last a minute in production is real work. That stuff does pay well, and by working with the ML folks, you'd naturally get pulled into the ML tasks as well. DS write all sorts of bugs that surface as production evolves and the full team works together on, and new features that needs a team to make real. So taking a job that mixes engineering specialties with ML specialties is a smoother pivot path for the typical CS backgrounds I've seen. Over time, drift to more ML-y aspects of the projects happening until you can do the full hop. (Nit: That won't teach the math & deeper intuition, so I'd still do courses + projects on the side.)
- i_love_music 4y ago
  Great comments. I agree with your take on what being an ML Eng actually means. Of course this will vary to a degree from team to team and company to company, but I think you still capture it well. I absolutely think MLEng is important and much needed, but too often under appreciated. Being this half breed part engineer part ML leaves you on a lonely island often in many orgs. The ML managers don't really understand what you do and neither do the engineering managers. It is kind of thankless unless your management really understands your role and appropriately advocates for you. MLEng is often an engineer who wanted to get into the sexy ML space and since it is in the title it feels cool. Then you realize you're more an Ops engineer who deals with the inane code of many "true" DS/ML scientists. Thankless, indeed.
  - fisf 4y ago
    Especially in the edge / embedded space, MLEng will imply more than just doing ops. Stuff to do could include: - Getting a network architecture to run. - Applying optimization depending on target arch (pruning, quantisation, custom cuda kernels, etc). - Integrating models (rule of thumb: a product is 95% ordinary code, 5% is ML related). - Constructing benchmarks, monitoring
psi75 4y ago
I think he meant "real work" from the perspective of the overeducated people who tend to end up in (and resent being in) corporate data science roles. From the boss's perspective, the grungy IT support stuff is closer to real work (although the only thing that's actually respected in management, because it's what they do) than the shiny ML stuff that one star hire is allowed to do because it makes the company look cooler than it actually is.

LeetHacks 4y ago

It highly depends. I was hired for a small research group that didn't have a product in production. Got hired for programming, was on the table discussing and contributing to research within a couple months without any background in ML.

bitL 4y ago
In smaller teams/companies one gets to wear multiple hats. However, the term "Data engineer" was specifically created by/for ML folks to get rid of unpleasant repetitive work that has to be done but nobody looks forward to it.
- natch 4y ago
  Sorry, no. The term "ML scientist" was specifically created by/for data folks to get rid of unpleasant repetitive work with math equations that has to be done but nobody looks forward to it. If you've ever crafted a pipeline and tuned it to hum along, then watched it break with new/more/messier data, then figured out creative ways to fix it or replace parts of it with more robust parts, iterating on that and scaling it up, you would know some of the fantastic pleasure of whatever you call it, data engineering.
  - bsenftner 4y ago
    Sounds like what the media entertainment companies call a "pipeline engineer". Hmmm...
- hnhg 4y ago
  Data engineers exist at organisations without any ML work.
  - bitL 4y ago
    Yes, but they are basically what DBAs were before with the addition of ETL. OP is asking about data engineers in the context of ML.
- 4y ago

theteapot 4y ago

SRE = Site Reliability Engineer?

shoo 4y ago
yes

psi75 4y ago

This, 100%. That said, most data scientists don't do what you would consider real work (meaning, I assume, interesting work with significant mathematical/analytical meat). There just isn't a lot that's both interesting and useful to private-sector rent-seekers whose opinions of your work determine whether or not you advance. Most of the people doing real ML in industry are prestige hires--they're hired because their names draw people in, but basically get to work on whatever they want--and you need a top-10 PhD at an absolute minimum to be eligible for those. The ugly truth about industry is that 99.9997% of it is flow capture based on power relationships, found artifacts (i.e., corruption opportunities) within the state, and the implementation of very simple processes but in a way such that the threat to executive reputations as a first priority, and profit as an important second one, are minimized. This doesn't exactly make a market for ML innovation, unless your boss for some weird reason still cares about being a co-author on your papers (which his bosses will pressure him not to let you publish, because after all, this publishing is a distraction from your paid work). On the other hand, if you want to be able to afford a house in the Bay Area, and to be tapped for (indeed, most likely forced into, both due to losing interest in and being unhireable for IC work) management in your mid-30s... then go for industry. The poison carrot will make you sick but it will kill you more slowly than poverty, so that ain't so bad, now is it?

urthor 4y ago
Strongly disagree. There's a vast amount of work that doesn't involve unethical recommendation systems. Expand your horizon outside the Bay Area. The plurality of work I see is straightforward computer vision/NLP applications.
- natch 4y ago
  I suspect the work you're talking about could be easily handled by an intern working with Core ML and a MacBook. The landscape is varied. There are companies doing real actual big leading edge stuff, there are companies where ML is sprinkled onto projects as a buzzword but no real interesting work happens, and companies that just need a practical small solution like the ones you mentioned, and could get by with Core ML, but don't because they hire a PhD who isn't aware of Core ML.

deleted 4y ago

[deleted]

urthor 4y ago

Can confirm. While data engineer is an excellent role for a fresh graduate, the data engineering profession shares many similarities with the SRE/IT professonional. The best data engineers are the folk who had job title of DBAs of yonder year. You will always be the supporting cast, rarely the star.

deleted 4y ago

[deleted]