6 ms·

It highly depends. I was hired for a small research group that didn't have a product in production. Got hired for programming, was on the table discussing and c

It highly depends. I was hired for a small research group that didn't have a product in production. Got hired for programming, was on the table discussing and contributing to research within a couple months without any background in ML.

bitL 4y ago
In smaller teams/companies one gets to wear multiple hats. However, the term "Data engineer" was specifically created by/for ML folks to get rid of unpleasant repetitive work that has to be done but nobody looks forward to it.
- natch 4y ago
  Sorry, no. The term "ML scientist" was specifically created by/for data folks to get rid of unpleasant repetitive work with math equations that has to be done but nobody looks forward to it. If you've ever crafted a pipeline and tuned it to hum along, then watched it break with new/more/messier data, then figured out creative ways to fix it or replace parts of it with more robust parts, iterating on that and scaling it up, you would know some of the fantastic pleasure of whatever you call it, data engineering.
  - bsenftner 4y ago
    Sounds like what the media entertainment companies call a "pipeline engineer". Hmmm...
- hnhg 4y ago
  Data engineers exist at organisations without any ML work.
  - bitL 4y ago
    Yes, but they are basically what DBAs were before with the addition of ETL. OP is asking about data engineers in the context of ML.
- chrstr 4y ago
  > However, the term "Data engineer" was specifically created by/for ML folks to get rid of unpleasant repetitive work that has to be done but nobody looks forward to it. This may indeed be how the term "data engineer" is used sometimes, but I have my doubts that it was originally created with this meaning. Not really sure where/when the term "data engineer" was actually created, but ICDE started in 1984 [1] and the Data Engineering Bulletin was renamed in 1987 [2] (from "Database Engineering"). It seems likely that the term "data engineer" has also been used since at least then. Of course ML did also already exist then, but it's certainly a while before the current "big data" / "deep learning" time. And regarding the topics considered "data engineering" at that time, this is from the foreword of the December 1987 issue of the Data Engineering bulletin: > The reasons for the recent surge of interest in the area of Databases and Logic go beyond the theoretical foundations that were explored by early work [...] and include the following three motivations: > a) The projected future demand for Knowledge Management Systems. These will have to combine inference mechanisms from Logic with the efficient and secure management of large sets of information from Database Systems. Which sounds just as relevant today as it did back then. It also does sound like a rather challenging task, and not exactly like "unpleasant repetitive work". Or at least not any more repetitive than: change some model parameters / retrain model / evaluate results / repeat ;) [1]: https://ieeexplore.ieee.org/xpl/conhome/1000178/all-proceedings https://ieeexplore.ieee.org/xpl/conhome/1000178/all-proceedi... [2]: http://sites.computer.org/debull/bull_issues.html http://sites.computer.org/debull/bull_issues.html
  - bitL 4y ago
    Data engineering jobs named as such started to pop up only in the past few years, coinciding with Map Reduce/Spark availability. I wouldn't be surprised if it was re-introduced by one of the companies developing those systems to distinguish themselves (like Databricks, Cloudera etc.), a sort of a marketing. In the past we had DBAs, now DBA + DevOps + unspecified everything morphed into data engineering. I used to be a member of SIGMOD and the "data engineering" you mentioned was just an academic term.
listenallyall 4y ago
These types of anecdotes make the actual practice of both ML and AI seem rather, well, less than scientific. There is supposed to be Ph.D. level math behind all of this, yet an amateur with admittedly no ML background is part of the team. In Star Wars, it takes Luke Skywalker years to learn to use a light saber skillfully. Then in The Force Awakens, some ex-Stormtrooper with no training picks up the light saber and within 5 minutes is a pro. Kinda ruined the mystique of Star Wars, just like people jumping into ML with no training ruins the mystique of ML.
- jokethrowaway 4y ago
  You're disillusioned if you think a PhD is what makes the difference. Smart people will be able to contribute even if they don't have a PhD. Some PhD are useless and everyone is wondering how the hell they go through that.
- mkl 4y ago
  Is mystique worth having? In fiction, sure, but I think not in R&D. I think a lot of ML and AI isn't especially scientific, but I also think there's a lot of low-hanging fruit in applying it to new areas, and both of those make it easier for an amateur to contribute.
- deleted 4y ago
  [deleted]
- natch 4y ago
  Fuck mystique.
- LeetHacks 4y ago
  I am very against the idea that somebody with a PhD is the only one that can do a certain kind of work. But I am ofcourse biased given that you call me out. Creative and critical thinking is not exclusive to people with a PhD. The ability to understand ones strengths and weaknesses is not exclusive to a PhD. I would never attempt to write or publish a paper without help of somebody with stronger mathematical or statistical knowledge. On the other hand they should not write source code for a paper without consulting somebody with a strong background in sw engineering. You complement each other. Power is in recognizing that. You would be surprised how many software bugs I have found that invalidated entire (draft) papers. A PhD in ML doesn't save you from that.

bitL 4y ago

In smaller teams/companies one gets to wear multiple hats. However, the term "Data engineer" was specifically created by/for ML folks to get rid of unpleasant repetitive work that has to be done but nobody looks forward to it.

natch 4y ago
Sorry, no. The term "ML scientist" was specifically created by/for data folks to get rid of unpleasant repetitive work with math equations that has to be done but nobody looks forward to it. If you've ever crafted a pipeline and tuned it to hum along, then watched it break with new/more/messier data, then figured out creative ways to fix it or replace parts of it with more robust parts, iterating on that and scaling it up, you would know some of the fantastic pleasure of whatever you call it, data engineering.
- bsenftner 4y ago
  Sounds like what the media entertainment companies call a "pipeline engineer". Hmmm...
hnhg 4y ago
Data engineers exist at organisations without any ML work.
- bitL 4y ago
  Yes, but they are basically what DBAs were before with the addition of ETL. OP is asking about data engineers in the context of ML.
chrstr 4y ago
> However, the term "Data engineer" was specifically created by/for ML folks to get rid of unpleasant repetitive work that has to be done but nobody looks forward to it. This may indeed be how the term "data engineer" is used sometimes, but I have my doubts that it was originally created with this meaning. Not really sure where/when the term "data engineer" was actually created, but ICDE started in 1984 [1] and the Data Engineering Bulletin was renamed in 1987 [2] (from "Database Engineering"). It seems likely that the term "data engineer" has also been used since at least then. Of course ML did also already exist then, but it's certainly a while before the current "big data" / "deep learning" time. And regarding the topics considered "data engineering" at that time, this is from the foreword of the December 1987 issue of the Data Engineering bulletin: > The reasons for the recent surge of interest in the area of Databases and Logic go beyond the theoretical foundations that were explored by early work [...] and include the following three motivations: > a) The projected future demand for Knowledge Management Systems. These will have to combine inference mechanisms from Logic with the efficient and secure management of large sets of information from Database Systems. Which sounds just as relevant today as it did back then. It also does sound like a rather challenging task, and not exactly like "unpleasant repetitive work". Or at least not any more repetitive than: change some model parameters / retrain model / evaluate results / repeat ;) [1]: https://ieeexplore.ieee.org/xpl/conhome/1000178/all-proceedings https://ieeexplore.ieee.org/xpl/conhome/1000178/all-proceedi... [2]: http://sites.computer.org/debull/bull_issues.html http://sites.computer.org/debull/bull_issues.html
- bitL 4y ago
  Data engineering jobs named as such started to pop up only in the past few years, coinciding with Map Reduce/Spark availability. I wouldn't be surprised if it was re-introduced by one of the companies developing those systems to distinguish themselves (like Databricks, Cloudera etc.), a sort of a marketing. In the past we had DBAs, now DBA + DevOps + unspecified everything morphed into data engineering. I used to be a member of SIGMOD and the "data engineering" you mentioned was just an academic term.

listenallyall 4y ago

These types of anecdotes make the actual practice of both ML and AI seem rather, well, less than scientific. There is supposed to be Ph.D. level math behind all of this, yet an amateur with admittedly no ML background is part of the team. In Star Wars, it takes Luke Skywalker years to learn to use a light saber skillfully. Then in The Force Awakens, some ex-Stormtrooper with no training picks up the light saber and within 5 minutes is a pro. Kinda ruined the mystique of Star Wars, just like people jumping into ML with no training ruins the mystique of ML.

jokethrowaway 4y ago
You're disillusioned if you think a PhD is what makes the difference. Smart people will be able to contribute even if they don't have a PhD. Some PhD are useless and everyone is wondering how the hell they go through that.
mkl 4y ago
Is mystique worth having? In fiction, sure, but I think not in R&D. I think a lot of ML and AI isn't especially scientific, but I also think there's a lot of low-hanging fruit in applying it to new areas, and both of those make it easier for an amateur to contribute.
deleted 4y ago
[deleted]
natch 4y ago
Fuck mystique.
LeetHacks 4y ago
I am very against the idea that somebody with a PhD is the only one that can do a certain kind of work. But I am ofcourse biased given that you call me out. Creative and critical thinking is not exclusive to people with a PhD. The ability to understand ones strengths and weaknesses is not exclusive to a PhD. I would never attempt to write or publish a paper without help of somebody with stronger mathematical or statistical knowledge. On the other hand they should not write source code for a paper without consulting somebody with a strong background in sw engineering. You complement each other. Power is in recognizing that. You would be surprised how many software bugs I have found that invalidated entire (draft) papers. A PhD in ML doesn't save you from that.