7 ms·

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and enginee

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effectively 3. Build ETL pipelines and data management systems 4. Manage enterprise data sharing and resiliency etc... What ends up happening is you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team. That is typically missing data or poorly formatted and they spend 90% of their time cleaning it up then running some basic regression with numpy or whatever. Need better understanding of the data lifecycle by organizations and investment in instrumentation and data management.

mgh2 6y ago
Who will do the proper cleaning then?
- alex_anglin 6y ago
  The aspiration that GP was getting to was that less cleaning is required as a result of better data engineering, I believe.
  - AndrewKemendo 6y ago
    Correct. If you build your instrumentation correctly, then you don't really need to do any "cleaning." Doesn't mean you might not need to do transformation for different uses but ideally wouldn't need to, for example change data types like turning a bool into an int.
    - dpayonk 6y ago
      If we only had to use 1st party data, that might be easier. But then again, if you’re building your product incrementally, you’re still going to have instrumentation holes that you may or may not be able to partially backfill.
    - darksaints 6y ago
      The problem is that data engineers that are geared towards analytics very very rarely control the systems that create the data. If you're lucky, you have the task of hounding a team within your company to get their data management practices in order. And the conversation there is whether they should make their job harder in order to make your job easier. Unfortunately, data engineers rarely deal with purely in-house data. You're gonna be pulling data from a variety of data sources. I can assure you that if you're pulling from government data sources, you're gonna have a hell of a time. Speaking from direct experience, my team is probably going to spend $10M/year just trying to keep a government dataset in order, because they won't do it themselves. I'm talking lawyers, legal analysts, data engineers, data scientists, data entry personnel, etc.. just to fix data that should have never been broken in the first place. It shouldn't be a shock that cleaning the data is the path of least resistance for many.
      - AndrewKemendo 6y ago
        Hence why I said DE need to be involved as early as possible. Aspirational sure, but that's what I've seen work the best and repeatably. It's the only scalable solution IMO otherwise you're perpetually playing catch-up. On the point about the govt I literally built a completely new contract type and civilian hiring practices for the DoD to bring in Data Engineers so they could do exactly what I describe to make your life easier.
    - mgh2 6y ago
      Do data engineers have good analysis skills? Do business analysts have good engineering skills? I don't think either of them can fill the data scientist role. The scientific training and mindset (scientific method, hypothesis, experiment setup, etc.) to even create an accurate model is an undervalued skill here no? Even if data cleaning is automated, these skills cannot be easily learned. There is a reason why so many PhDs get into the field, because they were trained in the exploratory/research mindset that no engineering or analytics skills can fill. Correct me if I am wrong.
      - dijksterhuis 6y ago
        > Do data engineers have good analysis skills? Yes. > Do business analysts have good engineering skills? Depends on the analyst. > I don't think either of them can fill the data scientist role. > The scientific training and mindset (scientific method, hypothesis, experiment setup, etc.) to even create an accurate model is an undervalued skill here no? Even if data cleaning is automated, these skills cannot be easily learned. It's not about replacing data scientists with data engineers, it's about both roles working together to make everything more efficient. The hiring rate for data scientists has plateaued. The industry doesn't need any more of them. Why? Because data scientists often can't solve problems fast enough. It's a commonly quoted statistic that 70% of any data science task is data cleansing and/or etl. A data engineer's job is to take that 70% and turn it into 10%. The data engineer saves the data scientist time, meaning they can focus on what they're supposed to do -- build models.
- nerdponx 6y ago
  It doesn't matter, as long as you don't make the person with the PhD in biostatistics spend their time writing ETL pipelines, which is a wildly inefficient use of a very expensive resource.
  - names_are_hard 6y ago
    Do people with PhDs in biostatistics earn significantly more than programmers? I honestly know nothing about the market for biostatisticians, but my impression was that advanced degrees in the natural sciences don't really pay that well compared to software engineers, especially given that they're much more educated.
    - aldanor 6y ago
      If they work e.g. in a hedge fund / trading firm, then - yea. And you see lots of PhDs from unrelated fields working as quants there.
- Enginerrrd 6y ago
  Not to worry, corporate will just outsource to firm which hires Data Janitors
antipaul 6y ago
I have a dream - and it looks like this!
alexfromapex 6y ago
I agree, if anything the data engineers (folks with engineering backgrounds) should be doing the applied work while a department of data scientists works on the theoretical or novel data analysis methods. Right now our product has accumulated a lot of technical debt on the data validation side because data scientists designed the test code in a way that dramatically slows the development process.
- watermanio 6y ago
  This feels especially true when you have access to things like BigQuery ML. It's very easy for an average engineer (like me) to start using ML using these tools, but a lot harder to explain how it works, or exactly which type of models to use. In my mind a DS would be really useful to just point us in the right direction and check work. Like a super specialist QA...
- listenallyall 6y ago
  > novel data analysis methods Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. Asking them to develop a "novel" method would be fruitless. Asking them to clean and scrub the source data set is like telling an amateur pie-baker the store was out of pie crusts, you'll have to make your own from scratch -- it's not going to happen, they just don't have that skill, the instructions on the box don't account for that possibility. As soon as the task diverges from the simple step 1, step 2, step 3 that they were originally taught, you realize they have very little ability to adapt. YMMV of course.
  - tchalla 6y ago
    > Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. This is because they rarely hire people with scientific thinking ability. They just hire people who can code and program from set recipes. Once you hire such people you can not expect them to do non-recipe work. If you don't want recipe work, don't hire people will recipe skills. Do not have job interviews that select for recipe people. But, that is exactly what most companies do.
bitcharmer 6y ago
This same sentiment (which I personally agree with) applies to software engineering. As in: engineers deliver more practical value than comp scientists. Now you can down-vote me to oblivion.
- vbtemp 6y ago
  You and me both, friend. Except I normally get down-voted to oblivion for saying the opposite.
  - adamisom 6y ago
    I guess depends on what valuable means. I imagine most comp scientists are less replaceable than most software engineers, so point for compsci.
    - collyw 6y ago
      Depends if you have a computer scientist doing a software engineers job
    - johnqpub 6y ago
      The two are complimentary. Engineers can't do anything without the fundamental insights scientists provide. But scientists don't have the practical experience of writing end products that real users use. Obviously this is a huge generalization but I think it's a useful way to think about it. And when I say scientist, I mean "Professor of CS" not "24 year old with a BS in CS".
- spacemanmatt 6y ago
  Is this sentiment perhaps due to someone "practicing CS" on your engineering schedule? What's the real harm you're describing?
- TheCoelacanth 6y ago
  I think it would apply if companies were hiring large number of computer scientists and using them to try to build usable software. I don't see many making that mistake. Most recognize that computer scientists belong in a research or academic setting.
deleted 6y ago
[deleted]
mynameisash 6y ago
> What ends up happening is you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team Not to disparage the amazing data scientists I've worked with, but I've been on teams where this is very much the approach to operationalizing models. It's basically, "Here's the sklearn model and some fragile featurization scripts we built. Can you take this to prod ASAP?" The problem I've seen is that DS & DE teams were in different parts of the org and had their own sprints that were in no way connected. So they kept chucking models over the wall and we kept trying to faithfully operationalize. Once we convinced leadership that we had to collaborate from the get-go, things went a whole lot better. It also improved the working relationship of engineers and scientists. I learned a hell of a lot from the scientists; they learned how to write better code. They also learned what code they didn't need to write because I could do it faster or better than them, leaving them to focus on more important things. It was pretty amazing to find what manual processes they would setup in lieu of proper (or even any) engineering support. Again, these are amazingly smart people, but they were being square-pegged into a lot of round-hole engineering tasks. Now, the much more frustrating issue I had was being in a very data-heavy organization and being told by a distinguished engineer (my skip-level) plus my direct manager that, "data engineering isn't a real discipline." I left that org very shortly thereafter.
- civilized 6y ago
  This is 100% my experience as a data scientist. The engineering support we get is restricted to submitting a ticket for database access or moving data from one system to another. Wouldn't dream of involving an engineer in a data science project team, because I have no evidence that they have any experience or expertise in anything other than tickets to move data around.
  - Mauricebranagh 6y ago
    That's first line support not engineering
    - killtimeatwork 6y ago
aqme28 6y ago
> you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team. I feel seen. At a previous job, our output after some cleaning and transforming was a pg_dump for the data scientists to load. We had little visibility of what they did to that database once they got it.
- nitrogen 6y ago
  I suspect in rare cases this is by design, because engineers would object to the behavior of the Business Intelligence department on ethical grounds.
  - aqme28 6y ago
    If anything, we would have objected to the quality of the code they were writing.
dumb1224 6y ago
In specific research areas such as biomedical science it is certainly tricky to get involved because of the data governance / confidentiality issue... so we have to do both roles to some extent
hinkley 6y ago
This is a systemic problem. We ask non software engineers to write code, and then we expect them to apply a level of robustness and long term planning that even we have difficulty achieving. Not because we're being picky, but because we know the failure modes that are likely, and we know that people convince themselves that they aren't. We've been through this with installer writers, database admins, test automation, operations people, and now 'devops' people who were supposed to be the answer to these problems. It never stops.
- AndrewKemendo 6y ago
  It's a two way street. SWE need to learn some data practices and data folks need to learn some SWE practices.
  - hinkley 6y ago
    Oh absolutely. We'll build completely the wrong thing, but build it well (which just makes it all the harder to throw it away).
fatnoah 6y ago
>The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. Reading this thread has made me realize just how lucky I am to work very closely with strong a very strong Data Scientist, who is complemented by a very strong Data Engineer. Conversations with the Data Scientist are always about strategy, product alignment, and ensuring we're optimizing what we build for learning. The Data Engineer works very closely to ensure we're actually capturing the data we think we are, getting it to analysis systems, and making sure those data pipelines stay healthy.
bigger_cheese 6y ago
>The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. This matches my observations as well. I'm an engineer (The non software kind) at an Industrial plant, I have noticed similar in my involvement with data scientists. I think in a lot of cases it needs to be acknowledged that data scientists are not domain subject matter experts. Very often the data scientists we have worked with lack knowledge I take for granted as an engineer such as knowledge of basic chemistry, physics etc. I can sanity check plant data almost instantly. For example I will know if a material reacts in an endothermic or exothermic manner and can verify that its effect on a temperature prediction model make sense. As a result I often feel like Data Scientists are not empowered to bring their full expertise to bear, they don't understand our process fully and lack a lot "engineering" knowledge to make value added inferences about what their models are demonstrating. Often they can deliver a model and show that a particular term is significant but they have a very shallow understanding of what the term actually represents and can't provide concrete recommendations as to how we could modify our plant to benefit from what their model is demonstrating. Sometimes I feel like we need an additional translator sitting between who can speak both "Data science" and "Engineer" I don't think this is quite what "Data Engineer" as suggested by parent article is but possibly the role could be expanded to incorporate this.
ipiz0618 6y ago
As a Data Scientist, I do everything you mentioned because I came from a SWE background. I think Data Scientists, even if they are only interesting in the "fun" part of the job, should know how and why data are captured their way, so that they understand which models are better suited, which saves a lot of time.
EdwardDiego 6y ago
We have a sister company with many data scientists, and very few (actually I don't think that they ever hired any with the specific title) data engineers. And, their production alleged "machine learning" (it's pretty much standard linear regression, but calling it ML is sexy) systems are slow motion train-wrecks. If the string and duct tape holds, then it works, but it's unfortunately continually breaking. Hell, in Slack, I watch their data scientists continuously wrestle with how to actually make their Jupyter notebooks work in production. Whereas my company has far more data engineers than data scientists. The plan from higher up the corporate food chain was always that we'd give them our data to do their data science voodoo on, but we ended up getting a few data scientists of our own for specific projects. So, we focused on ensuring our data stream was reliable, consistent and sufficiently timely, for them to work on. But as soon as it hits their systems, it's a forest fire of hacks upon hacks, which inevitably break. In the end, we had to send in our data engineers to stabilise their flagship "real time reporting" product that corporate was so amped about. So yeah, I think that there's probably a happy ratio of data scientists to data engineers, of about, say, 1:5 or 1:10, because the maths generally scales O(1), it's the beauty of maths, but the actual engineering to get clean data delivered timely without breaking anything scales very differently indeed.
- Jugurtha 6y ago
  >Hell, in Slack, I watch their data scientists continuously wrestle with how to actually make their Jupyter notebooks work in production. Could you go into more details on what their struggles are? We had many problems as a company doing machine learning projects, and we built our internal platform (https://iko.ai https://iko.ai) to keep our sanity. I'm always interested in problems others may be having.
  - EdwardDiego 6y ago
    Python hell, basically. Getting the right Python version, right dependency versions etc.
    - Jugurtha 6y ago

mgh2 6y ago

Who will do the proper cleaning then?

alex_anglin 6y ago
The aspiration that GP was getting to was that less cleaning is required as a result of better data engineering, I believe.
- AndrewKemendo 6y ago
  Correct. If you build your instrumentation correctly, then you don't really need to do any "cleaning." Doesn't mean you might not need to do transformation for different uses but ideally wouldn't need to, for example change data types like turning a bool into an int.
  - dpayonk 6y ago
    If we only had to use 1st party data, that might be easier. But then again, if you’re building your product incrementally, you’re still going to have instrumentation holes that you may or may not be able to partially backfill.
  - darksaints 6y ago
    The problem is that data engineers that are geared towards analytics very very rarely control the systems that create the data. If you're lucky, you have the task of hounding a team within your company to get their data management practices in order. And the conversation there is whether they should make their job harder in order to make your job easier. Unfortunately, data engineers rarely deal with purely in-house data. You're gonna be pulling data from a variety of data sources. I can assure you that if you're pulling from government data sources, you're gonna have a hell of a time. Speaking from direct experience, my team is probably going to spend $10M/year just trying to keep a government dataset in order, because they won't do it themselves. I'm talking lawyers, legal analysts, data engineers, data scientists, data entry personnel, etc.. just to fix data that should have never been broken in the first place. It shouldn't be a shock that cleaning the data is the path of least resistance for many.
    - AndrewKemendo 6y ago
      Hence why I said DE need to be involved as early as possible. Aspirational sure, but that's what I've seen work the best and repeatably. It's the only scalable solution IMO otherwise you're perpetually playing catch-up. On the point about the govt I literally built a completely new contract type and civilian hiring practices for the DoD to bring in Data Engineers so they could do exactly what I describe to make your life easier.
  - mgh2 6y ago
    Do data engineers have good analysis skills? Do business analysts have good engineering skills? I don't think either of them can fill the data scientist role. The scientific training and mindset (scientific method, hypothesis, experiment setup, etc.) to even create an accurate model is an undervalued skill here no? Even if data cleaning is automated, these skills cannot be easily learned. There is a reason why so many PhDs get into the field, because they were trained in the exploratory/research mindset that no engineering or analytics skills can fill. Correct me if I am wrong.
    - dijksterhuis 6y ago
      > Do data engineers have good analysis skills? Yes. > Do business analysts have good engineering skills? Depends on the analyst. > I don't think either of them can fill the data scientist role. > The scientific training and mindset (scientific method, hypothesis, experiment setup, etc.) to even create an accurate model is an undervalued skill here no? Even if data cleaning is automated, these skills cannot be easily learned. It's not about replacing data scientists with data engineers, it's about both roles working together to make everything more efficient. The hiring rate for data scientists has plateaued. The industry doesn't need any more of them. Why? Because data scientists often can't solve problems fast enough. It's a commonly quoted statistic that 70% of any data science task is data cleansing and/or etl. A data engineer's job is to take that 70% and turn it into 10%. The data engineer saves the data scientist time, meaning they can focus on what they're supposed to do -- build models.
nerdponx 6y ago
It doesn't matter, as long as you don't make the person with the PhD in biostatistics spend their time writing ETL pipelines, which is a wildly inefficient use of a very expensive resource.
- names_are_hard 6y ago
  Do people with PhDs in biostatistics earn significantly more than programmers? I honestly know nothing about the market for biostatisticians, but my impression was that advanced degrees in the natural sciences don't really pay that well compared to software engineers, especially given that they're much more educated.
  - aldanor 6y ago
    If they work e.g. in a hedge fund / trading firm, then - yea. And you see lots of PhDs from unrelated fields working as quants there.
Enginerrrd 6y ago
Not to worry, corporate will just outsource to firm which hires Data Janitors

antipaul 6y ago

I have a dream - and it looks like this!

alexfromapex 6y ago

I agree, if anything the data engineers (folks with engineering backgrounds) should be doing the applied work while a department of data scientists works on the theoretical or novel data analysis methods. Right now our product has accumulated a lot of technical debt on the data validation side because data scientists designed the test code in a way that dramatically slows the development process.

watermanio 6y ago
This feels especially true when you have access to things like BigQuery ML. It's very easy for an average engineer (like me) to start using ML using these tools, but a lot harder to explain how it works, or exactly which type of models to use. In my mind a DS would be really useful to just point us in the right direction and check work. Like a super specialist QA...
listenallyall 6y ago
> novel data analysis methods Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. Asking them to develop a "novel" method would be fruitless. Asking them to clean and scrub the source data set is like telling an amateur pie-baker the store was out of pie crusts, you'll have to make your own from scratch -- it's not going to happen, they just don't have that skill, the instructions on the box don't account for that possibility. As soon as the task diverges from the simple step 1, step 2, step 3 that they were originally taught, you realize they have very little ability to adapt. YMMV of course.
- tchalla 6y ago
  > Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. This is because they rarely hire people with scientific thinking ability. They just hire people who can code and program from set recipes. Once you hire such people you can not expect them to do non-recipe work. If you don't want recipe work, don't hire people will recipe skills. Do not have job interviews that select for recipe people. But, that is exactly what most companies do.

bitcharmer 6y ago

This same sentiment (which I personally agree with) applies to software engineering. As in: engineers deliver more practical value than comp scientists. Now you can down-vote me to oblivion.

vbtemp 6y ago
You and me both, friend. Except I normally get down-voted to oblivion for saying the opposite.
- adamisom 6y ago
  I guess depends on what valuable means. I imagine most comp scientists are less replaceable than most software engineers, so point for compsci.
  - collyw 6y ago
    Depends if you have a computer scientist doing a software engineers job
  - johnqpub 6y ago
    The two are complimentary. Engineers can't do anything without the fundamental insights scientists provide. But scientists don't have the practical experience of writing end products that real users use. Obviously this is a huge generalization but I think it's a useful way to think about it. And when I say scientist, I mean "Professor of CS" not "24 year old with a BS in CS".
spacemanmatt 6y ago
Is this sentiment perhaps due to someone "practicing CS" on your engineering schedule? What's the real harm you're describing?
TheCoelacanth 6y ago
I think it would apply if companies were hiring large number of computer scientists and using them to try to build usable software. I don't see many making that mistake. Most recognize that computer scientists belong in a research or academic setting.

deleted 6y ago

[deleted]

mynameisash 6y ago

> What ends up happening is you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team Not to disparage the amazing data scientists I've worked with, but I've been on teams where this is very much the approach to operationalizing models. It's basically, "Here's the sklearn model and some fragile featurization scripts we built. Can you take this to prod ASAP?" The problem I've seen is that DS & DE teams were in different parts of the org and had their own sprints that were in no way connected. So they kept chucking models over the wall and we kept trying to faithfully operationalize. Once we convinced leadership that we had to collaborate from the get-go, things went a whole lot better. It also improved the working relationship of engineers and scientists. I learned a hell of a lot from the scientists; they learned how to write better code. They also learned what code they didn't need to write because I could do it faster or better than them, leaving them to focus on more important things. It was pretty amazing to find what manual processes they would setup in lieu of proper (or even any) engineering support. Again, these are amazingly smart people, but they were being square-pegged into a lot of round-hole engineering tasks. Now, the much more frustrating issue I had was being in a very data-heavy organization and being told by a distinguished engineer (my skip-level) plus my direct manager that, "data engineering isn't a real discipline." I left that org very shortly thereafter.

civilized 6y ago
This is 100% my experience as a data scientist. The engineering support we get is restricted to submitting a ticket for database access or moving data from one system to another. Wouldn't dream of involving an engineer in a data science project team, because I have no evidence that they have any experience or expertise in anything other than tickets to move data around.
- Mauricebranagh 6y ago
  That's first line support not engineering
  - killtimeatwork 6y ago

aqme28 6y ago

> you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team. I feel seen. At a previous job, our output after some cleaning and transforming was a pg_dump for the data scientists to load. We had little visibility of what they did to that database once they got it.

nitrogen 6y ago
I suspect in rare cases this is by design, because engineers would object to the behavior of the Business Intelligence department on ethical grounds.
- aqme28 6y ago
  If anything, we would have objected to the quality of the code they were writing.

dumb1224 6y ago

In specific research areas such as biomedical science it is certainly tricky to get involved because of the data governance / confidentiality issue... so we have to do both roles to some extent

hinkley 6y ago

This is a systemic problem. We ask non software engineers to write code, and then we expect them to apply a level of robustness and long term planning that even we have difficulty achieving. Not because we're being picky, but because we know the failure modes that are likely, and we know that people convince themselves that they aren't. We've been through this with installer writers, database admins, test automation, operations people, and now 'devops' people who were supposed to be the answer to these problems. It never stops.

AndrewKemendo 6y ago
It's a two way street. SWE need to learn some data practices and data folks need to learn some SWE practices.
- hinkley 6y ago
  Oh absolutely. We'll build completely the wrong thing, but build it well (which just makes it all the harder to throw it away).

fatnoah 6y ago

>The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. Reading this thread has made me realize just how lucky I am to work very closely with strong a very strong Data Scientist, who is complemented by a very strong Data Engineer. Conversations with the Data Scientist are always about strategy, product alignment, and ensuring we're optimizing what we build for learning. The Data Engineer works very closely to ensure we're actually capturing the data we think we are, getting it to analysis systems, and making sure those data pipelines stay healthy.

bigger_cheese 6y ago

>The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. This matches my observations as well. I'm an engineer (The non software kind) at an Industrial plant, I have noticed similar in my involvement with data scientists. I think in a lot of cases it needs to be acknowledged that data scientists are not domain subject matter experts. Very often the data scientists we have worked with lack knowledge I take for granted as an engineer such as knowledge of basic chemistry, physics etc. I can sanity check plant data almost instantly. For example I will know if a material reacts in an endothermic or exothermic manner and can verify that its effect on a temperature prediction model make sense. As a result I often feel like Data Scientists are not empowered to bring their full expertise to bear, they don't understand our process fully and lack a lot "engineering" knowledge to make value added inferences about what their models are demonstrating. Often they can deliver a model and show that a particular term is significant but they have a very shallow understanding of what the term actually represents and can't provide concrete recommendations as to how we could modify our plant to benefit from what their model is demonstrating. Sometimes I feel like we need an additional translator sitting between who can speak both "Data science" and "Engineer" I don't think this is quite what "Data Engineer" as suggested by parent article is but possibly the role could be expanded to incorporate this.

ipiz0618 6y ago

As a Data Scientist, I do everything you mentioned because I came from a SWE background. I think Data Scientists, even if they are only interesting in the "fun" part of the job, should know how and why data are captured their way, so that they understand which models are better suited, which saves a lot of time.

EdwardDiego 6y ago

We have a sister company with many data scientists, and very few (actually I don't think that they ever hired any with the specific title) data engineers. And, their production alleged "machine learning" (it's pretty much standard linear regression, but calling it ML is sexy) systems are slow motion train-wrecks. If the string and duct tape holds, then it works, but it's unfortunately continually breaking. Hell, in Slack, I watch their data scientists continuously wrestle with how to actually make their Jupyter notebooks work in production. Whereas my company has far more data engineers than data scientists. The plan from higher up the corporate food chain was always that we'd give them our data to do their data science voodoo on, but we ended up getting a few data scientists of our own for specific projects. So, we focused on ensuring our data stream was reliable, consistent and sufficiently timely, for them to work on. But as soon as it hits their systems, it's a forest fire of hacks upon hacks, which inevitably break. In the end, we had to send in our data engineers to stabilise their flagship "real time reporting" product that corporate was so amped about. So yeah, I think that there's probably a happy ratio of data scientists to data engineers, of about, say, 1:5 or 1:10, because the maths generally scales O(1), it's the beauty of maths, but the actual engineering to get clean data delivered timely without breaking anything scales very differently indeed.

Jugurtha 6y ago
>Hell, in Slack, I watch their data scientists continuously wrestle with how to actually make their Jupyter notebooks work in production. Could you go into more details on what their struggles are? We had many problems as a company doing machine learning projects, and we built our internal platform (https://iko.ai https://iko.ai) to keep our sanity. I'm always interested in problems others may be having.
- EdwardDiego 6y ago
  Python hell, basically. Getting the right Python version, right dependency versions etc.
  - Jugurtha 6y ago