5 ms·

As a DS, I collect and clean my own data (sometimes literally as they’re coming off the upload line, if I’m not building the upload pipeline in question too), s

As a DS, I collect and clean my own data (sometimes literally as they’re coming off the upload line, if I’m not building the upload pipeline in question too), serving the raw data AS WELL AS metrics/models/algorithms Generated by notebooks/containers from raw data in from hive via Spark queries.

luckydata 7y ago
do you also instrument your own monitoring, are on call if one of your models breaks and have built the system where you "just drop a container"? If you are, then you are a data engineer too, otherwise you're standing on the shoulders of your data eng team and they make it look easy for you. Go buy them some donuts first time you go back to the office.
- Feyn_man_ 7y ago
  >>> do you also instrument your own monitoring, Our architect wrote some cute Datadog wrappers that I implement in every pipe I roll out (and is that instrumental in diagnosing bugs). He also programmed in a call to keys in a store that times out from too many requests and hangs the whole function the wrapper is decorated on, that took me a while to diagnose pinpoint >>>are on call if one of your models breaks and have built the system where you "just drop a container"? Our motto is ‘you wrote it, you fix it!’ If the container pipe works, and you dropped a bomb in it that doesn’t work, why should the DE have to pickup the DS’s garbage? >>>If you are, then you are a data engineer too, :D >>>otherwise you're standing on the shoulders of your data eng team and they make it look easy for you. Go buy them some donuts first time you go back to the office. I actually support a few data scientists in the manner you described above where they generate some metrics notebooks or container and I have to diagnose their crap in a DE capacity (recent problem involved optimizing their pyspark code to be less memory intensive)
  - luckydata 7y ago
    Then they should buy you some donuts!
threeseed 7y ago
Data Scientist absolutely will do data engineering as part of their job. But Data Engineers will able to make sure that data collection and data cleansing runs optimally and successfully every day. And that's a real skill when you have upstream schemas changing, fluctuations in data volume, instabilities in systems etc.

luckydata 7y ago

do you also instrument your own monitoring, are on call if one of your models breaks and have built the system where you "just drop a container"? If you are, then you are a data engineer too, otherwise you're standing on the shoulders of your data eng team and they make it look easy for you. Go buy them some donuts first time you go back to the office.

Feyn_man_ 7y ago
>>> do you also instrument your own monitoring, Our architect wrote some cute Datadog wrappers that I implement in every pipe I roll out (and is that instrumental in diagnosing bugs). He also programmed in a call to keys in a store that times out from too many requests and hangs the whole function the wrapper is decorated on, that took me a while to diagnose pinpoint >>>are on call if one of your models breaks and have built the system where you "just drop a container"? Our motto is ‘you wrote it, you fix it!’ If the container pipe works, and you dropped a bomb in it that doesn’t work, why should the DE have to pickup the DS’s garbage? >>>If you are, then you are a data engineer too, :D >>>otherwise you're standing on the shoulders of your data eng team and they make it look easy for you. Go buy them some donuts first time you go back to the office. I actually support a few data scientists in the manner you described above where they generate some metrics notebooks or container and I have to diagnose their crap in a DE capacity (recent problem involved optimizing their pyspark code to be less memory intensive)
- luckydata 7y ago
  Then they should buy you some donuts!

threeseed 7y ago

Data Scientist absolutely will do data engineering as part of their job. But Data Engineers will able to make sure that data collection and data cleansing runs optimally and successfully every day. And that's a real skill when you have upstream schemas changing, fluctuations in data volume, instabilities in systems etc.