3 ms·

I agree with this. I think you really need to be a good data engineer to be a good data scientist. Now, these terms are different roles at different organizatio

• edmundsauto 5y ago
  Do you think it makes sense for someone to combine these skills? My (unexamined) belief is that its better to have them split out. My reasoning: 1. Data science is often low latency work, running investigations to provide insight for strategic decisions. This is more of a political/comms role - and the cadence/requirements of the role tend to be counter to the long periods of uninterrupted focus that an engineer needs. 2. DE work is foundational, but once it's built, you probably want someone who is fast at adhoc analysis. (Unless you want to push into production - but that's a 3rd skillset, IMO) 3. It's hard to be both user and designer; having two brains has saved me from many mistakes related to bias and assumptions that I make. The DS can function almost as a product manager, synthesizing information and focusing on deliverables (the "what"). The DE can focus on the how. Would love to hear your thoughts - this is an area that I'm very interested in.
  • nerdponx 5y ago
    I agree with the fact that they need to be treated as separate roles, although a lot of organizations just don't understand this well enough and you end up having data engineers doing data science, or data scientists doing data engineering, or both, maybe without the teams ever really talking to each other. However I think that's a somewhat different issue from the one under discussion here. The problem here is that data scientists can end up wasting time trying to apply fancy techniques, when the real impact would come from basic data analysis using old-school statistics. Interestingly, I have never encountered this in my personal experience. I don't personally know any data scientist who would rather use in neural network win something simpler would suffice. Also, I disagree with the idea that data scientists don't need long periods of concentration. If anything, data science is more of a "creative" enterprise, whereas coding tends to be somewhat methodical and mechanical. For me at least, that means long periods of time spent trying different things and contemplating, which requires just as much deep focus as writing a complicated piece of software.
  • mateo411 5y ago
    I think if you have the desire and the time to combine these skills, then it's a good idea to do this. If you enjoy building and optimizing pipelines and if you enjoy doing creating analytics, stats and building models, you will be very much in demand and it will be pretty lucrative. Most people don't do this, because it takes a long time, and they enjoy one over the other. 1. This really depends on the DS role at a company. You are describing a company where DS is highly valued and has a lot of power. They probably have representation with a VP or in the C suite. Some companies are like this. 2. DE should be foundational, and they should also be able to push to production. You might have another team which works on setting up new platform environments, but DE better be comfortable pushing and debugging issues in production. It's helpful for DE to be able to perform exploratory analysis on a dataset. It's easy to introduce bugs, if DE can catch this, then it really improves the velocity. If they can't do this, then you need to wait for DS or somebody to act like QA for a dataset. You can find cheaper DEs who are good at shoveling data around, but they don't know what a probability distribution is. 3. It's helpful for DS to have some domain knowledge. If they have that, and DS is highly valued at the company, then it makes sense for the DS role to contain a Product Management component. I'm glad you are interested in my thoughts on this. If you have any more follow up questions, feel free to ask.
  • zwaps 5y ago
    It depends a bit on the definition of the roles, however this is basically what I am arguing against. I think you are right that there are organizational trade-offs, especially in day-to-day operations. However, I would claim that if DE and DS are divorced enough, there is essentially no way that DS work can be done correctly and it would be up to chance whether the DS division in your firm actually does useful work, or does all the things that the article bemoans. Much frustration with Data Science (also as indicated in the article) arise because DS is seen as an ad-hoc creative enterprise to squeeze "insights" out of an existing data-repository and pipeline. Instead, I'd argue, that anything that touches data needs to be handled by both a DE and a DS (not necessarily the same person). There are fundamental reasons for this. If you really want to draw insights from data, then you can not simply take some data set and go at it. From a standpoint of statistical identification, data and model go hand in hand. It's not a model that is identified, it's a model on a specific statistical process (that results in a set of data). And frequently, thinking about how to generate and process the data is much more important than the choice of which model to run. In other words, DS need to be involved from the moment the data starts to exist as such, in particular in the moment the data is collected or generated.Otherwise, DS are working blindfolded - which is really the norm right now. Think of it this way: if the DS process in your company starts with the tabular data that goes into the model, then there is essentially no way that the process of creating a statistical model can be right. If the outcome works, it is as much chance as it is skill. If the work of DS is defined by starting with some mysterious yet usable set of input data, collected somewhere else "in the machine" and squeezed together for the DS's convenience, then I highly doubt that DS will be a popular job description in the next ten years! Again, this does not mean DS and DE should be the same person. But since many points about data are as important as they are subtle, an organizational disconnect is at least risky. Edit: Currently the endorsed process is to build a model, eventually be surprised by "data drift" and then rebuild everything. In my view, this is all backwards. Sure, data changes and one needs to iterate. But this is something the DS team should have been aware of beforehand (in most cases). What I mean here is that the surprise factor conclusively shows that no one really thought about crucial properties of the data while using it. From a statistical standpoint, this casts great doubts on these sort of analyses and models. If you are surprised by data drift, at least to the degree that it seems totally unanticipated, most likely you spent too little time on data and too much time on model. We all know examples of this. You get "some" data collected on marketing impact on young adults and you find out that doing X does Y. And then later on, the whole thing stops working. NOW you begin an error analysis and find out that data was collected in some semi-random fashion from phone users in the US, whereas now you have new audiences from Twitch (or whatever) in SE Asia. The problem here is not the model. The problem is also not that you need to iterate. The problem is that through the opaque process of data generation no one knew, could know, or cared to know what exactly the properties of the data were. Right now we are using an ad-hoc approach. Either, because nobody really knows how to characterize the data and what implications for inference this has (for example because statistical issues are dense, poorly communicated and consequently rarely taught in anything ML and DS), or because the organizational disconnect from data collection makes it impossible to reason about these issues.