3 ms·

Data Engineers Supporting Their Data Science Teammates "By the way, we are chartered to work separately from ____ infrastructure teams, to better enable our fr

Data Engineers Supporting Their Data Science Teammates "By the way, we are chartered to work separately from ____ infrastructure teams, to better enable our freedom and not hinder our own progress. We’re allowed to run our own computing environments, cluster(s), olap database(s), _____, and we’ve been doing it ourselves. Can you run/manage/provision those for us too?" From a talk I've given a few times called, "Life of a Data Engineer" (Google slides link: https://docs.google.com/presentation/d/1Oer3Z9OXPsk9H9WE5g6xlu4ATGEfEsU84xW3UMcIdd8/edit?usp=sharing https://docs.google.com/presentation/d/1Oer3Z9OXPsk9H9WE5g6x...)

philsnow 7y ago
> we'd be more efficient if we could get someone else to do this for us This is spot on.. I don't mean to vilify the data folks, they were hired to do a job and unless an explicit part of that job description is "make your stuff play nice with the existing stuff" (which it never is), why would they swim against the current and spend their cycles doing things that aren't going to get them promoted / that they're not naturally inclined to care about? The sentiment of the quoted statement really highlights that they want to be in their own world and do things the way that makes sense to them. Again, I can't fault them for it. It just turns into an absolutely staggering problem later when the company tries to get some certifications. Maybe engineering leadership needs to see this particular iceberg earlier.
- sixdimensional 7y ago
  At the large-ish global company I work for, there are many siloed groups, many who are make their own demand for full control over their own isolated infrastructure, and not just for data purposes (sometimes for workflow/process automation and control, and many other reasons). The old, oft feared “shadow-IT” is a good term to represent what is actually an opportunity to recognize the opportunity for IT to reframe itself as the trusted service/platform/infrastructure provider and advisor, but perhaps not the only ones who implement and control every aspect of using that infrastructure going forward. The extent to which digital technology has become fused with work, means people need more access and control over systems and tools within their daily work, but solid infrastructure, governance, etc. is still required. Clearly the trick is finding the balance between centralization and federation. I think directions such as “citizen developer” and “data democracy” are not just fluff, but a pointer in the right direction of how to think about the newer classes of customer for information systems and technology.

philsnow 7y ago

> we'd be more efficient if we could get someone else to do this for us This is spot on.. I don't mean to vilify the data folks, they were hired to do a job and unless an explicit part of that job description is "make your stuff play nice with the existing stuff" (which it never is), why would they swim against the current and spend their cycles doing things that aren't going to get them promoted / that they're not naturally inclined to care about? The sentiment of the quoted statement really highlights that they want to be in their own world and do things the way that makes sense to them. Again, I can't fault them for it. It just turns into an absolutely staggering problem later when the company tries to get some certifications. Maybe engineering leadership needs to see this particular iceberg earlier.

sixdimensional 7y ago
At the large-ish global company I work for, there are many siloed groups, many who are make their own demand for full control over their own isolated infrastructure, and not just for data purposes (sometimes for workflow/process automation and control, and many other reasons). The old, oft feared “shadow-IT” is a good term to represent what is actually an opportunity to recognize the opportunity for IT to reframe itself as the trusted service/platform/infrastructure provider and advisor, but perhaps not the only ones who implement and control every aspect of using that infrastructure going forward. The extent to which digital technology has become fused with work, means people need more access and control over systems and tools within their daily work, but solid infrastructure, governance, etc. is still required. Clearly the trick is finding the balance between centralization and federation. I think directions such as “citizen developer” and “data democracy” are not just fluff, but a pointer in the right direction of how to think about the newer classes of customer for information systems and technology.