3 ms·

I would pay in solid gold for a data engineer that knew how to glue <the things the data scientists need> to <the rest of the infrastructure> in a way that fixe

I would pay in solid gold for a data engineer that knew how to glue <the things the data scientists need> to <the rest of the infrastructure> in a way that fixes the impedance mismatch that seems to exist in the tooling. In my experience data tools don't mesh well with "cloud"-y IAM, monitoring, or auditing frameworks. Data folks ssh to shared cloud workstations and of course use agent forwarding because that's what the tooling expects. They want to use EFS to share data sets even though NFS on machines where people have sudo is a bad idea / EFS is maybe a poor fit if you're thinking about governance / provenance. There's a mix of "notebooks" running locally (or on the shared workstations) and DAGs running in the cloud with bespoke access control that either doesn't map to IAM or else there's no access control so to get to the dashboard you forward a port with SSH. It's enough to make me want to wall them off in a separate AWS account, but maybe I'm just being a grumpy old SRE. edit: as I mention downthread, this is a knee-jerk reaction and is not likely to "succeed" for whatever definition of "success" your business has.

moandcompany 7y ago
Data Engineers Supporting Their Data Science Teammates "By the way, we are chartered to work separately from ____ infrastructure teams, to better enable our freedom and not hinder our own progress. We’re allowed to run our own computing environments, cluster(s), olap database(s), _____, and we’ve been doing it ourselves. Can you run/manage/provision those for us too?" From a talk I've given a few times called, "Life of a Data Engineer" (Google slides link: https://docs.google.com/presentation/d/1Oer3Z9OXPsk9H9WE5g6xlu4ATGEfEsU84xW3UMcIdd8/edit?usp=sharing https://docs.google.com/presentation/d/1Oer3Z9OXPsk9H9WE5g6x...)
- philsnow 7y ago
  > we'd be more efficient if we could get someone else to do this for us This is spot on.. I don't mean to vilify the data folks, they were hired to do a job and unless an explicit part of that job description is "make your stuff play nice with the existing stuff" (which it never is), why would they swim against the current and spend their cycles doing things that aren't going to get them promoted / that they're not naturally inclined to care about? The sentiment of the quoted statement really highlights that they want to be in their own world and do things the way that makes sense to them. Again, I can't fault them for it. It just turns into an absolutely staggering problem later when the company tries to get some certifications. Maybe engineering leadership needs to see this particular iceberg earlier.
  - sixdimensional 7y ago
    At the large-ish global company I work for, there are many siloed groups, many who are make their own demand for full control over their own isolated infrastructure, and not just for data purposes (sometimes for workflow/process automation and control, and many other reasons). The old, oft feared “shadow-IT” is a good term to represent what is actually an opportunity to recognize the opportunity for IT to reframe itself as the trusted service/platform/infrastructure provider and advisor, but perhaps not the only ones who implement and control every aspect of using that infrastructure going forward. The extent to which digital technology has become fused with work, means people need more access and control over systems and tools within their daily work, but solid infrastructure, governance, etc. is still required. Clearly the trick is finding the balance between centralization and federation. I think directions such as “citizen developer” and “data democracy” are not just fluff, but a pointer in the right direction of how to think about the newer classes of customer for information systems and technology.
sixdimensional 7y ago
I see this problem a lot, and in my experience there are at least two pieces to this puzzle - 1) many data science tools were originally desktop oriented or required specialized, siloed engines to run, causing a parallel universe of data to need to be imported into those tools and environments (for example SPSS) and 2) traditional infrastructure teams need to think of data infrastructure and architecture as a different subdiscipline running at layers 5-7 of the OSI model. My problem as an infrastructure provider and data architect is how to provide a globally consistent, governed platform and model on top of which different classes of users have different levels of access rights to data in different forms and qualities, through different interfaces. My 2 cents, I accept silver bars too lol :) For what it’s worth, I don’t buy the argument that data folks should operate in an isolated infrastructure - we just need to adapt how we serve their needs, which can be quite extensive when you are talking about essentially anything ranging from someone writing highly complex algorithmic code to process large volumes of raw data (high level of support and access may be needed) vs. someone just designing a report or dashboard using just a graphical tool on top of a predefined data model (much lower infrastructure access needed).
- philsnow 7y ago
  I agree especially on these two points: * it's counter-productive in a number of ways to create a data ghetto where they can do whatever they need to do. It doesn't engender trust or communication between Infra/SRE/DataEng/DataSci teams, and leads to "throw it over the wall" behavior. * we need to adapt to how we serve their needs, not the other way around. It's a lot more likely to be successful if we are the ones who start bridging the gap. Data engineers are pretty specialized at enabling data scientists to do their job, they don't necessarily share the same skillset as Infra/SRE engineers.
  - sixdimensional 7y ago
    Definitely agree - these folks are a class of customer and they are experiencing “pain”. It’s difficult to get there, as you have said, but I could not agree more, that we need to adapt to ease that pain. It’s actually a great opportunity.

moandcompany 7y ago

Data Engineers Supporting Their Data Science Teammates "By the way, we are chartered to work separately from ____ infrastructure teams, to better enable our freedom and not hinder our own progress. We’re allowed to run our own computing environments, cluster(s), olap database(s), _____, and we’ve been doing it ourselves. Can you run/manage/provision those for us too?" From a talk I've given a few times called, "Life of a Data Engineer" (Google slides link: https://docs.google.com/presentation/d/1Oer3Z9OXPsk9H9WE5g6xlu4ATGEfEsU84xW3UMcIdd8/edit?usp=sharing https://docs.google.com/presentation/d/1Oer3Z9OXPsk9H9WE5g6x...)

philsnow 7y ago
> we'd be more efficient if we could get someone else to do this for us This is spot on.. I don't mean to vilify the data folks, they were hired to do a job and unless an explicit part of that job description is "make your stuff play nice with the existing stuff" (which it never is), why would they swim against the current and spend their cycles doing things that aren't going to get them promoted / that they're not naturally inclined to care about? The sentiment of the quoted statement really highlights that they want to be in their own world and do things the way that makes sense to them. Again, I can't fault them for it. It just turns into an absolutely staggering problem later when the company tries to get some certifications. Maybe engineering leadership needs to see this particular iceberg earlier.
- sixdimensional 7y ago
  At the large-ish global company I work for, there are many siloed groups, many who are make their own demand for full control over their own isolated infrastructure, and not just for data purposes (sometimes for workflow/process automation and control, and many other reasons). The old, oft feared “shadow-IT” is a good term to represent what is actually an opportunity to recognize the opportunity for IT to reframe itself as the trusted service/platform/infrastructure provider and advisor, but perhaps not the only ones who implement and control every aspect of using that infrastructure going forward. The extent to which digital technology has become fused with work, means people need more access and control over systems and tools within their daily work, but solid infrastructure, governance, etc. is still required. Clearly the trick is finding the balance between centralization and federation. I think directions such as “citizen developer” and “data democracy” are not just fluff, but a pointer in the right direction of how to think about the newer classes of customer for information systems and technology.

sixdimensional 7y ago

I see this problem a lot, and in my experience there are at least two pieces to this puzzle - 1) many data science tools were originally desktop oriented or required specialized, siloed engines to run, causing a parallel universe of data to need to be imported into those tools and environments (for example SPSS) and 2) traditional infrastructure teams need to think of data infrastructure and architecture as a different subdiscipline running at layers 5-7 of the OSI model. My problem as an infrastructure provider and data architect is how to provide a globally consistent, governed platform and model on top of which different classes of users have different levels of access rights to data in different forms and qualities, through different interfaces. My 2 cents, I accept silver bars too lol :) For what it’s worth, I don’t buy the argument that data folks should operate in an isolated infrastructure - we just need to adapt how we serve their needs, which can be quite extensive when you are talking about essentially anything ranging from someone writing highly complex algorithmic code to process large volumes of raw data (high level of support and access may be needed) vs. someone just designing a report or dashboard using just a graphical tool on top of a predefined data model (much lower infrastructure access needed).

philsnow 7y ago
I agree especially on these two points: * it's counter-productive in a number of ways to create a data ghetto where they can do whatever they need to do. It doesn't engender trust or communication between Infra/SRE/DataEng/DataSci teams, and leads to "throw it over the wall" behavior. * we need to adapt to how we serve their needs, not the other way around. It's a lot more likely to be successful if we are the ones who start bridging the gap. Data engineers are pretty specialized at enabling data scientists to do their job, they don't necessarily share the same skillset as Infra/SRE engineers.
- sixdimensional 7y ago
  Definitely agree - these folks are a class of customer and they are experiencing “pain”. It’s difficult to get there, as you have said, but I could not agree more, that we need to adapt to ease that pain. It’s actually a great opportunity.