3 ms·

It sounds like they have data science and data engineering in one organization. Is that team structure something that others have seen work well?

by csears 5y ago

erulabs 5y ago
One of the most interesting bits of devops work I've done was when I was embedded with a data science team. Infrastructure for data science is just so different than traditional ops - but I feel like I was able to both help the team move more quickly and also prevent them from spending all of the companies money - so at least in that case, it worked quite well. I've never understood why data science teams are typically so far removed from "normal" engineering teams. Maybe it's the DevOps kool-aide speaking, but in my opinion, teams should be more horizontal than vertical!
cromd 5y ago
I've been in orgs where it was on same team, and on different teams, both as a modeler and a data engineer. So far, I personally prefer when they're on the same team. Pros of same-team: fewer ideas "lost in translation" between data scientists and data engineers, better understanding of which datasets/flows are top priority, can sometimes share some stack components and help datascientists improve their code, better chances of getting data scientists to contribute their own batch jobs (there's just more trust as opposed to dealing with some "engineering" team that is less connected to you) Cons of same team: data engineers may not be as in-the-loop on what's happening with production datasets, may not be as tightly integrated with a devops team, may get overly caught up in "business logic" as opposed to "plumbing".
thenipper 5y ago
I work with operations research teams in a blended model of engineering being embedded with the OR Scientists. I really prefer it. Code can get to prod a lot quicker and we don’t have the “throw it over the fence to engineering” issues that can arise.
quadrature 5y ago
Data scientists are embedded in product teams and data platform engineers are in a platform engineer org
lumost 5y ago
In my experience you tend to get better engineering staff when it's one organization, along with a better customer/product focus. When it's two independent teams, you tend to get a more research focused Data Science organization and a team of engineers more focused on plumbing. Which option is better will depend on the organization goals. If you think that you have a straight research problem than a dedicated research team is useful. If you want to ship product than one team is better.

It sounds like they have data science and data engineering in one organization. Is that team structure something that others have seen work well?

by csears 5y ago

It sounds like they have data science and data engineering in one organization. Is that team structure something that others have seen work well?

erulabs 5y ago

One of the most interesting bits of devops work I've done was when I was embedded with a data science team. Infrastructure for data science is just so different than traditional ops - but I feel like I was able to both help the team move more quickly and also prevent them from spending all of the companies money - so at least in that case, it worked quite well. I've never understood why data science teams are typically so far removed from "normal" engineering teams. Maybe it's the DevOps kool-aide speaking, but in my opinion, teams should be more horizontal than vertical!

cromd 5y ago

I've been in orgs where it was on same team, and on different teams, both as a modeler and a data engineer. So far, I personally prefer when they're on the same team. Pros of same-team: fewer ideas "lost in translation" between data scientists and data engineers, better understanding of which datasets/flows are top priority, can sometimes share some stack components and help datascientists improve their code, better chances of getting data scientists to contribute their own batch jobs (there's just more trust as opposed to dealing with some "engineering" team that is less connected to you) Cons of same team: data engineers may not be as in-the-loop on what's happening with production datasets, may not be as tightly integrated with a devops team, may get overly caught up in "business logic" as opposed to "plumbing".

thenipper 5y ago

I work with operations research teams in a blended model of engineering being embedded with the OR Scientists. I really prefer it. Code can get to prod a lot quicker and we don’t have the “throw it over the fence to engineering” issues that can arise.

quadrature 5y ago

Data scientists are embedded in product teams and data platform engineers are in a platform engineer org

lumost 5y ago

In my experience you tend to get better engineering staff when it's one organization, along with a better customer/product focus. When it's two independent teams, you tend to get a more research focused Data Science organization and a team of engineers more focused on plumbing. Which option is better will depend on the organization goals. If you think that you have a straight research problem than a dedicated research team is useful. If you want to ship product than one team is better.