4 ms·
I see this problem a lot, and in my experience there are at least two pieces to this puzzle - 1) many data science tools were originally desktop oriented or req
by sixdimensional 7y ago
I see this problem a lot, and in my experience there are at least two pieces to this puzzle - 1) many data science tools were originally desktop oriented or required specialized, siloed engines to run, causing a parallel universe of data to need to be imported into those tools and environments (for example SPSS) and 2) traditional infrastructure teams need to think of data infrastructure and architecture as a different subdiscipline running at layers 5-7 of the OSI model.
My problem as an infrastructure provider and data architect is how to provide a globally consistent, governed platform and model on top of which different classes of users have different levels of access rights to data in different forms and qualities, through different interfaces.
My 2 cents, I accept silver bars too lol :)
For what it’s worth, I don’t buy the argument that data folks should operate in an isolated infrastructure - we just need to adapt how we serve their needs, which can be quite extensive when you are talking about essentially anything ranging from someone writing highly complex algorithmic code to process large volumes of raw data (high level of support and access may be needed) vs. someone just designing a report or dashboard using just a graphical tool on top of a predefined data model (much lower infrastructure access needed).
- philsnow 7y agoI agree especially on these two points: * it's counter-productive in a number of ways to create a data ghetto where they can do whatever they need to do. It doesn't engender trust or communication between Infra/SRE/DataEng/DataSci teams, and leads to "throw it over the wall" behavior. * we need to adapt to how we serve their needs, not the other way around. It's a lot more likely to be successful if we are the ones who start bridging the gap. Data engineers are pretty specialized at enabling data scientists to do their job, they don't necessarily share the same skillset as Infra/SRE engineers.
- sixdimensional 7y agoDefinitely agree - these folks are a class of customer and they are experiencing “pain”. It’s difficult to get there, as you have said, but I could not agree more, that we need to adapt to ease that pain. It’s actually a great opportunity.
- closeparen 7y agoWe just have a big Hadoop cluster with a big Airflow instance, and nice frontends for submitting adhoc Spark jobs and SQL queries before they get checked in and productionized under Airflow. What more do you really need?