4 ms·
Agreed. The second point about pipelines stuck out to me: > [Good DS] will often build these pipelines themselves. Bad DS thinks it is someone else’s job. In
by monkeybutton 5y ago
Agreed. The second point about pipelines stuck out to me:
> [Good DS] will often build these pipelines themselves. Bad DS thinks it is someone else’s job.
In a small environment, sure, do the job so it gets done! But in larger more corporate settings the 'cowboy' approach to pipeline building is not sustainable or even feasible. Am I a bad DS because I can't provision VMs, open firewalls, replicate production DBs and build hooks in other teams' services to expose data? No, its not my job. A good DS collaborates with other teams and sysadmins to build a pipeline that is maintainable and monitorable, and doesn't do it all themselves.
- ian-whitestone 5y agoCompletely agree a DS shouldn't be responsible for all those things you mentioned (all the infrastructure, networking, security, etc.). In today's age, "data pipelines" can look like plugging into tools & frameworks that abstract away a lot of this complexity. For example - fivetran for extraction & dbt for data modelling. So once those engineering/devops heavy things you listed are abstracted away, I think it's fair for a DS to be responsible for building the data transformation logic in the pipeline. Of course, not all companies will have such tooling available, in which case it's not fair to place those expectations on a DS.
- nerdponx 5y agoSpending all your time on the clock acting as your own database admin, data engineer, app developer, sysadmin, etc. is a great way to accomplish very little data science and burn yourself out in a hurry. Source: personal experience. Are these useful skills to have? Yes, especially if it gets you out of a bad situation and makes a customer very happy. Are these requisites for being a "good" data scientist? No; arguably, they are distractions and wasted room in your brain.