3 ms·
It's not a straw man, I've seen it clear as day in several companies. When it comes to data science, it's "garbage in, garbage out". I've seen companies do lots
by hn_throwaway_99 6y ago
It's not a straw man, I've seen it clear as day in several companies. When it comes to data science, it's "garbage in, garbage out". I've seen companies do lots of "data science" with a bunch of data scientists skilled in python and jupyter notebooks, only to discover a ton of work was useless because the incoming event data was tagged incorrectly due to a bug.
The actual process of collecting, aggregating, cleaning and verifying data is a hugely important skill, and not one I've really seen typical data scientists possess.
- avs733 6y agoI have experienced the same thing...but I just don't think it has anything to do with whether the positions are labeled data scientist or data engineer. And I would warn you from my experience teaching statistics to undergraduate engineers...they are not going to be much better. Regularly get 'hey we have this data what test can we run?' 'what are you trying to show?' 'we don't care we just need to run a statistical test' conversations.
- hn_throwaway_99 6y agoTo be clear, I totally agree with you. I wasn't just arguing for changing labels, I was arguing that there is one set of "engineering" focused skills (e.g. building data pipelines, data warehouses, tagging events, etc.) and a different set of analysis skills (e.g. machine learning, statistical tests, etc.) and you shouldn't over-index on the latter without having enough of the former.
- finnthehuman 6y ago>The actual process of collecting, aggregating, cleaning and verifying data is a hugely important skill, and not one I've really seen typical data scientists possess. Then they are not scientists. They have a label "scientist" but lack of rigor of actual science. I don't see why changing the label to "engineer" would suddenly make them have rigor.
- avs733 6y agoRight?! This is sort of the meta failure of the argument. They are arguing that people's data skillsets are wrong. To make that argument they are analyzing based on the wrong variable in a data set.
- deleted 6y ago[deleted]
- bonniemuffin 6y agoI suspect this may actually be an issue of school vs real world rather than scientist vs engineer. Data in the classroom setting is pristine and beautiful; data in the real world is messy and buggy. You have to get burned by buggy data a few times (or maybe a bunch of times) in the real world to learn to look for bad data smells -- I don't think schools effectively teach this kind of intuition, regardless of whether the students are training as data engineers or data scientists. If data scientists are spending more time in school getting advanced degrees, they're not getting as much exposure to buggy data, whereas data engineers with a BS and a few years of industry experience would already have built up this skill.
- avs733 6y ago>Data in the classroom setting is pristine and beautiful; data in the real world is messy and buggy. I got to take over our department's undergraduate statistics course a few years back. The first change I made was all homework, tests, and projects used real data set. I intentionally have them collect bad data (they don't know its bad before hand). First day of class we collect data using the board game operation...I give basic instructions and then halfway through ask everyone to stop and agree on how they are entering data for the variable of 'success or failure' of the surgery. Oops... In my experience teaching the course, the reason the students (engineers) find statistical reasoning hard is: * They have never been given anything 'broken', everything is curated to avoid things not working. The result is they think data has inherent meaning. A right answer. * Their entire learning experience has been stripped of context and the need to make decisions with information. They can give me a p value but are terrified (not unable, just unwilling) to interpret it or give it meaning. * They have never encountered the concept of variability...everything is presented as systems with exact inputs and outputs. When I work with postdocs, I sometimes (less frequently) encounter many of the same challenges. Data is treated as sacred and external and inherent. It's wild to me.