4 ms·

The author's pie chart showing data science to be 60% data manipulation is accurate. The biggest gap between good and bad data scientist is their comfort level

The author's pie chart showing data science to be 60% data manipulation is accurate. The biggest gap between good and bad data scientist is their comfort level with data wrangling. When interviewing candidates for data science positions, one of the simplest questions is to have them sort a 1 GB tab-delimited file. 1. Poor candidates will try to open the file in Excel. 2a. Marginal candidates will use R or Stata. 2b. Okay candidates will use a scripting language like Python. 3. Good candidates will use Unix sort. To my knowledge, there are no university courses teaching the Unix toolchain and it remains very much a skill learned through practice.

blackbear_ 6y ago
Not sure why you think a candidate who uses R is inferior to one who uses Python? Also, a really good candidate should use the right tool for the job, so if you expect them to use Unix sort you should somehow imply a situation where that is the best approach.
- dgellow 6y ago
  I would also add: if it’s a one time thing, I would just do it in visual studio code or any other editor that doesn’t die on a 1GB file. And I have 12+ years experience with bash and unix tools, so it’s not about a lack of knowledge or experience. There isn’t anything magical regarding “sort” versus another tool, if there isn’t a need for automation they are equivalent.
- deleted 6y ago
  [deleted]
- whym 6y ago
  > if you expect them to use Unix sort you should somehow imply a situation where that is the best approach. I think the implied question is whether the interviewee is aware of the fact that trying to load a 1 GB text file could use up too much RAM space of the system. Unix sort is arguably the most memory efficient among the 4 choices there. It depends on the amount of available resources (which was not specified), and some companies might be willing to casually let people use 100 GB machines, though.
  - blackbear_ 6y ago
    That is a fair point. A good script would be to move towards this scenario of more data than RAM and see what the candidate comes up with.
jsinai 6y ago
In my experience the real meat of data manipulation/wrangling is in extracting statistical value from the data. If the data is noisy, what aggregations and what filters produce the best signal without smudging the underlying statistical properties? While this is a good skill to have and is a good sign of efficiency, for me the deal breaker is how well the purpose of data transformations is understood and the care to which statistical value is extracted and tested.
em500 6y ago
Demonstrating again that interviewing is mostly about confirming the prejudices of the interviewer.
oli5679 6y ago
Why is chunking, with Pandas, a bad choice?
- jmatthews 6y ago
  It's not. Welcome to okay-land.
jmatthews 6y ago
This may doom me as a merely okay candidate but a simple paging strategy in python trivializes the problem and has the added benefit of probably being the "okay-est" tool for whatever the next transform is as well.

blackbear_ 6y ago

Not sure why you think a candidate who uses R is inferior to one who uses Python? Also, a really good candidate should use the right tool for the job, so if you expect them to use Unix sort you should somehow imply a situation where that is the best approach.

dgellow 6y ago
I would also add: if it’s a one time thing, I would just do it in visual studio code or any other editor that doesn’t die on a 1GB file. And I have 12+ years experience with bash and unix tools, so it’s not about a lack of knowledge or experience. There isn’t anything magical regarding “sort” versus another tool, if there isn’t a need for automation they are equivalent.
deleted 6y ago
[deleted]
whym 6y ago
> if you expect them to use Unix sort you should somehow imply a situation where that is the best approach. I think the implied question is whether the interviewee is aware of the fact that trying to load a 1 GB text file could use up too much RAM space of the system. Unix sort is arguably the most memory efficient among the 4 choices there. It depends on the amount of available resources (which was not specified), and some companies might be willing to casually let people use 100 GB machines, though.
- blackbear_ 6y ago
  That is a fair point. A good script would be to move towards this scenario of more data than RAM and see what the candidate comes up with.

jsinai 6y ago

In my experience the real meat of data manipulation/wrangling is in extracting statistical value from the data. If the data is noisy, what aggregations and what filters produce the best signal without smudging the underlying statistical properties? While this is a good skill to have and is a good sign of efficiency, for me the deal breaker is how well the purpose of data transformations is understood and the care to which statistical value is extracted and tested.

em500 6y ago

Demonstrating again that interviewing is mostly about confirming the prejudices of the interviewer.

oli5679 6y ago

Why is chunking, with Pandas, a bad choice?

jmatthews 6y ago
It's not. Welcome to okay-land.

jmatthews 6y ago

This may doom me as a merely okay candidate but a simple paging strategy in python trivializes the problem and has the added benefit of probably being the "okay-est" tool for whatever the next transform is as well.