4 ms·

Dude, do you even data science? Not knowing Unix tools like cut and sort is a hard fail on a senior individual contributor in data science role, as is using sq

Dude, do you even data science? Not knowing Unix tools like cut and sort is a hard fail on a senior individual contributor in data science role, as is using sqlite which totally doesn't scale the way sort and cut does. Separates sheep from goats in data science land. You should really learn them if you're in the field and work with reasonably big data sets. Frankly you should learn them if you work with data at all, ever. I've literally seen FAANGs recommendation engines powered with these tools running nightly on someone's desktop.

BlueTemplar 7y ago
Hmm, tell us more? Databases are generally known for their good performance (under locking conditions - this is the relevant factor I guess?), after all...
- redis_mlc 7y ago
  > Databases are generally known for their good performance If the data is on a filesystem, then sed, grep and cut pipelines will likely be your fastest option (Yahoo! processed petabytes of logfiles for decades that way.) If the data is already inside a database table and indexed well, that could be fast enough. But generally speaking, the ETL is often a bottleneck. And DBAs are $$$$ compared to "the UNIX way." Source: DBA
ansgri 7y ago
Well, they do work, but streaming processing in Python is not difficult and it’s much easier to have graceful failure processing and self-documenting code. Not to say of extending the logic if it would ever be needed. But maybe that’s more about separating sheep from shepherds.
Aeolun 7y ago
I’m not sure if I should be horrified or not. Both by the fact this happens, and by the fact that you seem proud of it. Learning sort and cut takes literally takes all of 10 minutes, so if it makes you pass over an otherwise qualified candidate you have your priorities completely backwards.
- scottlocklin 7y ago
  Feel free to be horrified: a data scientist who doesn't understand where and why to use unix command line tools for data preparation and ETL is about as useful to me as one who doesn't understand the conditions where a t-test breaks down or what a ROC curve is. Generally speaking, people like this have never actually dealt with large data sets, never dealt with issues involved with installing "unapproved software" on a machine (ridiculously common in The Real World), has probably never cleaned a dirty data set (what do you do when your giant csv is formatted in a way that Wes McKinney didn't think of?), and will in a senior role be a long term liability for a data science team that works on serious problems. Sure at one point I didn't know about them either: I wasn't a senior data scientist then. I submit that if you don't know about them and haven't actually used them, you aren't either.
  - blub 7y ago
    I think that the people not being impressed by cut and sort are approaching this from the Linux end of things, where those tools are nothing special at all. I guess we kind of expected that the data science wizards would be using fancier tools.
    - scottlocklin 7y ago
      Yeah, well, people who have enough self regard to think of themselves as "wizards" are super unlikely to be able to actually do the day to day grind of getting, cleaning and preparing data for feature generation, which is about 95% of the job. Another good weeder for a person claiming to be senior: discuss how you would fix the performance of the default R naive Bayes implementation in e1071. It's numerically more or less correct, but written by deranged ape-men who don't understand how computers work (a problem in a lot of the R ecosystem; in the Python ecosystem, the problem is nobody has yet written algorithms for X, which ends up being a very similar problem: aka it's your job to code up sane algorithms).
- whalabi 7y ago
  So I think this is a perfect example of what happens in tech interviews. OP is using knowledge of a specific technology as a heuristic for "has experience in role x" But this always makes me wonder, couldn't you see that experience from a resume? If the candidate filled a data science role at somewhere reputable for 3 years, and you verify that they successfully filled that role, why rely on that heuristic? As you say testing for the specific technology, when it can be learnt in 10 minutes, does not seem logical.

BlueTemplar 7y ago

Hmm, tell us more? Databases are generally known for their good performance (under locking conditions - this is the relevant factor I guess?), after all...

redis_mlc 7y ago
> Databases are generally known for their good performance If the data is on a filesystem, then sed, grep and cut pipelines will likely be your fastest option (Yahoo! processed petabytes of logfiles for decades that way.) If the data is already inside a database table and indexed well, that could be fast enough. But generally speaking, the ETL is often a bottleneck. And DBAs are $$$$ compared to "the UNIX way." Source: DBA

ansgri 7y ago

Well, they do work, but streaming processing in Python is not difficult and it’s much easier to have graceful failure processing and self-documenting code. Not to say of extending the logic if it would ever be needed. But maybe that’s more about separating sheep from shepherds.

Aeolun 7y ago

I’m not sure if I should be horrified or not. Both by the fact this happens, and by the fact that you seem proud of it. Learning sort and cut takes literally takes all of 10 minutes, so if it makes you pass over an otherwise qualified candidate you have your priorities completely backwards.

scottlocklin 7y ago
Feel free to be horrified: a data scientist who doesn't understand where and why to use unix command line tools for data preparation and ETL is about as useful to me as one who doesn't understand the conditions where a t-test breaks down or what a ROC curve is. Generally speaking, people like this have never actually dealt with large data sets, never dealt with issues involved with installing "unapproved software" on a machine (ridiculously common in The Real World), has probably never cleaned a dirty data set (what do you do when your giant csv is formatted in a way that Wes McKinney didn't think of?), and will in a senior role be a long term liability for a data science team that works on serious problems. Sure at one point I didn't know about them either: I wasn't a senior data scientist then. I submit that if you don't know about them and haven't actually used them, you aren't either.
- blub 7y ago
  I think that the people not being impressed by cut and sort are approaching this from the Linux end of things, where those tools are nothing special at all. I guess we kind of expected that the data science wizards would be using fancier tools.
  - scottlocklin 7y ago
    Yeah, well, people who have enough self regard to think of themselves as "wizards" are super unlikely to be able to actually do the day to day grind of getting, cleaning and preparing data for feature generation, which is about 95% of the job. Another good weeder for a person claiming to be senior: discuss how you would fix the performance of the default R naive Bayes implementation in e1071. It's numerically more or less correct, but written by deranged ape-men who don't understand how computers work (a problem in a lot of the R ecosystem; in the Python ecosystem, the problem is nobody has yet written algorithms for X, which ends up being a very similar problem: aka it's your job to code up sane algorithms).
whalabi 7y ago
So I think this is a perfect example of what happens in tech interviews. OP is using knowledge of a specific technology as a heuristic for "has experience in role x" But this always makes me wonder, couldn't you see that experience from a resume? If the candidate filled a data science role at somewhere reputable for 3 years, and you verify that they successfully filled that role, why rely on that heuristic? As you say testing for the specific technology, when it can be learnt in 10 minutes, does not seem logical.