8 ms·
Hi, (I wrote the article). A few people commented noting that I included "Data Science" in the title, but the content doesn't include any statistics or machine
by robertelder 7y ago
Hi, (I wrote the article). A few people commented noting that I included "Data Science" in the title, but the content doesn't include any statistics or machine learning which is closer to the core definition of 'data science'. I still think the title is appropriate since any kind of low-fidelity data science task you do on some had-hoc data (log files, heaps of text, web pages) is going to start with setting up a processing pipeline that involves these commands. I could have re-named it "An intro to text processing" or "An intro to data processing", but then the people who need to see this content won't associate the title with something they're interested in, so they never benefit from it. The list of commands was chosen specifically with the question "What Linux commands would someone answering data science/business intelligence questions use?" in mind. These commands are also among the list of ones that are usually already installed on every system.
- LeftHandPath 7y agoInteresting. For anyone who is interested in going a little deeper into data science, I’d also recommend the “Introduction to Data Science with R” series by David Langer: https://youtu.be/32o0DnuRjfg https://youtu.be/32o0DnuRjfg
- NHQ 7y agoYou forgot an important one: man
- nurettin 7y agoyes, this article describes - what we used to call in the early 2000s - using linux.
- dang 7y agoNow that you mention it, 'data processing' seems more neutral and accurate, so I've put that in the title above.
- bitminer 7y agoMuch can be done just with awk. My pet peeve is the "grep | awk" idiom. No, just use awk. Awk does map/reduce, relational joins, associative memory, table lookup, and so on. Just use awk arrays, begin block, and end block.
- dvrx 7y agoIf I am going to maintain the script myself and never tweak anything in the middle of the night, sure. But most people don't know awk. And awk requires more awareness. I break my awk when I fix things when tired.