33 ms·

Exploratory data analysis for humanities data

177 points by yarapavan 3y ago

yarapavan 3y ago
Course website (linked from the article): https://www.hum307.com/ https://www.hum307.com/
wslh 3y ago
Ah! That is awkard! Sorry, I couldn't resist, I have all respect for Awk.
smlavine 3y ago
If I were at Princeton, I would take every one of Kernighan's classes that I could! I wonder if that's a problem there.
- EvanKelly 3y ago
  I'm at this point 15 years removed, but Prof Kernighan was one of the most accessible professors and taught the most popular CS survey course (333). I have at least a half a dozen times where I was pointed his direction from another professor and Kernighan spent an hour with me looking into how to scrape a dynamic website for my auction theory project. When he was stumped he introduced me to a professor at another school who he knew had looked into the topic.
mbb70 3y ago
I'm all for this kind of exploratory hacking around before booting up python/R/Excel/duckdb, especially in constrained environments. A classic pain point is having to deal with column numbers, so I'll share my favorite trick: `head -n1 /path/to/file.csv | tr ',' '\n' | nl | grep desired_column` gives you the column number of desired_column
- chaps 3y ago
  grep -n also works in place of `nl`!
- tejtm 3y ago
  yep, without knowing about `nl` I used `...| grep -n column_header` or `...|grep -n .` to replicate the 'nl' behavior. edit: I like your 'nl' better as it is using white space instead of colon as a separator.
  - bear8642 3y ago
    oh, didn't realise you could do that - I've tended to use `... |awk '{print $NF, $0}'` to add line numbers to something
- patrec 3y ago
  Unless there is a quoted comma or an empty column beforehand (nl “helpfully” skips empty lines for numbering purposes).
- deleted 3y ago
  [deleted]
- i15e 3y ago
  Something to watch out for with nl is that by default it doesn't number empty lines. e.g.: $ printf 'one\n\nthree\n' | nl 1 one 2 three Set -ba to enable numbering all lines. For this use case I usually end up running cat -n instead since I find it easier to remember.
qsort 3y ago
The article nails down a very real pain point with libraries like Pandas: > looping over a set of input lines seems more natural than the dataframe selectors that Pandas favors Row-oriented operations, as opposed to aggregations and other OLAP-style queries are kind of painful. The generator machinery (yield from) is a partial fix to this, but Pandas itself offers little relief.
- esafak 3y ago
  pandas has a poor API. I'd rather use SQL with DuckDB.
  - slt2021 3y ago
    pandas is way more powerful that most people use it. when you have to deal with thousands of text files, mish mash of csv, tsv, some rows overlap between the files, some files spread across multiple different locations (shared drive, s3 bucket, URL, SQL db, etc), with column names that look similar but not exactly similar - this is perfect use case for pandas. read csv file? just pd.read_csv() read and concat N csv files? just pd.concat([pd.read_csv(f) for f in glob("*.csv")]) read parquet or read_sql()? not a problem at all. need to do some custom rules for data cleansing, or regex matching or fuzzy matching on column names, converting data from/to csv/parquet/sql - it will be pandas 1 liner a lot of painful data processing/cleaning, correcting data is just 1-liner in pandas, and I dont know of better tool that can beat pandas - probably tidyR but it is essentially same pandas just for R
    - esafak 3y ago
      You can do that with other tools too. https://duckdb.org/docs/data/csv/overview.html https://duckdb.org/docs/data/csv/overview.html https://duckdb.org/docs/data/parquet/overview https://duckdb.org/docs/data/parquet/overview https://duckdb.org/docs/data/multiple_files/overview.html https://duckdb.org/docs/data/multiple_files/overview.html
      - slt2021 3y ago
pmarreck 3y ago
Recent Awk convert (after, like most people, just using it for one-liners for years); it's aged remarkably well (although I wish it used more functional constructs, permitted proper variable initialization, and had interrupt handling... but at that point, it's probably best to switch to a "full" language...)
jph 3y ago
Awk is awesome and Dr. Kernighan has taught me so much. If you like exploratory data analysis using awk, you may like the "num" command: https://github.com/numcommand/num https://github.com/numcommand/num Num uses awk for command line statistics, such as standard deviation, kurtosis, quartiles, uniqueness, ordering, and more. Num runs on a very wide range of Unix systems, such as systems without package managers. Feature requests and PRs are welcome.
simonw 3y ago
Wow, imagine being a humanities major and having Brian Kernighan teach you Awk!
1vuio0pswjnm7 3y ago
Are others noticing the author. This is the co-creator of UNIX, co-author of The C Programming Language and the "K" in AWK, and more, teaching UNIX to non-CS majors. That's pretty cool.
dleeftink 3y ago
The mighty awk: a great tool, but not the one I start with teaching people interested in applying computational methods. What I notice is that it moves 'too quick' on prompt execution--great for sysadmins, developers and on the job data analysts, but a 'slower' tool is often easier for newcomers to get their heads around, who'd often like to see what happens between steps as well as natural language error messages when something goes wrong. KNIME, Orange and GSheets + Apps Script fill this niche to some extent (and I've been wanting to give ENSO Lang a try), enabling rapid prototyping, iteration and visual clues when steps fail. Then, through repeatedly running the same steps in these environments, it suddenly 'clicks' for some that many of the data preprocessing steps follow a similar pattern, and that much time can be saved using scripting. You have to tease out this eagerness somewhat, by having learners first run through the more arduous process of clicking, dragging, navigating and debugging through a 'slow' interface. In this regard, some have started to create workflows and test suites around software onboarding, which will yield some valuable insights into what pedagogical strategies work best for different computational skill levels [0][1][2]. [0]: https://jku-vds-lab.at/publications/2022_eurovis_dashboard_onboarding/ https://jku-vds-lab.at/publications/2022_eurovis_dashboard_o... [1]: https://jku-vds-lab.at/publications/2022_visinf_compare_evaluations_onboarding/ https://jku-vds-lab.at/publications/2022_visinf_compare_eval... [2]: https://ideah.pubpub.org/pub/yia9z29r/release/1 https://ideah.pubpub.org/pub/yia9z29r/release/1
wrp 3y ago
I teach text processing to linguistics students who usually have had zero programming experience. I start by introducing regular expressions in the environment of a text editor. The students take quickly to the declarative nature of regexes and become excited by the power of automation. Moving on to scripting in a language is hard. I would prefer awk, but I usually have to introduce R, because that is what "everyone" is using these days. When faced with the complexity of a full language, most students lose their enthusiasm and never touch programming again after their thesis. You can accomplish quite a lot with a powerful text editor and just regexes. I think it would be better for most students to just stick with that.
- nestorD 3y ago
  Nowadays I would introduce chatgpt as an intermediate layer to use the language. I did some experiments recently with chatgpt4.0 and R and, while not perfect, it did great being able to go from specification to code to interpretation of the results.
- 7952 3y ago
  Maybe ETL tools could be a good next step (like FME). You get really high quality input and output drivers. Functional tools for basic data tasks. The data can be easily explored at any stage. And you can drop in regexp or python where it is needed.
ZeroGravitas 3y ago
No mention of visidata yet? If you like vi style interfaces, or TUIs and have data to explore then check it out. (Native language for manipulating data is python) https://www.visidata.org/ https://www.visidata.org/
hcks 3y ago
Sorry but presenting awk as a serious alternative to pandas in 2023 to people who aren’t very computing savvy is just mischievous.
- jncfhnb 3y ago
  Yeah it’s just terrible advice considering what tooling peers will be using in the field. The author mentions being biased a lot but like… that’s a LOT of bias.

177 points by yarapavan 3y ago

yarapavan 3y ago

Course website (linked from the article): https://www.hum307.com/ https://www.hum307.com/

wslh 3y ago

Ah! That is awkard! Sorry, I couldn't resist, I have all respect for Awk.

smlavine 3y ago

If I were at Princeton, I would take every one of Kernighan's classes that I could! I wonder if that's a problem there.

EvanKelly 3y ago
I'm at this point 15 years removed, but Prof Kernighan was one of the most accessible professors and taught the most popular CS survey course (333). I have at least a half a dozen times where I was pointed his direction from another professor and Kernighan spent an hour with me looking into how to scrape a dynamic website for my auction theory project. When he was stumped he introduced me to a professor at another school who he knew had looked into the topic.

mbb70 3y ago

I'm all for this kind of exploratory hacking around before booting up python/R/Excel/duckdb, especially in constrained environments. A classic pain point is having to deal with column numbers, so I'll share my favorite trick: `head -n1 /path/to/file.csv | tr ',' '\n' | nl | grep desired_column` gives you the column number of desired_column

chaps 3y ago
grep -n also works in place of `nl`!
tejtm 3y ago
yep, without knowing about `nl` I used `...| grep -n column_header` or `...|grep -n .` to replicate the 'nl' behavior. edit: I like your 'nl' better as it is using white space instead of colon as a separator.
- bear8642 3y ago
  oh, didn't realise you could do that - I've tended to use `... |awk '{print $NF, $0}'` to add line numbers to something
patrec 3y ago
Unless there is a quoted comma or an empty column beforehand (nl “helpfully” skips empty lines for numbering purposes).
deleted 3y ago
[deleted]
i15e 3y ago
Something to watch out for with nl is that by default it doesn't number empty lines. e.g.: $ printf 'one\n\nthree\n' | nl 1 one 2 three Set -ba to enable numbering all lines. For this use case I usually end up running cat -n instead since I find it easier to remember.

qsort 3y ago

The article nails down a very real pain point with libraries like Pandas: > looping over a set of input lines seems more natural than the dataframe selectors that Pandas favors Row-oriented operations, as opposed to aggregations and other OLAP-style queries are kind of painful. The generator machinery (yield from) is a partial fix to this, but Pandas itself offers little relief.

esafak 3y ago
pandas has a poor API. I'd rather use SQL with DuckDB.
- slt2021 3y ago
  pandas is way more powerful that most people use it. when you have to deal with thousands of text files, mish mash of csv, tsv, some rows overlap between the files, some files spread across multiple different locations (shared drive, s3 bucket, URL, SQL db, etc), with column names that look similar but not exactly similar - this is perfect use case for pandas. read csv file? just pd.read_csv() read and concat N csv files? just pd.concat([pd.read_csv(f) for f in glob("*.csv")]) read parquet or read_sql()? not a problem at all. need to do some custom rules for data cleansing, or regex matching or fuzzy matching on column names, converting data from/to csv/parquet/sql - it will be pandas 1 liner a lot of painful data processing/cleaning, correcting data is just 1-liner in pandas, and I dont know of better tool that can beat pandas - probably tidyR but it is essentially same pandas just for R
  - esafak 3y ago
    You can do that with other tools too. https://duckdb.org/docs/data/csv/overview.html https://duckdb.org/docs/data/csv/overview.html https://duckdb.org/docs/data/parquet/overview https://duckdb.org/docs/data/parquet/overview https://duckdb.org/docs/data/multiple_files/overview.html https://duckdb.org/docs/data/multiple_files/overview.html
    - slt2021 3y ago

pmarreck 3y ago

Recent Awk convert (after, like most people, just using it for one-liners for years); it's aged remarkably well (although I wish it used more functional constructs, permitted proper variable initialization, and had interrupt handling... but at that point, it's probably best to switch to a "full" language...)

jph 3y ago

Awk is awesome and Dr. Kernighan has taught me so much. If you like exploratory data analysis using awk, you may like the "num" command: https://github.com/numcommand/num https://github.com/numcommand/num Num uses awk for command line statistics, such as standard deviation, kurtosis, quartiles, uniqueness, ordering, and more. Num runs on a very wide range of Unix systems, such as systems without package managers. Feature requests and PRs are welcome.

simonw 3y ago

Wow, imagine being a humanities major and having Brian Kernighan teach you Awk!

1vuio0pswjnm7 3y ago

Are others noticing the author. This is the co-creator of UNIX, co-author of The C Programming Language and the "K" in AWK, and more, teaching UNIX to non-CS majors. That's pretty cool.

dleeftink 3y ago

The mighty awk: a great tool, but not the one I start with teaching people interested in applying computational methods. What I notice is that it moves 'too quick' on prompt execution--great for sysadmins, developers and on the job data analysts, but a 'slower' tool is often easier for newcomers to get their heads around, who'd often like to see what happens between steps as well as natural language error messages when something goes wrong. KNIME, Orange and GSheets + Apps Script fill this niche to some extent (and I've been wanting to give ENSO Lang a try), enabling rapid prototyping, iteration and visual clues when steps fail. Then, through repeatedly running the same steps in these environments, it suddenly 'clicks' for some that many of the data preprocessing steps follow a similar pattern, and that much time can be saved using scripting. You have to tease out this eagerness somewhat, by having learners first run through the more arduous process of clicking, dragging, navigating and debugging through a 'slow' interface. In this regard, some have started to create workflows and test suites around software onboarding, which will yield some valuable insights into what pedagogical strategies work best for different computational skill levels [0][1][2]. [0]: https://jku-vds-lab.at/publications/2022_eurovis_dashboard_onboarding/ https://jku-vds-lab.at/publications/2022_eurovis_dashboard_o... [1]: https://jku-vds-lab.at/publications/2022_visinf_compare_evaluations_onboarding/ https://jku-vds-lab.at/publications/2022_visinf_compare_eval... [2]: https://ideah.pubpub.org/pub/yia9z29r/release/1 https://ideah.pubpub.org/pub/yia9z29r/release/1

wrp 3y ago

I teach text processing to linguistics students who usually have had zero programming experience. I start by introducing regular expressions in the environment of a text editor. The students take quickly to the declarative nature of regexes and become excited by the power of automation. Moving on to scripting in a language is hard. I would prefer awk, but I usually have to introduce R, because that is what "everyone" is using these days. When faced with the complexity of a full language, most students lose their enthusiasm and never touch programming again after their thesis. You can accomplish quite a lot with a powerful text editor and just regexes. I think it would be better for most students to just stick with that.

nestorD 3y ago
Nowadays I would introduce chatgpt as an intermediate layer to use the language. I did some experiments recently with chatgpt4.0 and R and, while not perfect, it did great being able to go from specification to code to interpretation of the results.
7952 3y ago
Maybe ETL tools could be a good next step (like FME). You get really high quality input and output drivers. Functional tools for basic data tasks. The data can be easily explored at any stage. And you can drop in regexp or python where it is needed.

ZeroGravitas 3y ago

No mention of visidata yet? If you like vi style interfaces, or TUIs and have data to explore then check it out. (Native language for manipulating data is python) https://www.visidata.org/ https://www.visidata.org/

hcks 3y ago

Sorry but presenting awk as a serious alternative to pandas in 2023 to people who aren’t very computing savvy is just mischievous.

jncfhnb 3y ago
Yeah it’s just terrible advice considering what tooling peers will be using in the field. The author mentions being biased a lot but like… that’s a LOT of bias.