5 ms·
How expensive is SF rent? Use Unix commands to find out
How expensive is SF rent? Here's a coding challenge that uses Unix command line tools to find out.
Hacker friends suggested I post the result and code here because I'm writing open source statistics software and this is a practical example of results and code.
The results for a one bedroom in SF: mean $3348, median $3475, interquartile range $1514, and standard deviation $1130.
How to use Unix commands to find out: use `curl` to fetch Craigslist apartment listings, then `sed`, `grep`, and `tr` to filter and parse the data, then `num` to calculate the statistics.
curl -s http://sfbay.craigslist.org/search/sfc/apa\?min_price\=1\&bedrooms\=1 |
sed 's/\(<p class="row"\)/\n\1/g' |
grep ' 1br ' |
tr ">" "\n" |
sed -n 's/^\$\([0-9]\+\).*/\1/p' |
num mean median interquartile-range standard-deviation OFMT="%.0d"
The software I'm personally writing is the `num` command to calculate the statistics. The software is free at http://www.numcommand.com
- DrScump 11y agoWhat happens if the price overflows the limit of a 32-bit integer?
- jph 11y agoIf the price overflows 32-bit, then everyone in SF moves. :) For this script, the fetch returns at most 100 rows, so won't overflow. More broadly, the Num command runs using awk, and number handling depends on your system's awk implementation and C libraries. Current awk implementations use an internal representation of all numbers, including integers, that uses double-precision floating-point numbers. I know GNU awk supports arbitrary precision arithmetic, and also can use --bignum, GNU MPFR and GNU MP (GMP); these will be great additions to the Num command for the next major version upgrade.
- deleted 11y ago[deleted]
- gesman 11y ago@echo "very expensive"
- romanovcode 11y agosed: RE error: illegal byte sequence gawk: cmd. line:1691: fatal: division by zero attempted
- jph 11y agoCan you contact me about this? I'll diagnose it. My direct email is joel@joelparkerhenderson.com. I really appreciate it.
- anon6_ 11y agoFreeBSD 11-CURRENT num: command not found tr: Illegal byte sequence
- jph 11y agoTo install the num command, use http://www.numcommand.com http://www.numcommand.com
- atsaloli 11y agoI had a perl script to do mean, median and std dev; it'd be great to have this functionality available through the OS vendor as a package.