3 ms·
Heres the .q file I use to read HN. Hobbyist programmer; corrections/improvements welcome. Better yet, how would I do this in J? cat 1.q \c 2000 200 k
by aplorbust 9y ago
Heres the .q file I use to read HN. Hobbyist programmer; corrections/improvements welcome. Better yet, how would I do this in J?
cat 1.q
\c 2000 200
k)-23!t:select from .:`:t.kdb
k).Q.fs[{`t insert +:`url`title`user`item!("SSSS";",")0:x }] `:hn.csv
k)-23!t:?:select from t
`:t.kdb set t
\\
As for how to get hn.csv, I use shell scripts to download all the pages of HN html (in accordance with robots.txt specified delay). Then I use a simple lexer (made with flex) to transform the html to hn.csv with the above selected columns.
After some years, the file hn.csv can grow quite large, larger than available memory.
To read and select articles from t.kdb, I use less(1) and some small shell scripts.
This lists all the articles, starting after a given offset, or zero by default:
echo -e '\\c 2000 2000\nk)select i,title from .:`:t.kdb where i>'${1-0}'\n' \
|exec q > 1;
exec less 1;
Then, to select and read an article or its comments I use the ":!cmd" feature in less(1) to invoke an appropriate shell script (actually, an execline script). Quit, and I am returned to less.
Have seen lots of systems for reading HN made by HN readers, using various databases I guess. This is mine, using kdb+.
- adrianN 9y agoNeat, but why don't you use lynx?
- 0b01 9y agoLet me guess... You work at a bulge bracket
- LfLxfxxLxfxx 9y agohow much did you pay for your kdb+?
- aplorbust 9y ago/After/s/hn.csv/t.kdb/