6 ms·

Slightly unrelated: I often find myself in the browser console, exploring data scraped off webpages (often from Wikipedia tables). I think REPLs are another kin

by kaoD 10y ago

Slightly unrelated: I often find myself in the browser console, exploring data scraped off webpages (often from Wikipedia tables). I think REPLs are another kind of "tool of thought". E.g.: I recently scraped my country's election results off Wikipedia to see how voting areas affected the distribution of seats. Very insightful. Being tightly integrated to the browser offers the whole web as a data set. Unfortunately JavaScript does not lend itself very well to data analysis. What is your tool of choice for interactive data exploration? I've been looking for something similar for my phone, a sort of calculator-on-steroids even if not web-scraping ready. J for Android[0] fits the bill perfectly... except the language itself seems to get in the way with its short mnemonics, unrememberable verb train rules, function composition words and other weirdness in the construction of tacit phrases. I've (partially) learned and forgotten J three times already. Every single time I come back is like starting anew. I tried Clojure too but the Android REPL[1] is stuck on Clojure 1.4.0 and the lack of an S-expression enabled keyboard/editor makes it a pain to code. Any recommendations in this area? [0] http://www.jsoftware.com/ http://www.jsoftware.com/ [1] https://play.google.com/store/apps/details?id=com.sattvik.clojure_repl&hl=en https://play.google.com/store/apps/details?id=com.sattvik.cl...

RodgerTheGreat 10y ago
It may not really be what you want, but I've been developing a mobile-friendly frontend for my K interpreter which makes it essentially a K-based graphing calculator: http://johnearnest.github.io/ok/mobile.html http://johnearnest.github.io/ok/mobile.html The browser security model severely limits potential for doing web scraping with it, but I have found it pleasant to use so far. Docs can be found here: https://github.com/JohnEarnest/ok#mobile https://github.com/JohnEarnest/ok#mobile
dTal 10y ago
It's not Free Software, but Frink is my goto "calculator on steroids". Not sure if it's suited to what you have in mind. https://frinklang.org/ https://frinklang.org/
carussell 10y ago
Side note: > I often find myself in the browser console, exploring data scraped off webpages (often from Wikipedia tables) Are you aware of Wikidata? I don't have a lot of familiarity with it, only that some (a lot?) of the programmer types who were drawn to Wikipedia in its infancy have shifted their focus to Wikidata nowadays. As I understand it, Wikidata is meant to be the data source behind those tables that you're scraping. (Forgive me if you're already involved with Wikidata and one of your primary uses for scraping is in pursuit of migrating content to Wikidata.)
qwertyuiop924 10y ago
My goto tool for data analysis is AWK: it's not interactive, but it's succinct, and very good at handling DSV data. As for handling non DSV data... I dunno. Lisp is pretty nice for this, but it's a pain to get the data into the system. TXR and its lisp dialect are pretty good with this, so that's an option. But if you really want to handle tabular data interactively, doing manipulation and analysis on the fly... that's quite literally what Excel was designed for.
- kazinator 10y ago
  Speaking of Awk and TXR, TXR Lisp has an awk macro (introduced in mid 2016), with which you can transliterate many kinds of Awk programs. However, it's a lot slower than GNU Awk. You don't get the duck typing of Awk (whereby strings that look like numbers can be treated arithmetically, and nonexistent or blank variables are zero, et cetera). To convert fields to numbers with reasonable succinctness, an "awk local macro" called fconv (field convert) is provided. There is no function-for-function compatibility with the Awk or GNU Awk library, and regular expressions are TXR: that means no register capture, and ^ and $ done with functional combinators.
  - qwertyuiop924 10y ago
    ...Does it by any chance resemble the SCSH awk macro?
    - kazinator 10y ago
      In some superficial ways it does, like the basic clause structure. "Prior art" I was well aware of is cl-awk. This awk macro was intended to be a detailed implementation of the Awk paradigm (with some GNU extensions), complete with obscure features like assigning to the nf variable to change the number of fields (extending with empty strings if the new count is larger), and a working regex-based record separation via the rs variable. I see Shrivers implemented something that's on my TODO list: variants of ranges that exclude the start, end or both. However, he didn't lift the restriction that Awk ranges do not combine with other conditions, including ranges.

by kaoD 10y ago

RodgerTheGreat 10y ago

It may not really be what you want, but I've been developing a mobile-friendly frontend for my K interpreter which makes it essentially a K-based graphing calculator: http://johnearnest.github.io/ok/mobile.html http://johnearnest.github.io/ok/mobile.html The browser security model severely limits potential for doing web scraping with it, but I have found it pleasant to use so far. Docs can be found here: https://github.com/JohnEarnest/ok#mobile https://github.com/JohnEarnest/ok#mobile

dTal 10y ago

It's not Free Software, but Frink is my goto "calculator on steroids". Not sure if it's suited to what you have in mind. https://frinklang.org/ https://frinklang.org/

carussell 10y ago

Side note: > I often find myself in the browser console, exploring data scraped off webpages (often from Wikipedia tables) Are you aware of Wikidata? I don't have a lot of familiarity with it, only that some (a lot?) of the programmer types who were drawn to Wikipedia in its infancy have shifted their focus to Wikidata nowadays. As I understand it, Wikidata is meant to be the data source behind those tables that you're scraping. (Forgive me if you're already involved with Wikidata and one of your primary uses for scraping is in pursuit of migrating content to Wikidata.)

qwertyuiop924 10y ago

My goto tool for data analysis is AWK: it's not interactive, but it's succinct, and very good at handling DSV data. As for handling non DSV data... I dunno. Lisp is pretty nice for this, but it's a pain to get the data into the system. TXR and its lisp dialect are pretty good with this, so that's an option. But if you really want to handle tabular data interactively, doing manipulation and analysis on the fly... that's quite literally what Excel was designed for.

kazinator 10y ago
Speaking of Awk and TXR, TXR Lisp has an awk macro (introduced in mid 2016), with which you can transliterate many kinds of Awk programs. However, it's a lot slower than GNU Awk. You don't get the duck typing of Awk (whereby strings that look like numbers can be treated arithmetically, and nonexistent or blank variables are zero, et cetera). To convert fields to numbers with reasonable succinctness, an "awk local macro" called fconv (field convert) is provided. There is no function-for-function compatibility with the Awk or GNU Awk library, and regular expressions are TXR: that means no register capture, and ^ and $ done with functional combinators.
- qwertyuiop924 10y ago
  ...Does it by any chance resemble the SCSH awk macro?
  - kazinator 10y ago
    In some superficial ways it does, like the basic clause structure. "Prior art" I was well aware of is cl-awk. This awk macro was intended to be a detailed implementation of the Awk paradigm (with some GNU extensions), complete with obscure features like assigning to the nf variable to change the number of fields (extending with empty strings if the new count is larger), and a working regex-based record separation via the rs variable. I see Shrivers implemented something that's on my TODO list: variants of ranges that exclude the start, end or both. However, he didn't lift the restriction that Awk ranges do not combine with other conditions, including ranges.