3 ms·

Have any data scientists here who were primarily working in R or Python, learned J? What did you think? I love programming languages that teach you new ways of

by kyllo 8y ago

Have any data scientists here who were primarily working in R or Python, learned J? What did you think? I love programming languages that teach you new ways of thinking about programming and I'm very curious about the APL family--thinking of learning J or q/kdb+. I do most of my work in query languages on column-store databases, then some visualization and hypothesis testing on top. So I'm wondering if I could find J more expressive than R/Python + some SQL-like language.

4thaccount 8y ago
I use Python a bunch and have used R in the past. I've dabbled in Julia and both APL & J. I know just enough to say that APL & J are much more expressive as long as Python doesn't have a fix_problem() method built-in. I believe ScottLocklin says something similar below, but he has a lot more experience in R and J than I. Now q/kdb+ might be expensive, but I can only dream of doing analysis using it. SQL is great, but having the full power of K And Q-SQL would be pretty awesome.
- kyllo 8y ago
  Yeah it really seems to me like a highly expressive data analysis language with a tightly integrated column-store database system could have saved many, many millions of dollars and person-hours compared to Hadoop and Spark.
- kd0amg 8y ago
  I know just enough to say that APL & J are much more expressive as long as Python doesn't have a fix_problem() method built-in. This is pretty much the same impression I got, though I came from the other direction. I was kind of surprised how many StackOverflow posts ask about adapting a fix_problem() method to fix some very_similar_problem.
  - 4thaccount 8y ago
    Yep. It's one reason why I prefer to write my own code where possible instead of using 3rd party libraries. When you need to change something, it is nice to not read tons of other code.
alexeiz 8y ago
I don't know about J, but on occasion I have to deal with q/kdb+ and I hate it with passion. That system is pure crap. I'm convinced that the primary purpose of q/kdb+ is to extract as much money as possible on support from financial firms that happen to use it. Support contracts must run in millions of dollars if not more. The system is designed to be incomprehensible by human beings. All in a name of "performance." But is it really any faster than, say, Pandas/Numpy? (Numpy can automatically parallelize computations, whereas q/kdb is inherently single-threaded.) Q error messages drive me crazy. Basically, there are just a few error messages, like "'type" or "'rank" with no further explanation what caused an error. The Q language is full of idiosyncrasies you have to remember. I code in multiple languages including functional ones, but Q is completely illogical. There are arbitrary restrictions like number of local variables or the precise place you put a closing brace that ends a function. It's also impossible to find any help for it due to its abuse of special symbols, incomprehensible function names, and its very low popularity. There are C and Python interfaces for Q/kdb and they suck too. The C interface is actually on Github; you only need to take a single look at it to understand what I mean.

by kyllo 8y ago

4thaccount 8y ago

I use Python a bunch and have used R in the past. I've dabbled in Julia and both APL & J. I know just enough to say that APL & J are much more expressive as long as Python doesn't have a fix_problem() method built-in. I believe ScottLocklin says something similar below, but he has a lot more experience in R and J than I. Now q/kdb+ might be expensive, but I can only dream of doing analysis using it. SQL is great, but having the full power of K And Q-SQL would be pretty awesome.

kyllo 8y ago
Yeah it really seems to me like a highly expressive data analysis language with a tightly integrated column-store database system could have saved many, many millions of dollars and person-hours compared to Hadoop and Spark.
kd0amg 8y ago
I know just enough to say that APL & J are much more expressive as long as Python doesn't have a fix_problem() method built-in. This is pretty much the same impression I got, though I came from the other direction. I was kind of surprised how many StackOverflow posts ask about adapting a fix_problem() method to fix some very_similar_problem.
- 4thaccount 8y ago
  Yep. It's one reason why I prefer to write my own code where possible instead of using 3rd party libraries. When you need to change something, it is nice to not read tons of other code.

alexeiz 8y ago

I don't know about J, but on occasion I have to deal with q/kdb+ and I hate it with passion. That system is pure crap. I'm convinced that the primary purpose of q/kdb+ is to extract as much money as possible on support from financial firms that happen to use it. Support contracts must run in millions of dollars if not more. The system is designed to be incomprehensible by human beings. All in a name of "performance." But is it really any faster than, say, Pandas/Numpy? (Numpy can automatically parallelize computations, whereas q/kdb is inherently single-threaded.) Q error messages drive me crazy. Basically, there are just a few error messages, like "'type" or "'rank" with no further explanation what caused an error. The Q language is full of idiosyncrasies you have to remember. I code in multiple languages including functional ones, but Q is completely illogical. There are arbitrary restrictions like number of local variables or the precise place you put a closing brace that ends a function. It's also impossible to find any help for it due to its abuse of special symbols, incomprehensible function names, and its very low popularity. There are C and Python interfaces for Q/kdb and they suck too. The C interface is actually on Github; you only need to take a single look at it to understand what I mean.