7 ms·

I've programmed in J professionally (admittedly not for all that long) as a data scientist and, coincidentally, have just completed a small analysis using J as

I've programmed in J professionally (admittedly not for all that long) as a data scientist and, coincidentally, have just completed a small analysis using J as part of an internal workshop about data analysis I am planning. I typically work in R and Python and I have to say that at this stage there is almost no reason I would pick up J to do any work. Unless code-golf level conciseness is your only goal, these other platforms offer superior performance, clarity, ease of use, access to libraries and are, as programming languages, substantially better designed. I say this as a great lover of function-level programming and as a J enthusiast. I would say I am quite familiar with J's programming paradigm and conceptual widgets and doodads (I know the verbs, nouns, adverbs and conjunctions and can use them appropriately). I even remembered a pretty good portion of the Nuvoc. But doing even the simplest analysis in J was _excrutiatingly_ slow and inconvenient compared to using R and the tidyverse (in particular, I missed dplyr and ggplot). The tidyverse CSV readers are, for example, much faster and smarter and more convenient and informative than anything you'll get from the J universe. I love vector languages but at this point J can't compete with the major platforms for data analysis. Its less convenient, often _slower_, much more low level, strange, and its library situation is anemic at best. I recommend learning J because it will expand your mind, but I can't imagine picking it up for real work.

VHRanger 5y ago
The fact there are vector languages in subsets of python (numpy, pandas, etc.) and R. And these already have great large columnar dataset support (eg. Apache Arrow) And an open source community intent on developing and maintaining the ecosystem.
- nathan_compton 5y ago
  One of the nicest thing about J is the notion of verb rank. For non-J-programmers, you can apply a rank to a verb and this effects how the verb operates on its vector operands. A rank of zero means "operate on the entire object" whereas a rank of 1 means "operate on the (1) elements of the operands. Other ranks change the meaning of what counts as "an element." However, like most things in J, support for this excellent idea (which eliminates the need for most looping constructs and can be very performant) is irregular: it is limited to monadic and dyadic verbs. Nothing about verb rank forbids functions which accept more than two arguments, but the idea of a function which accepts more than 2 arguments is poorly supported in J (the idiom is to pass a boxed array to a monad, but the boxing of the items to be passed makes supporting rank behavior for the "arguments" impossible or absurdly complicated. Other beefs with J: J doesn't have first class functions as such. While you can represent functions as "nouns" in a few ways, you cannot have (for example) an anonymous reference to a function as a thing unto itself (you may denote a verb tacitly in a context where you need a verb, however, but this is not the same thing). If you want to pass around verbs in a way familiar to you as a contemporary programmer you have to use "adverbs" and "conjunctions" which are just higher order functions which (more or less) return verbs. But adverbs and conjunctions have their own peculiarities and restrictions (not the least of which is that they are not themselves verbs or nouns and thus cannot be passed around either). In contemporary programming languages the verb/adverb/conjunction space would just be represented by "functions" and to great effect. As a functional programmer and Lisp guy, I find the limitations on "verbs" very frustrating in J. J's error messages are also bad, never more than a few words. There are some great ideas in the language, but it feels very old-fashioned and out of touch. What I would like to see is a "array scheme." A lexically scoped Scheme-like language where every object is an array and function argument slots can be independently "ranked" to support the elimination of loops over array arguments. I'm too busy to put this together, but it would be great to have if you wanted to fiddle with arrays for some reason but could do without any library support for actually doing data analysis.
  - bakul 5y ago
    > What I would like to see is a "array scheme." Me too! I even started on this a few years ago but then got distracted!
  - moonchild 5y ago
    > What I would like to see is a "array scheme." There was some academic project which did something similar.
recuter 5y ago
Thank you for this. What do you make of BQN? https://aplwiki.com/wiki/BQN https://aplwiki.com/wiki/BQN I get enamored with apl/k/j every time I see it and was looking for excuses to use it despite everything. I understand that due to the much smaller community the tooling and ecosystem is much weaker but there must be a reason why some people keep reaching for it, especially the guys in finance. I don't get the Cobol vibes from it like it is some sort of legacy burden. While the use case is narrow there must be an edge. This is HN after all. You wouldn't tell people not to mess with lisp and just reach for python now would you? *puppy eyes stare*
- all2 5y ago
  When I want to just "Get stuff done" TM, I reach for Python. Except that I've stopped doing that because setting up package versioning and venvs is a nightmare that gets more frustrating every time I try to do it. Now I'm looking for a "better" TM way to get my scripting needs met. I'm looking at Nim, specifically. I may also try to lean on a Scheme or a Lisp. My problem with the latter is lack of decent docs for getting stuff done. Maybe I'm missing something, but being productive in those languages for me is like a high jump when I can't even step up on a curb.
  - rscho 5y ago
    Racket has been rated as 'an acceptable python' by a famous programmer. Well deserved, I think.
    - klibertp 5y ago
      Not without quite a few libraries. You need at least rackjure[1] (for threading macros if nothing else) and data/collections[2] (for unified sequence/collection handling and convenient helpers). You might want data/functional[3] for the `do` notation (to replace all the for/... forms; only if you know what you're doing). You will probably need a few srfis - 26 for cut, for example - depending on your preferred coding style and domain. You won't need external libraries for streams, generators, objects and classes, contracts, graphics and plotting, documentation generation, concurrency, limited parallelism (futures) and multiprocessing (places) - as these are all built-in. Racket is a great language. I learned it around 2008 and get back to it regularly. However, due to its focus on pedagogy, the defaults tend to be verbose for practical work. The incredible power of `syntax-parse` makes it trivial to build DSLs, but that backfires, because everyone and their cat make DSLs instead of settling on a single set of helper libraries. Packages that require a lot of effort to make, like a numpy/pandas/data-frame implementation, tend to lag behind their counterparts in other languages due to the relatively small community. Without easy access to larger ecosystem that Clojure enjoys it's pretty much impossible for the situation to drastically improve on that front. Contracts and Typed Racket are still, to my knowledge, the most advanced in their respective fields. I love Racket. It's my first Lisp, and it'll have a place in my heart until the day it stops beating. However, for practical, day to day coding, that doesn't fit precisely into Racket strengths, it's not as good as any of the top 10 on TIOBE. [1] https://docs.racket-lang.org/rackjure/index.html https://docs.racket-lang.org/rackjure/index.html [2] https://docs.racket-lang.org/collections/collections-api.html?q=data%2Fcollection#%28part._.Generic_.Collections_and_.Sequences%29 https://docs.racket-lang.org/collections/collections-api.htm... [3] https://docs.racket-lang.org/functional/interfaces.html https://docs.racket-lang.org/functional/interfaces.html
moonchild 5y ago
The ecosystem problems are genuine. Though I do not think they are so great as you make them out to be. But with respect to semantics, numpy et al are but pale imitations. With respect to syntax, too (https://www.jsoftware.com/papers/tot.htm https://www.jsoftware.com/papers/tot.htm).
- nathan_compton 5y ago
  I sort of agree with you, especially about numpy. Nothing in the data science space in Python feels right to me. But you can't beat the network effects. Its still easier to actually do data analysis in Python than in J.
- user3939382 5y ago
  > I do not think they are so great as you make them out to be There's a dynamic with ecosystem problems I believe applies to all languages. You only need one missing or bad library that's critical to your project to make the whole language useless. An anecdotal example: I remember many years ago trying to give Python a go and within 15 minutes ran into a problem parsing XML. A search revealed this was a known issue that was being worked on with the foremost tool in Python for this job. You couldn't have credibly argued that Python had an ecosystem problem even at the time, but for me in that particular scenario Python had a show-stopping ecosystem problem. There were ways around this, but the most convenient way around it at the time was switching back to a more familiar language. My greater point is that, we can definitely make generalizations about a language's ecosystem health, but keep in mind there is a very context-sensitive, practical dimension to that type of language assessment.
  - moonchild 5y ago
    > You only need one missing or bad library that's critical to your project to make the whole language useless ...no? If there is functionality I need, and no library implements it, I will implement it myself. That goes for any language. Otherwise, the job of a programmer would simply be to string together existing libraries, not writing anything meaningful.
0des 5y ago
> substantially better designed Hey Siri, please remove Nathan_Compton from the Christmas card list.
beagle3 5y ago
I haven’t used R recently (10 years or so), but when I did, the speed with which K/kdb+ could scan through and summarize terabytes of data was orders of magnitude faster than R or any other system. Once the data was summarized into (say) a gigabyte or so, analyzing it with R or even Python was much easier thanks to the ecosystem and reasonable time (probably 10-100 times slower, but the time saved by using well tested stat code is more than worth it)
agumonkey 5y ago
How do you feel about the J/APL syntax in live coding sessions ? does it help iterating a bit faster than R/python ? or was it a totally irrelevant aspect ?

VHRanger 5y ago

The fact there are vector languages in subsets of python (numpy, pandas, etc.) and R. And these already have great large columnar dataset support (eg. Apache Arrow) And an open source community intent on developing and maintaining the ecosystem.

nathan_compton 5y ago
One of the nicest thing about J is the notion of verb rank. For non-J-programmers, you can apply a rank to a verb and this effects how the verb operates on its vector operands. A rank of zero means "operate on the entire object" whereas a rank of 1 means "operate on the (1) elements of the operands. Other ranks change the meaning of what counts as "an element." However, like most things in J, support for this excellent idea (which eliminates the need for most looping constructs and can be very performant) is irregular: it is limited to monadic and dyadic verbs. Nothing about verb rank forbids functions which accept more than two arguments, but the idea of a function which accepts more than 2 arguments is poorly supported in J (the idiom is to pass a boxed array to a monad, but the boxing of the items to be passed makes supporting rank behavior for the "arguments" impossible or absurdly complicated. Other beefs with J: J doesn't have first class functions as such. While you can represent functions as "nouns" in a few ways, you cannot have (for example) an anonymous reference to a function as a thing unto itself (you may denote a verb tacitly in a context where you need a verb, however, but this is not the same thing). If you want to pass around verbs in a way familiar to you as a contemporary programmer you have to use "adverbs" and "conjunctions" which are just higher order functions which (more or less) return verbs. But adverbs and conjunctions have their own peculiarities and restrictions (not the least of which is that they are not themselves verbs or nouns and thus cannot be passed around either). In contemporary programming languages the verb/adverb/conjunction space would just be represented by "functions" and to great effect. As a functional programmer and Lisp guy, I find the limitations on "verbs" very frustrating in J. J's error messages are also bad, never more than a few words. There are some great ideas in the language, but it feels very old-fashioned and out of touch. What I would like to see is a "array scheme." A lexically scoped Scheme-like language where every object is an array and function argument slots can be independently "ranked" to support the elimination of loops over array arguments. I'm too busy to put this together, but it would be great to have if you wanted to fiddle with arrays for some reason but could do without any library support for actually doing data analysis.
- bakul 5y ago
  > What I would like to see is a "array scheme." Me too! I even started on this a few years ago but then got distracted!
- moonchild 5y ago
  > What I would like to see is a "array scheme." There was some academic project which did something similar.

recuter 5y ago

Thank you for this. What do you make of BQN? https://aplwiki.com/wiki/BQN https://aplwiki.com/wiki/BQN I get enamored with apl/k/j every time I see it and was looking for excuses to use it despite everything. I understand that due to the much smaller community the tooling and ecosystem is much weaker but there must be a reason why some people keep reaching for it, especially the guys in finance. I don't get the Cobol vibes from it like it is some sort of legacy burden. While the use case is narrow there must be an edge. This is HN after all. You wouldn't tell people not to mess with lisp and just reach for python now would you? *puppy eyes stare*

all2 5y ago
When I want to just "Get stuff done" TM, I reach for Python. Except that I've stopped doing that because setting up package versioning and venvs is a nightmare that gets more frustrating every time I try to do it. Now I'm looking for a "better" TM way to get my scripting needs met. I'm looking at Nim, specifically. I may also try to lean on a Scheme or a Lisp. My problem with the latter is lack of decent docs for getting stuff done. Maybe I'm missing something, but being productive in those languages for me is like a high jump when I can't even step up on a curb.
- rscho 5y ago
  Racket has been rated as 'an acceptable python' by a famous programmer. Well deserved, I think.
  - klibertp 5y ago
    Not without quite a few libraries. You need at least rackjure[1] (for threading macros if nothing else) and data/collections[2] (for unified sequence/collection handling and convenient helpers). You might want data/functional[3] for the `do` notation (to replace all the for/... forms; only if you know what you're doing). You will probably need a few srfis - 26 for cut, for example - depending on your preferred coding style and domain. You won't need external libraries for streams, generators, objects and classes, contracts, graphics and plotting, documentation generation, concurrency, limited parallelism (futures) and multiprocessing (places) - as these are all built-in. Racket is a great language. I learned it around 2008 and get back to it regularly. However, due to its focus on pedagogy, the defaults tend to be verbose for practical work. The incredible power of `syntax-parse` makes it trivial to build DSLs, but that backfires, because everyone and their cat make DSLs instead of settling on a single set of helper libraries. Packages that require a lot of effort to make, like a numpy/pandas/data-frame implementation, tend to lag behind their counterparts in other languages due to the relatively small community. Without easy access to larger ecosystem that Clojure enjoys it's pretty much impossible for the situation to drastically improve on that front. Contracts and Typed Racket are still, to my knowledge, the most advanced in their respective fields. I love Racket. It's my first Lisp, and it'll have a place in my heart until the day it stops beating. However, for practical, day to day coding, that doesn't fit precisely into Racket strengths, it's not as good as any of the top 10 on TIOBE. [1] https://docs.racket-lang.org/rackjure/index.html https://docs.racket-lang.org/rackjure/index.html [2] https://docs.racket-lang.org/collections/collections-api.html?q=data%2Fcollection#%28part._.Generic_.Collections_and_.Sequences%29 https://docs.racket-lang.org/collections/collections-api.htm... [3] https://docs.racket-lang.org/functional/interfaces.html https://docs.racket-lang.org/functional/interfaces.html

moonchild 5y ago

The ecosystem problems are genuine. Though I do not think they are so great as you make them out to be. But with respect to semantics, numpy et al are but pale imitations. With respect to syntax, too (https://www.jsoftware.com/papers/tot.htm https://www.jsoftware.com/papers/tot.htm).

nathan_compton 5y ago
I sort of agree with you, especially about numpy. Nothing in the data science space in Python feels right to me. But you can't beat the network effects. Its still easier to actually do data analysis in Python than in J.
user3939382 5y ago
> I do not think they are so great as you make them out to be There's a dynamic with ecosystem problems I believe applies to all languages. You only need one missing or bad library that's critical to your project to make the whole language useless. An anecdotal example: I remember many years ago trying to give Python a go and within 15 minutes ran into a problem parsing XML. A search revealed this was a known issue that was being worked on with the foremost tool in Python for this job. You couldn't have credibly argued that Python had an ecosystem problem even at the time, but for me in that particular scenario Python had a show-stopping ecosystem problem. There were ways around this, but the most convenient way around it at the time was switching back to a more familiar language. My greater point is that, we can definitely make generalizations about a language's ecosystem health, but keep in mind there is a very context-sensitive, practical dimension to that type of language assessment.
- moonchild 5y ago
  > You only need one missing or bad library that's critical to your project to make the whole language useless ...no? If there is functionality I need, and no library implements it, I will implement it myself. That goes for any language. Otherwise, the job of a programmer would simply be to string together existing libraries, not writing anything meaningful.

0des 5y ago

> substantially better designed Hey Siri, please remove Nathan_Compton from the Christmas card list.

beagle3 5y ago

I haven’t used R recently (10 years or so), but when I did, the speed with which K/kdb+ could scan through and summarize terabytes of data was orders of magnitude faster than R or any other system. Once the data was summarized into (say) a gigabyte or so, analyzing it with R or even Python was much easier thanks to the ecosystem and reasonable time (probably 10-100 times slower, but the time saved by using well tested stat code is more than worth it)

agumonkey 5y ago

How do you feel about the J/APL syntax in live coding sessions ? does it help iterating a bit faster than R/python ? or was it a totally irrelevant aspect ?