15 ms·

J Notation as a Tool of Thought

160 points by janvdberg 6y ago

ipsum2 6y ago
Interesting blog post. All of these are implemented as trivial operations in numpy, so while J or APL might be the originator of the syntax, this 'tool of thought' is not as esoteric as the author makes it sound, and learning J if you already know numpy (or TensorFlow or PyTorch or..) isn't that remarkable. >> np.array([1, 2, 3]) * 2 array([2, 4, 6]) >>> np.array([1, 2, 3]) * np.array([4, 5, 6]) >>> np.arange(0, 16).reshape(4, 4) + np.array([5, 5, 5, 5]) array([[ 5, 6, 7, 8], [ 9, 10, 11, 12], [13, 14, 15, 16], [17, 18, 19, 20]])
- moonchild 6y ago
  You've reproduced the most trivial example, which many languages make easy. I would be very interested to see another language with a rank operator, for instance. ------------------------------------------------------------------------ Challenge for you: rewrite a nontrivial program in one of those frameworks, with the following restrictions: - No iteration (including implicit iterations—map, filter; reduce is ok) - No loops whatsoever. Recursion is ok, but should be avoided wherever possible. - No explicitly named arguments; everything in pointfree style. (I know, map/filter can be implemented recursively. But compared with the equivalent constructs in apl, they're about 10× as verbose, and harder to reason about and understand. Even reduce is somewhat of a gimme.)
  - ggrrhh_ta 6y ago
    I have want to chime in. Agree, it is great that ideas from APL have made it into the language... But there is, fortunately, a long and fruitful path still to explore... I love that J is like an alternative path to functional approaches like Haskell and how it plays in terms of providing expressive power. Still, I will not create code in J that is maintained jointly with other users from other domains (which is the case usually when you are using numpy, for example)
    - TeMPOraL 6y ago
      I like how APL made it into Common Lisp wholesale. There's a library that compiles APL to Common Lisp, allowing you to mix APL and Lisp code, working on the same data structures with both languages. https://github.com/phantomics/april https://github.com/phantomics/april Example session I just did, copying APL code from https://aplwiki.com/wiki/John_Scholes%27_Conway%27s_Game_of_Life https://aplwiki.com/wiki/John_Scholes%27_Conway%27s_Game_of_...: CL-USER> (april:with-april-context ((:space *lifespace*)) (april:april "Show←{'·⌺'[⎕IO+⍵]}") (april:april "glider←3 3⍴1 1 1 1 0 0 0 1 0") (april:april-f "Show glider") (print (april:april "glider")) ; to prove it's a Lisp array (terpri) (april:april "grid←¯10 ¯10↑glider") (april:april-f "Show grid") (terpri) (april:april "life ← {⊃1 ⍵ ∨.∧ 3 4 = +/ +⌿ ¯1 0 1 ∘.⊖ ¯1 0 1 ⌽¨ ⊂⍵}") (terpri) (april:april-f "Show¨{life⍣⍵⊢grid}¨0,⍳3") (terpri)) ⌺⌺⌺ ⌺·· ·⌺· #2A((1 1 1) (1 0 0) (0 1 0)) ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·······⌺⌺⌺ ·······⌺·· ········⌺· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ········⌺· ·······⌺⌺· ·······⌺⌺· ·······⌺⌺⌺ ·······⌺⌺· ·······⌺·⌺ ······⌺⌺·· ·······⌺·· ·······⌺·⌺ ·······⌺·· ········⌺· ········⌺· ·········· ·········· ·········· (APL being compiled to Lisp is important, not just because it gives you proper interoperability out of the box, but also because on implementations like SBCL this means APL code will get compiled, through Lisp, to native code. And you can do that at runtime too.) Now I wish I could find something that does the same for J, because I'm sure as hell not going to buy an APL-friendly keyboard.
      - icen 6y ago
        Emacs has an excellent APL keyboard implementation (via gnu-apl-mode) which uses . as a prefix for the apl keys.
      - moonchild 6y ago
        > APL being compiled to Lisp is important, not just because it gives you proper interoperability out of the box, but also because on implementations like SBCL this means APL code will get compiled, through Lisp, to native code Except that simd in lisp isn't great and needs to be done mostly by hand[1]; and simd is super important for making apl performant. Also because gc attributes that are good for lisp aren't great for apl. Also because compiling to native code doesn't actually really affect performance of apl, and might actually harm it for larger programs. To be clear, april is a super-cool project, but not because of any of its performance characteristics. > Now I wish I could find something that does the same for J, because I'm sure as hell not going to buy an APL-friendly keyboard. You don't need a special keyboard, just a layout. I don't know how it works on windows (there's an 'IME', I think?) or macos, but x comes with a keyboard layout. I use this: setxkbmap -layout us,apl -variant dyalog -option grp:rwin_switch Which makes it so that holding down the right super key and pressing any key produces an appropriate apl symbol. You may want to use a different key to switch; check 'man xkeyboard-config' and then search for 'Switching to another layout' for other options. Try: modifier key (whatever it is)+r, for ⍴ (rho). 1. https://www.reddit.com/r/lisp/comments/fkfgjn/sbcl_with_simd_how_to_optimize_sseavx2_to_pointer/ https://www.reddit.com/r/lisp/comments/fkfgjn/sbcl_with_simd...
  - SuperCuber 6y ago
    Rank Is actually implicitly done by numpy using a mechanism called broadcasting. For example: >>> np.array([10, 20, 30]) + np.array([[1,2,3], [4,5,6], [7,8,9]]) array([[11, 22, 33], [14, 25, 36], [17, 28, 39]]) Sieves exist in numpy, called masks: >>>np.array([10, 20, 30]) > 15 array([False, True, True]) Of course they can be operated on just like any other numpy array. Grades exist in numpy: >>>np.array([5,4,3,2,1]).argsort() array([4, 3, 2, 1, 0]) Of course they are a little more verbose since all of those operations are from the library and not native to python.
    - ipsum2 6y ago
      To do sieves in J like the blog post mentioned, the equivalent numpy would be: >>> arr = np.array([10, 20, 30]) >>> arr[arr > 15] array([20, 30])
    - moonchild 6y ago
      > Rank Is actually implicitly done by numpy using a mechanism called broadcasting. Numpy's broadcasting is scalar conformability, to which the rank operator (and general conformability) provides a general case. Example in j: ] x =. i. 2 3 4 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 ] y =. i. 3 4 0 1 2 3 4 5 6 7 8 9 10 11 x + y NB. this will error because + expects that, if its arguments' shapes are not the same, one will be a prefix of the other |length error | x +y NB. this is easy enough to fix, however x +"2 y NB. +"2 is shorthand for +"2 2; meaning, choose rank-2 arrays from both the left and right arguments 0 2 4 6 8 10 12 14 16 18 20 22 12 14 16 18 20 22 24 26 28 30 32 34 Numpy will actually do this without the rank operator, because it uses suffix agreement rather than prefix agreement (which is absolutely bonkers—j used suffix agreement for about 5 minutes in 1990, before realising it was an awful idea). For for numpy, see if you can add: np.array([[[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]], [[12, 13, 14, 15], [16, 17, 18, 19], [20, 21, 22, 23]]]) + np.array([[0, 1, 2], [3, 4, 5]]) Intelligently. (I'm sure it's not overly difficult to come up with a solution, but can you do it with a single higher-order function call which generalises to other argument shapes?) (The j equivalent, (i. 2 3 4) + (i. 2 3) also works without trouble.) ------------------------------------------------------------------------ Another example, which may be more illustrative, is the ability to perform reductions along arbitrary axes. For example: ] x =. i. 4 3 0 1 2 3 4 5 6 7 8 9 10 11 +/ x NB. sum reduced along leading axis, the default, producing an array of shape 3 18 22 26 +/"1 x NB. sum each rank-1 array (vector); or, reduce last axis, producing an array of shape 4 3 12 21 30 ------------------------------------------------------------------------ Another curiosity, which I have thus far neglected, is the extent to which numpy's being 'a little more verbose' is actually incredibly important in shaping the way you approach and think about problems. The great-uncle comment also addresses this, but Iverson probably says it better than either of us can: read https://www.jsoftware.com/papers/tot.htm https://www.jsoftware.com/papers/tot.htm
- ggrrhh_ta 6y ago
  Exactly, one of the things I like about numpy is how it drew from APL and J. It is a more verbose language, and truth be told, J still has more expressive power (non regular processing of data, even state-machine-like processing) which can still influence numpy, or Julia.
- smabie 6y ago
  Comparing numpy to J is like comparing a dirty rag to a designer suit. The examples you've listed are trivial; the power of J and other array languages cannot be appreciated from afar. The philosophy behind array languages runs much deeper than adding two arrays or transposing matrices.
  - ipsum2 6y ago
    Can you give an example of something that can be done in J that can't be done easily in numpy?
    - sn41 6y ago
      It is not about "cannot be done". Most high-level languages are Turing complete, hence every program in one can be written as some program in another. The real clincher is expressiveness: shorter code is better, modular code is better, and languages which avoid unnecessary repetitiveness of boiler plate code is better. Different languages provide advantages along one or more of these parameters.
      - shawnz 6y ago
        So can you give an example of something that can be expressed simply in APL/J/K that can't be expressed simply with numpy? And to be clear I don't think "less characters" makes the expression more simple. Maybe "less statements" or "less operators"
        smabie 6y ago
        Here's Conway's Game of Life by Arthur Whitney (creator of K): life:{3=a-x*4=a:2{+(0+':x)+1_x,0}/x} I'm guessing a numpy implementation would be between one and two orders of magnitude more verbose, even if you're just comparing the number of operations.
        worc3131 6y ago
        It doesn't quite seem fair to compare numpy to one of the most famous K code golf's ever created, but here is my attempt, 140 characters compared to Arthur's 30, so 5 times longer. Not worry about padding the edges would cut it to 100. I don't think the gap is as wide as you think it is. def life(M): MP = np.pad(M,[(1,1),(1,1)]) N = sum(np.roll(MP,(i,j),(0,1)) for i in [1,0,-1] for j in [1,0,-1]) return (3==N-MP*(4==N))[1:-1,1:-1] Speedtest (presumably memory bound): 100 x 100 matrix, numpy 0.2ms, K 0.2ms 200 x 200 matrix, numpy 0.5ms, K 0.8ms 500 x 500 matrix, numpy 5.2ms, K 8.0ms 1000x1000 matrix, numpy 20.0ms, K 36.0ms 5000x5000 matrix, numpy 0.5s , K 1.2s Interestingly the K code is designed to return a boolean matrix, but operates much more slowly for me on a boolean matrix compared to an integer matrix, with the result that: q)\t klife M Takes 1.2 seconds whilst q)\t klife klife M Takes 24 seconds. So whilst the idea seems to be that klife is used with scan/over to run a number of iterations, it's actually a bad idea to run it more than once.
wodenokoto 6y ago
This reminded me very much of working in R or working with Numpy. I however always thought having an array as the primitive in R was great only because R is focused on statistics and data science. My understanding is that J claims to be general purpose programming and as such I’m surprised the paradigm holds.
- ggrrhh_ta 6y ago
  If you run the J GUI, written in J, it is going with blow your mind about how general purpose it is...
  - mst 6y ago
    I wish somebody would write a walkthrough of writing something like a piece of GUI to show that off. (I keep trying to figure out J and ... sliding off ... every time I need to do something "normal" ... which is annoying and the fact I'm reduced to wishcasting in HN comments as a solution makes me disappointed in both myself and the universe)
justusw 6y ago
In J, I think it's very difficult to move on from the initial stage of dealing with "line noise" as the author mentions. For me, this was further complicated by the fact that error messages are very hard to understand in the beginning. I've written an article that describes how I typically deal with solving most of the domain/rank/length errors: https://www.justus.pw/posts/2020-07-31-errors-in-j.html https://www.justus.pw/posts/2020-07-31-errors-in-j.html
tsimionescu 6y ago
This seems very interesting for numerical computation, but at first glance few of these operations seem to apply directly to general computation (I'm sure that there are isomorphisms that can be used to apply them indirectly). The example with sorting was the oddest from this point of view. The author praised the idea of the permutation vector as being a relatively direct mathematical specification of sorting (return the permutation of the original array in sorted order) while leaving out the 'sorted' part. Also, this focuses entirely on the result, but there is no word on the algorithm - 'how is the array sorted' is the question that computer science is designed to solve, as opposed to 'what are the fixed points of a sorted array'. As such, as someone with no experience of J, I'm left wondering if this is another example of the infamous Haskell quicksort - well-picked examples of code which produces a desired result are extraordinarily terse and expressive, but actually implementing well-known algorithms is often just as verbose as C, give or take.
- ggrrhh_ta 6y ago
  Yes, once one goes really deep, it's more like C in the end. One good thing I find about J is that you can still have a clear picture in your mind about how it will be executed, without actually having to write the details yourself. It also ties with SICP lesson of developing a language over a language with right abstractions at each level.
- moonchild 6y ago
  > there is no word on the algorithm - 'how is the array sorted' is the question that computer science is designed to solve, as opposed to 'what are the fixed points of a sorted array' '[H]ow is the array sorted' is one question that computer science can be used to solve. However, assuming it has been solved (which, for many cases, it has), a much more interesting question is 'what can we do with a sorted array'; or, 'given that sorting has such-and-such time or memory complexity, how hard is this other algorithm?' The OP isn't showing off a particular sorting algorithm, he's showing off a sorting interface, because that's more novel in context. Sorting according to a predicate could be done with outer product/table+reduction, but usually you don't need anything like that. Let's say you have an array of 3-vectors, and you want to sort the vectors according to the 2nd item in each vector. So your sort predicate is v1[1] < v2[1]. We can just sort the whole array according to the vector of the 2nd items of each vector; so: ] x =. ?@] 9 3$10 2 4 9 3 8 2 8 0 4 3 9 6 3 1 8 2 6 8 1 2 5 9 4 0 5 0 9 (<a:;1) { x NB. index: the first element of every row of x 4 8 0 9 1 6 2 4 0 1 { |: x NB. first element of the transpose of x; equivalent 4 8 0 9 1 6 2 4 0 x /: 1 { |: x NB. x sorted according to the above. Notice that the second column is now sorted 8 0 4 5 0 9 3 1 8 1 2 5 2 4 9 9 4 0 2 6 8 3 8 2 3 9 6
siraben 6y ago
This was a nice tour of how concise J can be. To see more of this stuff I highly recommend the Concrete Mathematics Companion[0], which uses J as an algebraic tool to reason about the correctness of derivations in that excellent book by Knuth. [0] https://www.jsoftware.com/books/pdf/cmc.pdf https://www.jsoftware.com/books/pdf/cmc.pdf
gspr 6y ago
The article has the footnote > APL was the first language to use “monad” as a term. The popular FP meaning only appeared thirty years later. in its, to me odd, use of the word "monad". This is incorrect. The "popular FP meaning" arose because it's a very special case of the standard concept from mathematics [1]. The mathematical terminology harkens back to the 1950s, and thus predates APL. In Haskell, a monad is precisely a monad in the mathematical sense in the special case of the category of Haskell types (which, due to technical complications isn't exactly a perfect model for the actual Haskell types, but it's pretty close). It thus seems dishonest to me to refer to this use of the word "monad" as somehow being newer than APL's use. [1] https://en.wikipedia.org/wiki/Monad_(category_theory) https://en.wikipedia.org/wiki/Monad_(category_theory)
- 4bpp 6y ago
  > The mathematical terminology harkens back to the 1950 Does it? The earliest instances I can find are in Mac Lane's 1971 book or thereabouts; the older term was "triple". Wikipedia dates APL to 1966. "Monad" has also seen nebulous use in philosophy at least since Leibniz.
  - ggrrhh_ta 6y ago
    That's an interesting reference...
  - notagoodidea 6y ago
    In fact since the Pythagoreans [1] where the monad comes first and after the dyad and after the numbers and so on. It also was the term for unicellular organism before this was coined. Monadic was an adjective used in logic before the 20th century, using the noun dervied by the greek root monas which gave monad and monadic (and the same with dyad, dyadic). If I take for right what is on the french [2] and english pages [3] of wikipedia about the terminology history of the monad words its usages seems to date to around the 70 when Saunders Mac Lane named it in reference to the philosophical term. I feels some times people tend to forget how interconnected is and was mathematics and philosophy and how term from philosophy were used in a mathematical context based on their philosophical definition. [1] https://en.wikipedia.org/wiki/Monad_(philosophy) https://en.wikipedia.org/wiki/Monad_(philosophy) [2] https://fr.wikipedia.org/wiki/Monade_(th%C3%A9orie_des_cat%C3%A9gories) https://fr.wikipedia.org/wiki/Monade_(th%C3%A9orie_des_cat%C... [3] https://en.wikipedia.org/wiki/Monad_(category_theory)#Terminological_history https://en.wikipedia.org/wiki/Monad_(category_theory)#Termin...
chrispsn 6y ago
I agree with most of this article, except that - for me - grouping by table columns is a more intuitive and flexible way to control subset application than 'rank'. k9 is in development, but blending k7 and k9 we'd get something like: t: [[]sensor: 1 2 1 2 1 2 day: 1 1 2 2 1 1 reading: 1 2 3 4 5 6] 0 1 1 0 * select by sensor, day from t sensor day| ------ ---|---------------- 1 1 |[[]reading: 0 0] 1 2 |[[]reading: ,3] 2 1 |[[]reading: 2 6] 2 2 |[[]reading: ,0]
- wvlia5 6y ago
  This is 'pivot' in python/pandas and excel
BiteCode_dev 6y ago
J should be a dsl for array embedded in other languages, the way regexes are for text.
nonbirithm 6y ago
J and K remind me of the essay about the Lisp Curse[0], which mentions that the expressiveness of the language became a sort of Achilles heel in its culture. It talks about people writing their projects in Lisp and not expecting other people to adapt to their conventions or combine their efforts on one library, where every solution worked well enough initially - but only for one person, its author. In K the error messages are far harder to understand than most languages, and you'd need either enough comments or many hours of experience with the language to understand what a piece of code does, due to its sheer expressiveness. Also, there's comparatively less documentation for when you're trying to do something practical. I can't imagine introducing J or K in an open source project will be helpful if other contributors have to catch up to my understanding of the language and need to internalize and remember which single-character symbols mean "print" or "iter." I imagine that at least a Python programmer could reasonably understand the gist of a similar Ruby program, merely due to the fact that "class" and "map" and "length" are sequences of characters that usually show up in both languages. Not so with K. Still, I find the vastly different ways of doing things with K interesting to see, even if I will only use it personally. [0] http://winestockwebdesign.com/Essays/Lisp_Curse.html http://winestockwebdesign.com/Essays/Lisp_Curse.html
- ggrrhh_ta 6y ago
  Very interesting link.
- coliveira 6y ago
  That is fine. Most people simply cannot drive a Formula-1 car, but it would be stupid to conclude that a F-1 is a cursed car and nobody should want to drive one. Similarly, a few languages can only be used by highly trained people who understand well how it works. Lisp, Forth, and J are in that category of tools that require highly trained people. In their hand, one of these languages can do fantastic things, in the hands of an average programmer you have nothing.
  - dunefox 6y ago
    Lisp without macros can be and is used by 'normal programmers'. And even with macros it's not like it's witchcraft that only a select few can handle. It's a powerful language, but nonetheless a language that everyone can learn.
yladiz 6y ago
Out of curiosity, internally does a language like J `map` over arrays even though the array is a primitive? Meaning, is `map` essentially a “primitive function”?

160 points by janvdberg 6y ago

Interesting blog post. All of these are implemented as trivial operations in numpy, so while J or APL might be the originator of the syntax, this 'tool of thought' is not as esoteric as the author makes it sound, and learning J if you already know numpy (or TensorFlow or PyTorch or..) isn't that remarkable. >> np.array([1, 2, 3]) * 2 array([2, 4, 6]) >>> np.array([1, 2, 3]) * np.array([4, 5, 6]) >>> np.arange(0, 16).reshape(4, 4) + np.array([5, 5, 5, 5]) array([[ 5, 6, 7, 8], [ 9, 10, 11, 12], [13, 14, 15, 16], [17, 18, 19, 20]])

moonchild 6y ago
You've reproduced the most trivial example, which many languages make easy. I would be very interested to see another language with a rank operator, for instance. ------------------------------------------------------------------------ Challenge for you: rewrite a nontrivial program in one of those frameworks, with the following restrictions: - No iteration (including implicit iterations—map, filter; reduce is ok) - No loops whatsoever. Recursion is ok, but should be avoided wherever possible. - No explicitly named arguments; everything in pointfree style. (I know, map/filter can be implemented recursively. But compared with the equivalent constructs in apl, they're about 10× as verbose, and harder to reason about and understand. Even reduce is somewhat of a gimme.)
- ggrrhh_ta 6y ago
  I have want to chime in. Agree, it is great that ideas from APL have made it into the language... But there is, fortunately, a long and fruitful path still to explore... I love that J is like an alternative path to functional approaches like Haskell and how it plays in terms of providing expressive power. Still, I will not create code in J that is maintained jointly with other users from other domains (which is the case usually when you are using numpy, for example)
  - TeMPOraL 6y ago
    I like how APL made it into Common Lisp wholesale. There's a library that compiles APL to Common Lisp, allowing you to mix APL and Lisp code, working on the same data structures with both languages. https://github.com/phantomics/april https://github.com/phantomics/april Example session I just did, copying APL code from https://aplwiki.com/wiki/John_Scholes%27_Conway%27s_Game_of_Life https://aplwiki.com/wiki/John_Scholes%27_Conway%27s_Game_of_...: CL-USER> (april:with-april-context ((:space *lifespace*)) (april:april "Show←{'·⌺'[⎕IO+⍵]}") (april:april "glider←3 3⍴1 1 1 1 0 0 0 1 0") (april:april-f "Show glider") (print (april:april "glider")) ; to prove it's a Lisp array (terpri) (april:april "grid←¯10 ¯10↑glider") (april:april-f "Show grid") (terpri) (april:april "life ← {⊃1 ⍵ ∨.∧ 3 4 = +/ +⌿ ¯1 0 1 ∘.⊖ ¯1 0 1 ⌽¨ ⊂⍵}") (terpri) (april:april-f "Show¨{life⍣⍵⊢grid}¨0,⍳3") (terpri)) ⌺⌺⌺ ⌺·· ·⌺· #2A((1 1 1) (1 0 0) (0 1 0)) ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·······⌺⌺⌺ ·······⌺·· ········⌺· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ·········· ········⌺· ·······⌺⌺· ·······⌺⌺· ·······⌺⌺⌺ ·······⌺⌺· ·······⌺·⌺ ······⌺⌺·· ·······⌺·· ·······⌺·⌺ ·······⌺·· ········⌺· ········⌺· ·········· ·········· ·········· (APL being compiled to Lisp is important, not just because it gives you proper interoperability out of the box, but also because on implementations like SBCL this means APL code will get compiled, through Lisp, to native code. And you can do that at runtime too.) Now I wish I could find something that does the same for J, because I'm sure as hell not going to buy an APL-friendly keyboard.
    - icen 6y ago
      Emacs has an excellent APL keyboard implementation (via gnu-apl-mode) which uses . as a prefix for the apl keys.
    - moonchild 6y ago
      > APL being compiled to Lisp is important, not just because it gives you proper interoperability out of the box, but also because on implementations like SBCL this means APL code will get compiled, through Lisp, to native code Except that simd in lisp isn't great and needs to be done mostly by hand[1]; and simd is super important for making apl performant. Also because gc attributes that are good for lisp aren't great for apl. Also because compiling to native code doesn't actually really affect performance of apl, and might actually harm it for larger programs. To be clear, april is a super-cool project, but not because of any of its performance characteristics. > Now I wish I could find something that does the same for J, because I'm sure as hell not going to buy an APL-friendly keyboard. You don't need a special keyboard, just a layout. I don't know how it works on windows (there's an 'IME', I think?) or macos, but x comes with a keyboard layout. I use this: setxkbmap -layout us,apl -variant dyalog -option grp:rwin_switch Which makes it so that holding down the right super key and pressing any key produces an appropriate apl symbol. You may want to use a different key to switch; check 'man xkeyboard-config' and then search for 'Switching to another layout' for other options. Try: modifier key (whatever it is)+r, for ⍴ (rho). 1. https://www.reddit.com/r/lisp/comments/fkfgjn/sbcl_with_simd_how_to_optimize_sseavx2_to_pointer/ https://www.reddit.com/r/lisp/comments/fkfgjn/sbcl_with_simd...
- SuperCuber 6y ago
  Rank Is actually implicitly done by numpy using a mechanism called broadcasting. For example: >>> np.array([10, 20, 30]) + np.array([[1,2,3], [4,5,6], [7,8,9]]) array([[11, 22, 33], [14, 25, 36], [17, 28, 39]]) Sieves exist in numpy, called masks: >>>np.array([10, 20, 30]) > 15 array([False, True, True]) Of course they can be operated on just like any other numpy array. Grades exist in numpy: >>>np.array([5,4,3,2,1]).argsort() array([4, 3, 2, 1, 0]) Of course they are a little more verbose since all of those operations are from the library and not native to python.
  - ipsum2 6y ago
    To do sieves in J like the blog post mentioned, the equivalent numpy would be: >>> arr = np.array([10, 20, 30]) >>> arr[arr > 15] array([20, 30])
  - moonchild 6y ago
    > Rank Is actually implicitly done by numpy using a mechanism called broadcasting. Numpy's broadcasting is scalar conformability, to which the rank operator (and general conformability) provides a general case. Example in j: ] x =. i. 2 3 4 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 ] y =. i. 3 4 0 1 2 3 4 5 6 7 8 9 10 11 x + y NB. this will error because + expects that, if its arguments' shapes are not the same, one will be a prefix of the other |length error | x +y NB. this is easy enough to fix, however x +"2 y NB. +"2 is shorthand for +"2 2; meaning, choose rank-2 arrays from both the left and right arguments 0 2 4 6 8 10 12 14 16 18 20 22 12 14 16 18 20 22 24 26 28 30 32 34 Numpy will actually do this without the rank operator, because it uses suffix agreement rather than prefix agreement (which is absolutely bonkers—j used suffix agreement for about 5 minutes in 1990, before realising it was an awful idea). For for numpy, see if you can add: np.array([[[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]], [[12, 13, 14, 15], [16, 17, 18, 19], [20, 21, 22, 23]]]) + np.array([[0, 1, 2], [3, 4, 5]]) Intelligently. (I'm sure it's not overly difficult to come up with a solution, but can you do it with a single higher-order function call which generalises to other argument shapes?) (The j equivalent, (i. 2 3 4) + (i. 2 3) also works without trouble.) ------------------------------------------------------------------------ Another example, which may be more illustrative, is the ability to perform reductions along arbitrary axes. For example: ] x =. i. 4 3 0 1 2 3 4 5 6 7 8 9 10 11 +/ x NB. sum reduced along leading axis, the default, producing an array of shape 3 18 22 26 +/"1 x NB. sum each rank-1 array (vector); or, reduce last axis, producing an array of shape 4 3 12 21 30 ------------------------------------------------------------------------ Another curiosity, which I have thus far neglected, is the extent to which numpy's being 'a little more verbose' is actually incredibly important in shaping the way you approach and think about problems. The great-uncle comment also addresses this, but Iverson probably says it better than either of us can: read https://www.jsoftware.com/papers/tot.htm https://www.jsoftware.com/papers/tot.htm
ggrrhh_ta 6y ago
Exactly, one of the things I like about numpy is how it drew from APL and J. It is a more verbose language, and truth be told, J still has more expressive power (non regular processing of data, even state-machine-like processing) which can still influence numpy, or Julia.
smabie 6y ago
Comparing numpy to J is like comparing a dirty rag to a designer suit. The examples you've listed are trivial; the power of J and other array languages cannot be appreciated from afar. The philosophy behind array languages runs much deeper than adding two arrays or transposing matrices.
- ipsum2 6y ago
  Can you give an example of something that can be done in J that can't be done easily in numpy?
  - sn41 6y ago
    It is not about "cannot be done". Most high-level languages are Turing complete, hence every program in one can be written as some program in another. The real clincher is expressiveness: shorter code is better, modular code is better, and languages which avoid unnecessary repetitiveness of boiler plate code is better. Different languages provide advantages along one or more of these parameters.
    - shawnz 6y ago
      So can you give an example of something that can be expressed simply in APL/J/K that can't be expressed simply with numpy? And to be clear I don't think "less characters" makes the expression more simple. Maybe "less statements" or "less operators"
      - smabie 6y ago
        Here's Conway's Game of Life by Arthur Whitney (creator of K): life:{3=a-x*4=a:2{+(0+':x)+1_x,0}/x} I'm guessing a numpy implementation would be between one and two orders of magnitude more verbose, even if you're just comparing the number of operations.
        worc3131 6y ago
        It doesn't quite seem fair to compare numpy to one of the most famous K code golf's ever created, but here is my attempt, 140 characters compared to Arthur's 30, so 5 times longer. Not worry about padding the edges would cut it to 100. I don't think the gap is as wide as you think it is. def life(M): MP = np.pad(M,[(1,1),(1,1)]) N = sum(np.roll(MP,(i,j),(0,1)) for i in [1,0,-1] for j in [1,0,-1]) return (3==N-MP*(4==N))[1:-1,1:-1] Speedtest (presumably memory bound): 100 x 100 matrix, numpy 0.2ms, K 0.2ms 200 x 200 matrix, numpy 0.5ms, K 0.8ms 500 x 500 matrix, numpy 5.2ms, K 8.0ms 1000x1000 matrix, numpy 20.0ms, K 36.0ms 5000x5000 matrix, numpy 0.5s , K 1.2s Interestingly the K code is designed to return a boolean matrix, but operates much more slowly for me on a boolean matrix compared to an integer matrix, with the result that: q)\t klife M Takes 1.2 seconds whilst q)\t klife klife M Takes 24 seconds. So whilst the idea seems to be that klife is used with scan/over to run a number of iterations, it's actually a bad idea to run it more than once.

wodenokoto 6y ago

This reminded me very much of working in R or working with Numpy. I however always thought having an array as the primitive in R was great only because R is focused on statistics and data science. My understanding is that J claims to be general purpose programming and as such I’m surprised the paradigm holds.

ggrrhh_ta 6y ago
If you run the J GUI, written in J, it is going with blow your mind about how general purpose it is...
- mst 6y ago
  I wish somebody would write a walkthrough of writing something like a piece of GUI to show that off. (I keep trying to figure out J and ... sliding off ... every time I need to do something "normal" ... which is annoying and the fact I'm reduced to wishcasting in HN comments as a solution makes me disappointed in both myself and the universe)

justusw 6y ago

In J, I think it's very difficult to move on from the initial stage of dealing with "line noise" as the author mentions. For me, this was further complicated by the fact that error messages are very hard to understand in the beginning. I've written an article that describes how I typically deal with solving most of the domain/rank/length errors: https://www.justus.pw/posts/2020-07-31-errors-in-j.html https://www.justus.pw/posts/2020-07-31-errors-in-j.html

tsimionescu 6y ago

This seems very interesting for numerical computation, but at first glance few of these operations seem to apply directly to general computation (I'm sure that there are isomorphisms that can be used to apply them indirectly). The example with sorting was the oddest from this point of view. The author praised the idea of the permutation vector as being a relatively direct mathematical specification of sorting (return the permutation of the original array in sorted order) while leaving out the 'sorted' part. Also, this focuses entirely on the result, but there is no word on the algorithm - 'how is the array sorted' is the question that computer science is designed to solve, as opposed to 'what are the fixed points of a sorted array'. As such, as someone with no experience of J, I'm left wondering if this is another example of the infamous Haskell quicksort - well-picked examples of code which produces a desired result are extraordinarily terse and expressive, but actually implementing well-known algorithms is often just as verbose as C, give or take.

ggrrhh_ta 6y ago
Yes, once one goes really deep, it's more like C in the end. One good thing I find about J is that you can still have a clear picture in your mind about how it will be executed, without actually having to write the details yourself. It also ties with SICP lesson of developing a language over a language with right abstractions at each level.
moonchild 6y ago
> there is no word on the algorithm - 'how is the array sorted' is the question that computer science is designed to solve, as opposed to 'what are the fixed points of a sorted array' '[H]ow is the array sorted' is one question that computer science can be used to solve. However, assuming it has been solved (which, for many cases, it has), a much more interesting question is 'what can we do with a sorted array'; or, 'given that sorting has such-and-such time or memory complexity, how hard is this other algorithm?' The OP isn't showing off a particular sorting algorithm, he's showing off a sorting interface, because that's more novel in context. Sorting according to a predicate could be done with outer product/table+reduction, but usually you don't need anything like that. Let's say you have an array of 3-vectors, and you want to sort the vectors according to the 2nd item in each vector. So your sort predicate is v1[1] < v2[1]. We can just sort the whole array according to the vector of the 2nd items of each vector; so: ] x =. ?@] 9 3$10 2 4 9 3 8 2 8 0 4 3 9 6 3 1 8 2 6 8 1 2 5 9 4 0 5 0 9 (<a:;1) { x NB. index: the first element of every row of x 4 8 0 9 1 6 2 4 0 1 { |: x NB. first element of the transpose of x; equivalent 4 8 0 9 1 6 2 4 0 x /: 1 { |: x NB. x sorted according to the above. Notice that the second column is now sorted 8 0 4 5 0 9 3 1 8 1 2 5 2 4 9 9 4 0 2 6 8 3 8 2 3 9 6

siraben 6y ago

This was a nice tour of how concise J can be. To see more of this stuff I highly recommend the Concrete Mathematics Companion[0], which uses J as an algebraic tool to reason about the correctness of derivations in that excellent book by Knuth. [0] https://www.jsoftware.com/books/pdf/cmc.pdf https://www.jsoftware.com/books/pdf/cmc.pdf

gspr 6y ago

The article has the footnote > APL was the first language to use “monad” as a term. The popular FP meaning only appeared thirty years later. in its, to me odd, use of the word "monad". This is incorrect. The "popular FP meaning" arose because it's a very special case of the standard concept from mathematics [1]. The mathematical terminology harkens back to the 1950s, and thus predates APL. In Haskell, a monad is precisely a monad in the mathematical sense in the special case of the category of Haskell types (which, due to technical complications isn't exactly a perfect model for the actual Haskell types, but it's pretty close). It thus seems dishonest to me to refer to this use of the word "monad" as somehow being newer than APL's use. [1] https://en.wikipedia.org/wiki/Monad_(category_theory) https://en.wikipedia.org/wiki/Monad_(category_theory)

4bpp 6y ago
> The mathematical terminology harkens back to the 1950 Does it? The earliest instances I can find are in Mac Lane's 1971 book or thereabouts; the older term was "triple". Wikipedia dates APL to 1966. "Monad" has also seen nebulous use in philosophy at least since Leibniz.
- ggrrhh_ta 6y ago
  That's an interesting reference...
- notagoodidea 6y ago
  In fact since the Pythagoreans [1] where the monad comes first and after the dyad and after the numbers and so on. It also was the term for unicellular organism before this was coined. Monadic was an adjective used in logic before the 20th century, using the noun dervied by the greek root monas which gave monad and monadic (and the same with dyad, dyadic). If I take for right what is on the french [2] and english pages [3] of wikipedia about the terminology history of the monad words its usages seems to date to around the 70 when Saunders Mac Lane named it in reference to the philosophical term. I feels some times people tend to forget how interconnected is and was mathematics and philosophy and how term from philosophy were used in a mathematical context based on their philosophical definition. [1] https://en.wikipedia.org/wiki/Monad_(philosophy) https://en.wikipedia.org/wiki/Monad_(philosophy) [2] https://fr.wikipedia.org/wiki/Monade_(th%C3%A9orie_des_cat%C3%A9gories) https://fr.wikipedia.org/wiki/Monade_(th%C3%A9orie_des_cat%C... [3] https://en.wikipedia.org/wiki/Monad_(category_theory)#Terminological_history https://en.wikipedia.org/wiki/Monad_(category_theory)#Termin...

chrispsn 6y ago

I agree with most of this article, except that - for me - grouping by table columns is a more intuitive and flexible way to control subset application than 'rank'. k9 is in development, but blending k7 and k9 we'd get something like: t: [[]sensor: 1 2 1 2 1 2 day: 1 1 2 2 1 1 reading: 1 2 3 4 5 6] 0 1 1 0 * select by sensor, day from t sensor day| ------ ---|---------------- 1 1 |[[]reading: 0 0] 1 2 |[[]reading: ,3] 2 1 |[[]reading: 2 6] 2 2 |[[]reading: ,0]

wvlia5 6y ago
This is 'pivot' in python/pandas and excel

BiteCode_dev 6y ago

J should be a dsl for array embedded in other languages, the way regexes are for text.

nonbirithm 6y ago

J and K remind me of the essay about the Lisp Curse[0], which mentions that the expressiveness of the language became a sort of Achilles heel in its culture. It talks about people writing their projects in Lisp and not expecting other people to adapt to their conventions or combine their efforts on one library, where every solution worked well enough initially - but only for one person, its author. In K the error messages are far harder to understand than most languages, and you'd need either enough comments or many hours of experience with the language to understand what a piece of code does, due to its sheer expressiveness. Also, there's comparatively less documentation for when you're trying to do something practical. I can't imagine introducing J or K in an open source project will be helpful if other contributors have to catch up to my understanding of the language and need to internalize and remember which single-character symbols mean "print" or "iter." I imagine that at least a Python programmer could reasonably understand the gist of a similar Ruby program, merely due to the fact that "class" and "map" and "length" are sequences of characters that usually show up in both languages. Not so with K. Still, I find the vastly different ways of doing things with K interesting to see, even if I will only use it personally. [0] http://winestockwebdesign.com/Essays/Lisp_Curse.html http://winestockwebdesign.com/Essays/Lisp_Curse.html

ggrrhh_ta 6y ago
Very interesting link.
coliveira 6y ago
That is fine. Most people simply cannot drive a Formula-1 car, but it would be stupid to conclude that a F-1 is a cursed car and nobody should want to drive one. Similarly, a few languages can only be used by highly trained people who understand well how it works. Lisp, Forth, and J are in that category of tools that require highly trained people. In their hand, one of these languages can do fantastic things, in the hands of an average programmer you have nothing.
- dunefox 6y ago
  Lisp without macros can be and is used by 'normal programmers'. And even with macros it's not like it's witchcraft that only a select few can handle. It's a powerful language, but nonetheless a language that everyone can learn.

yladiz 6y ago

Out of curiosity, internally does a language like J `map` over arrays even though the array is a primitive? Meaning, is `map` essentially a “primitive function”?