5 ms·

BQN is great, but I rarely get to use it for anything... useful. I hope the ecosystem and tooling around bqn will get better in the future. I would like to see

BQN is great, but I rarely get to use it for anything... useful. I hope the ecosystem and tooling around bqn will get better in the future. I would like to see something like what dyalog is to apl for bqn. A jupyter notebook style ide+repl (maybe a bqn for of RIDE), libraries for common scripting tasks, embedding (a la lua) in more languages, maybe even a way to compile to libs so you can call bqn code from other languages. I'd definitely use it more if it was open to more use cases.

avmich 4y ago
Have some experience with J, would like to comment. > A jupyter notebook style ide+repl (maybe a bqn for of RIDE) J terminal looks pretty close. > libraries for common scripting tasks There are phrases (see www.jsoftware.com), but for many tasks it seems not many libraries are actually needed. > embedding (a la lua) in more languages, maybe even a way to compile to libs so you can call bqn code from other languages There is a way to call C from J and vice versa; not too convenient maybe... but still quite possible. APL languages don't benefit much from compilation, J interpreter is very fast.
- jjtheblunt 4y ago
  > APL languages don't benefit much from compilation Is the idea because most the action is in already compiled datastructure manipulations, in the standard library, specified by the terse source?
  - hoosieree 4y ago
    The operations on "entire arrays at once" don't benefit much from compilation, because the interpreter overhead is amortized across the size of the array. For "one potato two potato" scalar code code with lots of branches and control flow, compiled will always be faster because each little op has overhead and a compiler could optimize it away. A sufficiently smart interpreter or JIT could theoretically optimize this away too, but as far as I know no APL (or APL-like) language uses anything like a tracing JIT, instead they have historically focused on optimizing idiomatic expressions so "typical" code is faster.
- rscho 4y ago
  J also exposes an API to Python, allowing to use J from Python, meaning you don't have to wait: J in Jupyter is already here! https://code.jsoftware.com/wiki/Addons/api/python3 https://code.jsoftware.com/wiki/Addons/api/python3
  - joebo 4y ago
    I put this together about a year ago to run J/Jupyter in binder https://github.com/joebo/jkernel-docker https://github.com/joebo/jkernel-docker
- moonchild 4y ago
  > APL languages don't benefit much from compilation Sorry, but this is nonsense. APL tends to spend all its time waiting for memory, because it fuses nought. And it benefits as much as anybody else from things like common subexpression elimination and loop-invariant code motion. APL implementations appear to be fast because of a few extenuating factors: 1. C compilers kind of suck, and c kind of sucks for writing fast code, yet it is the de-facto standard, so it is what things are compared to. 2. Developers of apl implementation care about and prioritise performance. 3. In comparison with languages such as python, and in particular their popular implementations, apl spends very little time on dispatch.
  - mlochbaum 4y ago
    What kinds of numbers do you expect here? It's really only arithmetic that's memory-bound, even something like a scan or compress/filter does a fair amount of CPU work and wouldn't benefit that much from fusion (if you even can fuse compress). And with fewer loops I don't think loop optimization is as relevant. I think a factor of 2 is the median kind of improvement I'd expect from a compiled SIMD APL on array programs? Which is a long way from the 10x or so you get by adding a JIT to other dynamic interpreted languages, and likely harder. I would describe that as APL not benefitting much from compilation; maybe you wouldn't. (I've written some about what array compilation would mean at https://mlochbaum.github.io/BQN/implementation/compile/intro.html https://mlochbaum.github.io/BQN/implementation/compile/intro...)
    - moonchild 4y ago
      I think a factor of 2 is quite significant, but I also think you may understate the advantages of compilation. I do agree the benefit is prone to be less than for a language like python. Regarding scans (filter is a type of scan, howbeit easier to implement than the general case), there is in general an annoying work-span discrepancy, which bodes ill for contemporary computers with their finite parallelism. I would buffer heavily here, but fusing the scan with its input allows for the use of a work-efficient implementation, spending all latent parallelism on generating more inputs. When I speak of loops, I am including any usage of rank (incl. implicit); in this respect, apl is chock-full of loops. Some loop optimisations may be obviated, of course, because of referential transparency, but others arise in their place. Take for instance some recent work[0] done on futhark: rather than parallelise an already in-place algorithm, they had to in-place an already parallel algorithm! Two more points: A problem may be bound by memory latency. I spent some time tuning j's I. recently, and it does many searches in parallel (4 for a small search space, 12 for a large one, iirc), but it is still bound by latency. If I could fetch a pivot corresponding to the first cell of y, and then generate the next cell while I wait, I would make better use of resources. A compiler results in more transparent performance characteristics. When I target an interpreter, I must write my idioms and special combinations in exactly the way it expects, else nothing will happen; a compiler will care much less about exact phrasing. Similarly, like I mentioned, you get licm and cse for free, rather than needing to do them by hand. 0. https://futhark-lang.org/blog/2022-11-03-short-circuiting.html https://futhark-lang.org/blog/2022-11-03-short-circuiting.ht...

avmich 4y ago

Have some experience with J, would like to comment. > A jupyter notebook style ide+repl (maybe a bqn for of RIDE) J terminal looks pretty close. > libraries for common scripting tasks There are phrases (see www.jsoftware.com), but for many tasks it seems not many libraries are actually needed. > embedding (a la lua) in more languages, maybe even a way to compile to libs so you can call bqn code from other languages There is a way to call C from J and vice versa; not too convenient maybe... but still quite possible. APL languages don't benefit much from compilation, J interpreter is very fast.

jjtheblunt 4y ago
> APL languages don't benefit much from compilation Is the idea because most the action is in already compiled datastructure manipulations, in the standard library, specified by the terse source?
- hoosieree 4y ago
  The operations on "entire arrays at once" don't benefit much from compilation, because the interpreter overhead is amortized across the size of the array. For "one potato two potato" scalar code code with lots of branches and control flow, compiled will always be faster because each little op has overhead and a compiler could optimize it away. A sufficiently smart interpreter or JIT could theoretically optimize this away too, but as far as I know no APL (or APL-like) language uses anything like a tracing JIT, instead they have historically focused on optimizing idiomatic expressions so "typical" code is faster.
rscho 4y ago
J also exposes an API to Python, allowing to use J from Python, meaning you don't have to wait: J in Jupyter is already here! https://code.jsoftware.com/wiki/Addons/api/python3 https://code.jsoftware.com/wiki/Addons/api/python3
- joebo 4y ago
  I put this together about a year ago to run J/Jupyter in binder https://github.com/joebo/jkernel-docker https://github.com/joebo/jkernel-docker
moonchild 4y ago
> APL languages don't benefit much from compilation Sorry, but this is nonsense. APL tends to spend all its time waiting for memory, because it fuses nought. And it benefits as much as anybody else from things like common subexpression elimination and loop-invariant code motion. APL implementations appear to be fast because of a few extenuating factors: 1. C compilers kind of suck, and c kind of sucks for writing fast code, yet it is the de-facto standard, so it is what things are compared to. 2. Developers of apl implementation care about and prioritise performance. 3. In comparison with languages such as python, and in particular their popular implementations, apl spends very little time on dispatch.
- mlochbaum 4y ago
  What kinds of numbers do you expect here? It's really only arithmetic that's memory-bound, even something like a scan or compress/filter does a fair amount of CPU work and wouldn't benefit that much from fusion (if you even can fuse compress). And with fewer loops I don't think loop optimization is as relevant. I think a factor of 2 is the median kind of improvement I'd expect from a compiled SIMD APL on array programs? Which is a long way from the 10x or so you get by adding a JIT to other dynamic interpreted languages, and likely harder. I would describe that as APL not benefitting much from compilation; maybe you wouldn't. (I've written some about what array compilation would mean at https://mlochbaum.github.io/BQN/implementation/compile/intro.html https://mlochbaum.github.io/BQN/implementation/compile/intro...)
  - moonchild 4y ago
    I think a factor of 2 is quite significant, but I also think you may understate the advantages of compilation. I do agree the benefit is prone to be less than for a language like python. Regarding scans (filter is a type of scan, howbeit easier to implement than the general case), there is in general an annoying work-span discrepancy, which bodes ill for contemporary computers with their finite parallelism. I would buffer heavily here, but fusing the scan with its input allows for the use of a work-efficient implementation, spending all latent parallelism on generating more inputs. When I speak of loops, I am including any usage of rank (incl. implicit); in this respect, apl is chock-full of loops. Some loop optimisations may be obviated, of course, because of referential transparency, but others arise in their place. Take for instance some recent work[0] done on futhark: rather than parallelise an already in-place algorithm, they had to in-place an already parallel algorithm! Two more points: A problem may be bound by memory latency. I spent some time tuning j's I. recently, and it does many searches in parallel (4 for a small search space, 12 for a large one, iirc), but it is still bound by latency. If I could fetch a pivot corresponding to the first cell of y, and then generate the next cell while I wait, I would make better use of resources. A compiler results in more transparent performance characteristics. When I target an interpreter, I must write my idioms and special combinations in exactly the way it expects, else nothing will happen; a compiler will care much less about exact phrasing. Similarly, like I mentioned, you get licm and cse for free, rather than needing to do them by hand. 0. https://futhark-lang.org/blog/2022-11-03-short-circuiting.html https://futhark-lang.org/blog/2022-11-03-short-circuiting.ht...