7 ms·

I suppose if arrays language get popular enough, we will get a module to use them as a DSL for libs like numpy, just like we have regex for strings, instead of

I suppose if arrays language get popular enough, we will get a module to use them as a DSL for libs like numpy, just like we have regex for strings, instead of a whole language dedicated for them. It would be a win / win, you gain the strength of J, K and APL for array processing, without the weakness of them for anything else. And just like with regex, you'll get fat disclaimers in doc telling you to not get carried away too much as it can quickly become unreadable.

gugagore 5y ago
The regex DSL only works for strings. I lament that I cannot use something regex-like to match general sequences, e.g. a sequence of tokens, instead of only strings (sequence of characters). The operations could be the same. There are classes, and operators for matching 0, 1, or more repetitions, etc. Array languages are powerful especially when you have arrays with arbitrary elements, including arrays themselves. Good luck using a regex DSL to match a sequence of strings, where you might want to define a string class (analogous to a character class) as a regex itself.
- mumblemumble 5y ago
  > I lament that I cannot use something regex-like to match general sequences, e.g. a sequence of tokens You can. It's not typically built into a programming language's standard library. But there are plenty of general-purpose automata-building libraries out there, and some of them do provide DSLs. At least to the extent that the regular expressions you're using are actually regular (many aren't), all a regex is is a domain-specific specialization of nondeterministic finite automata. I sometimes lament that I don't see them, or hand-coded automata, more often. This was CS101-level stuff when I was in college, and it's pretty easy to code your own even if you don't have a good library available in your language. And, for problems where they're appropriate, using them typically yields a result that's simpler and easier to maintain than whatever ad-hoc alternative you might see instead.
- hcarvalhoalves 5y ago
  There are a few, here's one: https://clojure.org/guides/spec#_sequences https://clojure.org/guides/spec#_sequences
- jasonwatkinspdx 5y ago
  The now defunct Viewpoints Research org chased an idea similar to this. It was a meta language that was based around PEGs, intended to allow the easy creation of DSLs. I imagine the papers are still up somewhere. It was called OMeta.
- bmn__ 5y ago
  > using a regex DSL to match a sequence of strings, where you might want to define a string class (analogous to a character class) as a regex itself. http://p3rl.org/retut#Defining-named-patterns http://p3rl.org/retut#Defining-named-patterns https://docs.raku.org/language/grammar_tutorial#The_technical_overview https://docs.raku.org/language/grammar_tutorial#The_technica... If you want a demo, reply with a concrete example I can implement.
- b2gills 5y ago
  I'm not sure of the need to do that. Not for parsing anyway. A well designed regex system is enough without tokenization. The parser for the Raku language, for example, is a collection of regexes composed into grammars. (A grammar is a type of class where the methods are regexes.) We could probably do the token thing with multi functions, if we had to. multi parse ( 'if', $ where /\s+/, …, '{', *@_ ) {…} multi parse ( 'for', $ where /\s+/, …, '{', *@_ ) {…} … Or something like that anyway. (Note that `if` and `for` etc are keywords only when they are followed immediately by whitespace.) I'm not sure how well that would work in practice; as hypothetically Raku doesn't start with any keywords or operators. They are supposed to seem like they are added the same way keywords and operators are added by module authors. (In order to bootstrap it we of course need to actually have keywords and operators there to build upon.) Since modules can add new things, we would need to update the list of known tokens as we are parsing. Which means that even if Raku did the tokenization thing, it would have to happen at the same time as the other steps. Tokenization seems like an antiquated way to create compilers. It was needed as there wasn't enough memory to have all of the stages loaded at the same time. --- Here is an example parser for JSON files using regexes in a grammar to show the simplicity and power of parsing things this way. grammar JSON::Parser::Example { token TOP { \s* <value> \s* } rule object { '{' ~ '}' <pairlist> } rule pairlist { <pair> * % ',' } rule pair { <string> ':' <value> } rule array { '[' ~ ']' <arraylist> } rule arraylist { <value> * % ',' } proto token value {*} token value:sym<number> { '-'? # optional negation [ 0 | <[1..9]> <[0..9]>* ] # no leading 0 allowed [ '.' <[0..9]>+ ]? # optional decimal point [ <[eE]> <[+-]>? <[0..9]>+ ]? # optional exponent } token value:sym<true> { <sym> } token value:sym<false> { <sym> } token value:sym<null> { <sym> } token value:sym<object> { <object> } token value:sym<array> { <array> } token value:sym<string> { <string> } token string { ｢"｣ ~ ｢"｣ [ <str> | ｢\｣ <str=.str_escape> ]* } token str { <-["\\\t\x[0A]]>+ } token str_escape { <["\\/bfnrt]> } } A `token` is a `regex` with `:ratchet` mode enabled (no backtracking). A `rule` is a `token` with `:sigspace` also enabled (whitespace becomes the same as a call to `<.ws>`). The only one of those that really looks anything like traditional regexes is the `value:sym<number>` token. (Raku replaced non capturing grouping `(?:…)` with `[…]`, and character classes `[eE]` with `<[eE]>`) This code was copied from https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Grammar.pm https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Gra... but some parts were simplified to be slightly easier to understand. Mainly I removed the Unicode handling capabilities. It will generate a tree based structure when you use it. my $json = Q:to/END/; { "foo": ["bar", "baz"], "ultimate-answer": 42 } END my $result = JSON::Parser::Example.parse($json); say $result; The above will display the resultant tree structure like this: ｢{ "foo": ["bar", "baz"], "ultimate-answer": 42 } ｣ value => ｢{ "foo": ["bar", "baz"], "ultimate-answer": 42 } ｣ object => ｢{ "foo": ["bar", "baz"], "ultimate-answer": 42 } ｣ pairlist => ｢"foo": ["bar", "baz"], "ultimate-answer": 42 ｣ pair => ｢"foo": ["bar", "baz"]｣ string => ｢"foo"｣ str => ｢foo｣ value => ｢["bar", "baz"]｣ array => ｢["bar", "baz"]｣ arraylist => ｢"bar", "baz"｣ value => ｢"bar"｣ string => ｢"bar"｣ str => ｢bar｣ value => ｢"baz"｣ string => ｢"baz"｣ str => ｢baz｣ pair => ｢"ultimate-answer": 42 ｣ string => ｢"ultimate-answer"｣ str => ｢ultimate-answer｣ value => ｢42｣ You can access parts of the data using array and hash accesses. my @pairs = $result<value><object><pairlist><pair>; my @keys = @pairs.map( *.<string>.substr(1,*-1) ); my @values = @pairs.map( *.<value> ); say @pairs.first( *.<string><str> eq 'ultimate-answer' )<value>; # ｢42｣ You can also pass in an actions class to do your processing at the same time. It is also a lot less fragile. See https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Actions.pm https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Act... for an example of an actions class. --- Note that things which you would historically talk about as a token such as `true`, `false`, and `null` are written using `token`. This is a useful association as it will naturally cause you to write shorter, more composable regexes. Since they are composable, we could do things like extend the grammar to add the ability to have non string keys. Or perhaps add `Inf`, `-Inf`, and `NaN` as values. grammar Extended::JSON::Example is JSON::Parser::Example { rule pair { <key=.value> ':' <value> } # replaces existing rule # adds to the existing set of value tokens. token value:sym<Inf> { <sym> } token value:sym<-Inf> { <sym> } token value:sym<NaN> { <sym> } } This is basically how Raku handles adding new keywords and operators under the hood.
robomartin 5y ago
APL is not unreadable any more than mathematical or musical notation are. Sure, to someone who doesn’t know the notation it looks like an incomprehensible mess. So does math, music, greek, chinese, arabic, hebrew, etc. I used APL professionally every day for ten years. I can read it. I can touch-type it. And I don’t need a specially-labelled keyboard (even thirty years later). This should not be surprising at all. A pianist does not need labeled keys and people familiar enough with the above-listed spoken languages can touch-type them without much effort. While, sadly, APL has no practical application in modern software engineering (it stagnated and became irrelevant and impractical) it is wrong to look at the brilliant use of notation as a tool for the concise communication and expression of ideas and list it as a negative. Not being able to speak, read or write Chinese does not make it a bad language.
- 0xdeadbeefbabe 5y ago
  > Not being able to speak, read or write Chinese does not make it a bad language. Well that's the problem. Not being able to read or write APL makes it more fun to learn.
O_H_E 5y ago
In Julia: https://github.com/shashi/APL.jl https://github.com/shashi/APL.jl
- bluenose69 5y ago
  Thanks for posting this intriguing link. A glance at the Jupyter notebook on this site will bring a smile to a Julia user who grew up on (or has somehow encountered) APL.
- snicker7 5y ago
  What an incredibly concise, elegant implementation.
RyanHamilton 5y ago
I've written an open source version of q: http://www.timestored.com/jq/ http://www.timestored.com/jq/ It's implemented in java. You idea is interesting. Allowing q and java intermixed... combining it inside java like linq and C# would be interesting.

gugagore 5y ago

The regex DSL only works for strings. I lament that I cannot use something regex-like to match general sequences, e.g. a sequence of tokens, instead of only strings (sequence of characters). The operations could be the same. There are classes, and operators for matching 0, 1, or more repetitions, etc. Array languages are powerful especially when you have arrays with arbitrary elements, including arrays themselves. Good luck using a regex DSL to match a sequence of strings, where you might want to define a string class (analogous to a character class) as a regex itself.

mumblemumble 5y ago
> I lament that I cannot use something regex-like to match general sequences, e.g. a sequence of tokens You can. It's not typically built into a programming language's standard library. But there are plenty of general-purpose automata-building libraries out there, and some of them do provide DSLs. At least to the extent that the regular expressions you're using are actually regular (many aren't), all a regex is is a domain-specific specialization of nondeterministic finite automata. I sometimes lament that I don't see them, or hand-coded automata, more often. This was CS101-level stuff when I was in college, and it's pretty easy to code your own even if you don't have a good library available in your language. And, for problems where they're appropriate, using them typically yields a result that's simpler and easier to maintain than whatever ad-hoc alternative you might see instead.
hcarvalhoalves 5y ago
There are a few, here's one: https://clojure.org/guides/spec#_sequences https://clojure.org/guides/spec#_sequences
jasonwatkinspdx 5y ago
The now defunct Viewpoints Research org chased an idea similar to this. It was a meta language that was based around PEGs, intended to allow the easy creation of DSLs. I imagine the papers are still up somewhere. It was called OMeta.
bmn__ 5y ago
> using a regex DSL to match a sequence of strings, where you might want to define a string class (analogous to a character class) as a regex itself. http://p3rl.org/retut#Defining-named-patterns http://p3rl.org/retut#Defining-named-patterns https://docs.raku.org/language/grammar_tutorial#The_technical_overview https://docs.raku.org/language/grammar_tutorial#The_technica... If you want a demo, reply with a concrete example I can implement.
b2gills 5y ago
I'm not sure of the need to do that. Not for parsing anyway. A well designed regex system is enough without tokenization. The parser for the Raku language, for example, is a collection of regexes composed into grammars. (A grammar is a type of class where the methods are regexes.) We could probably do the token thing with multi functions, if we had to. multi parse ( 'if', $ where /\s+/, …, '{', *@_ ) {…} multi parse ( 'for', $ where /\s+/, …, '{', *@_ ) {…} … Or something like that anyway. (Note that `if` and `for` etc are keywords only when they are followed immediately by whitespace.) I'm not sure how well that would work in practice; as hypothetically Raku doesn't start with any keywords or operators. They are supposed to seem like they are added the same way keywords and operators are added by module authors. (In order to bootstrap it we of course need to actually have keywords and operators there to build upon.) Since modules can add new things, we would need to update the list of known tokens as we are parsing. Which means that even if Raku did the tokenization thing, it would have to happen at the same time as the other steps. Tokenization seems like an antiquated way to create compilers. It was needed as there wasn't enough memory to have all of the stages loaded at the same time. --- Here is an example parser for JSON files using regexes in a grammar to show the simplicity and power of parsing things this way. grammar JSON::Parser::Example { token TOP { \s* <value> \s* } rule object { '{' ~ '}' <pairlist> } rule pairlist { <pair> * % ',' } rule pair { <string> ':' <value> } rule array { '[' ~ ']' <arraylist> } rule arraylist { <value> * % ',' } proto token value {*} token value:sym<number> { '-'? # optional negation [ 0 | <[1..9]> <[0..9]>* ] # no leading 0 allowed [ '.' <[0..9]>+ ]? # optional decimal point [ <[eE]> <[+-]>? <[0..9]>+ ]? # optional exponent } token value:sym<true> { <sym> } token value:sym<false> { <sym> } token value:sym<null> { <sym> } token value:sym<object> { <object> } token value:sym<array> { <array> } token value:sym<string> { <string> } token string { ｢"｣ ~ ｢"｣ [ <str> | ｢\｣ <str=.str_escape> ]* } token str { <-["\\\t\x[0A]]>+ } token str_escape { <["\\/bfnrt]> } } A `token` is a `regex` with `:ratchet` mode enabled (no backtracking). A `rule` is a `token` with `:sigspace` also enabled (whitespace becomes the same as a call to `<.ws>`). The only one of those that really looks anything like traditional regexes is the `value:sym<number>` token. (Raku replaced non capturing grouping `(?:…)` with `[…]`, and character classes `[eE]` with `<[eE]>`) This code was copied from https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Grammar.pm https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Gra... but some parts were simplified to be slightly easier to understand. Mainly I removed the Unicode handling capabilities. It will generate a tree based structure when you use it. my $json = Q:to/END/; { "foo": ["bar", "baz"], "ultimate-answer": 42 } END my $result = JSON::Parser::Example.parse($json); say $result; The above will display the resultant tree structure like this: ｢{ "foo": ["bar", "baz"], "ultimate-answer": 42 } ｣ value => ｢{ "foo": ["bar", "baz"], "ultimate-answer": 42 } ｣ object => ｢{ "foo": ["bar", "baz"], "ultimate-answer": 42 } ｣ pairlist => ｢"foo": ["bar", "baz"], "ultimate-answer": 42 ｣ pair => ｢"foo": ["bar", "baz"]｣ string => ｢"foo"｣ str => ｢foo｣ value => ｢["bar", "baz"]｣ array => ｢["bar", "baz"]｣ arraylist => ｢"bar", "baz"｣ value => ｢"bar"｣ string => ｢"bar"｣ str => ｢bar｣ value => ｢"baz"｣ string => ｢"baz"｣ str => ｢baz｣ pair => ｢"ultimate-answer": 42 ｣ string => ｢"ultimate-answer"｣ str => ｢ultimate-answer｣ value => ｢42｣ You can access parts of the data using array and hash accesses. my @pairs = $result<value><object><pairlist><pair>; my @keys = @pairs.map( *.<string>.substr(1,*-1) ); my @values = @pairs.map( *.<value> ); say @pairs.first( *.<string><str> eq 'ultimate-answer' )<value>; # ｢42｣ You can also pass in an actions class to do your processing at the same time. It is also a lot less fragile. See https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Actions.pm https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Act... for an example of an actions class. --- Note that things which you would historically talk about as a token such as `true`, `false`, and `null` are written using `token`. This is a useful association as it will naturally cause you to write shorter, more composable regexes. Since they are composable, we could do things like extend the grammar to add the ability to have non string keys. Or perhaps add `Inf`, `-Inf`, and `NaN` as values. grammar Extended::JSON::Example is JSON::Parser::Example { rule pair { <key=.value> ':' <value> } # replaces existing rule # adds to the existing set of value tokens. token value:sym<Inf> { <sym> } token value:sym<-Inf> { <sym> } token value:sym<NaN> { <sym> } } This is basically how Raku handles adding new keywords and operators under the hood.

robomartin 5y ago

APL is not unreadable any more than mathematical or musical notation are. Sure, to someone who doesn’t know the notation it looks like an incomprehensible mess. So does math, music, greek, chinese, arabic, hebrew, etc. I used APL professionally every day for ten years. I can read it. I can touch-type it. And I don’t need a specially-labelled keyboard (even thirty years later). This should not be surprising at all. A pianist does not need labeled keys and people familiar enough with the above-listed spoken languages can touch-type them without much effort. While, sadly, APL has no practical application in modern software engineering (it stagnated and became irrelevant and impractical) it is wrong to look at the brilliant use of notation as a tool for the concise communication and expression of ideas and list it as a negative. Not being able to speak, read or write Chinese does not make it a bad language.

0xdeadbeefbabe 5y ago
> Not being able to speak, read or write Chinese does not make it a bad language. Well that's the problem. Not being able to read or write APL makes it more fun to learn.

O_H_E 5y ago

In Julia: https://github.com/shashi/APL.jl https://github.com/shashi/APL.jl

bluenose69 5y ago
Thanks for posting this intriguing link. A glance at the Jupyter notebook on this site will bring a smile to a Julia user who grew up on (or has somehow encountered) APL.
snicker7 5y ago
What an incredibly concise, elegant implementation.

RyanHamilton 5y ago

I've written an open source version of q: http://www.timestored.com/jq/ http://www.timestored.com/jq/ It's implemented in java. You idea is interesting. Allowing q and java intermixed... combining it inside java like linq and C# would be interesting.