6 ms·

The Tokay Programming Language

96 points by wazbug 4y ago

alimov 4y ago
The language looks pretty neat, has anyone had any experience with using it in their personal projects or work setting?
josephg 4y ago
Hey! Cool project. Why would I use this instead of python/javascript/rust/etc? Can you give some examples where Tokay shines compared to these languages? It'd be great to add those examples to the homepage! I had to dig a few levels deep to see what the language looked like. Clicking "Getting started" then scrolling all the way to the bottom seems like a lot of work to answer the question of "why should I care about this?".
- melony 4y ago
  It's a competitor to Awk.
  - keithnz 4y ago
    "Tokay is not intended to be a general purpose programming language, nor a replacement for awk."
    - NotTheDr01ds 4y ago
      Ironically, on the same page, "Tokay is inspired by awk, but follows its own philosophy and design principles. It might also serve as a general purpose scripting language, but it mainly focuses on processing textual input and work on trees with information extracted from this input." And, "If you like awk and sed, you should take a look into Tokay ...." So it "focuses on processing textual input" (sounds like awk), but it's "not intended ... to be a replacement for awk". It "might serve as a general purpose scripting language", but it's not "intended to be a general purpose programming language." If you like awk, you might should ... Oh, I give up!
- HelloNurse 4y ago
  Parsing and transforming text conforming to a rather simple grammar, for example output of typical command line tools, seems the best use case. Practical, non-Hello World examples (challenge: approximating traditional Unix text tools like wc, sort, rev, cut, sed) would be nice.
c_o_n_v_e_x 4y ago
Great name! Love hearing these guys when you get away from the cities.
- rob74 4y ago
  TIL about the Tokay gecko, and that its name is derived from its mating call - plus this little nugget: > The tokay gecko's call is also responsible for the name given to it by U.S. soldiers during the Vietnam War, the "Fuck-you lizard". My first thought was "why did they name it after a wine?" (https://en.wikipedia.org/wiki/Tokaji https://en.wikipedia.org/wiki/Tokaji, sometimes also spelled Tokay)
keithnz 4y ago
Really could do with some "Wow that's cool" examples.
- heroHACK17 4y ago
  Totally. I opened the site and thought "Ok this is cool," but didn't get much further.
- themodelplumber 4y ago
  It's like ack, so there's a good chance we'll hear the best examples anecdotally, in conversations about somebody running their job in Tokay on say, Haiku OS in a forgotten closet space, well pretty soon they just started forgetting to show up for work, and then one day somebody frantically called them and asked if they just crashed the company's stock, and it turns out that a wifi firmware upgrade had inadvertently turned off the closet's internet access. Or something like that.
stewbrew 4y ago
"You can exit the Tokay REPL with Ctrl+C." Other repls often use ctrl-d for end of Input Stream instead. Ctrl-c is rather expected to Interrupt the programme execution, not the repl.
- mananaysiempre 4y ago
  The Lua REPL also terminates on Ctrl-C, which can be infuriating when you’re accustomed to Readline’s interpretation of “abort history editing and clear input”.
  - ihateolives 4y ago
    Ctrl+C for Elixir's iex as well. When i first used it I had to Google, it was that special "how to exit vim" moment for me.
    - darrenf 4y ago
      Perl's re.pl exits on ^C too (as well as ^D). Ditto Clojure.
    - square_usual 4y ago
      The even more annoying thing is that it's Ctrl+C twice.
      - weavie 4y ago
        Clozure common lisp is Ctrl+D three times.
      - klibertp 4y ago
        Well, at least there's a reason for this: Elixir REPL is actually a remote shell, and the first Ctrl+C is an escape sequence that drops you into local connection manager. You of course know this, but I thought I'll leave it here for people who don't :)
        4y ago
ianthehenry 4y ago
I am very interested in this project as a "better awk" is something I have often fantasized about. I read all of the documentation that's available on https://tokay.dev/tokay-docs/ https://tokay.dev/tokay-docs/, but unfortunately it never really... describes itself? Many sections, including the section on "parselets" are just unwritten. "Consumable" values are mentioned but never described (there is a "stub" section that doesn't really explain what the term means). It begins with a pretty detailed description of value "severity" but doesn't really motivate why the concept exists. (I think that it's (basically) a way to very concisely discard certain matches? When there are "more important" matches around them?) There are no examples of how I could use Tokay to "parse" something -- there are lots of examples dotted through the docs, but none of them demonstrate working with structured file formats, and they feel a little bit contrived. I'm not complaining here: this project is not making any false claims about its status, the docs are clearly and explicitly unfinished, it is very clear that Tokay is still under active development. But I want to learn more about it! I came away from this with a sense that it has the potential to be really useful to me, but without any concrete evidence to support that. I guess the next step is to download the source and start reading through the tests. All this to say: please highlight some examples showcasing situations where Tokay shines! (Parsing CSVs containing quoted strings was making the rounds recently, right? What does that look like in Tokay?) Oh, actually, the GitHub readme has an example that is more involved than any in the documentation: https://github.com/tokay-lang/tokay https://github.com/tokay-lang/tokay _ : [ \t]+ # redefine whitespace to just tab and space Factor : @{ Integer _ # built-in 64-bit signed integer token '(' _ Expr ')' _ } Term : @{ Term '*' _ Factor $1 * $4 Term '/' _ Factor $1 / $4 Factor } Expr : @{ Expr '+' _ Term $1 + $4 Expr '-' _ Term $1 - $4 Term } Expr _ print("= " + $1) # gives some neat result output
- rob74 4y ago
  Now you're doing it too ;) You copied the example, but didn't mention why it makes Tokay shine (it's a parser for mathematical expressions supporting the four basic operations and round brackets)
- fjfaase 4y ago
Trufa 4y ago
I'm working on a side project that is slightly related to this, where you can basically use JS to quickly transform text any way you want. It would be really interesting to include many other languages other that JS and this one looks like a great fit (eventually). If anyone want's to give it spin you can but ONLY SOME FEATURES WORK HERE AND I WILL BREAK THE STUFF YOU DO! IT'S NOT EVEN ALPHA O BETA JUST BEING DEVELOPED. https://texttransform.io/ https://texttransform.io/ That said I am interested on HN's point of view of this idea even if very raw, specially people interested in text transformation.
samatman 4y ago
This might be really cool! The idea is solid. I must conclude that Tokay is too immature to use however. If it wasn't it would clearly display, within one click of the home page, the similarities and differences between awk and tokay, because that's what a developer would care about as opposed to anything else, and that feedback will be consistent so it must be very new. Good luck, look forward to seeing it again some day.
HelloNurse 4y ago
Is it good at parsing context sensitive languages, for example XML?
sshine 4y ago
A first impression reading the "Getting started": https://tokay.dev/getting-started/ https://tokay.dev/getting-started/ How does the Tokay executable know that some argument is a string vs. a file? # What if I have a file named "save the wales"? tokay program.tok -- file.txt tokay program.tok -- "save the wales" # What if I have a file named '.+'? tokay program.tok tokay '.+' This feels like someone thought this was neat because you don't need to specify `-e`, but it comes at the cost of ambiguity where your program does something different depending on the filesystem. This has a cost in environments where you're not in full control (e.g. when you distribute your executable for others to use).

96 points by wazbug 4y ago

The language looks pretty neat, has anyone had any experience with using it in their personal projects or work setting?

Hey! Cool project. Why would I use this instead of python/javascript/rust/etc? Can you give some examples where Tokay shines compared to these languages? It'd be great to add those examples to the homepage! I had to dig a few levels deep to see what the language looked like. Clicking "Getting started" then scrolling all the way to the bottom seems like a lot of work to answer the question of "why should I care about this?".

melony 4y ago
It's a competitor to Awk.
- keithnz 4y ago
  "Tokay is not intended to be a general purpose programming language, nor a replacement for awk."
  - NotTheDr01ds 4y ago
    Ironically, on the same page, "Tokay is inspired by awk, but follows its own philosophy and design principles. It might also serve as a general purpose scripting language, but it mainly focuses on processing textual input and work on trees with information extracted from this input." And, "If you like awk and sed, you should take a look into Tokay ...." So it "focuses on processing textual input" (sounds like awk), but it's "not intended ... to be a replacement for awk". It "might serve as a general purpose scripting language", but it's not "intended to be a general purpose programming language." If you like awk, you might should ... Oh, I give up!
HelloNurse 4y ago
Parsing and transforming text conforming to a rather simple grammar, for example output of typical command line tools, seems the best use case. Practical, non-Hello World examples (challenge: approximating traditional Unix text tools like wc, sort, rev, cut, sed) would be nice.

c_o_n_v_e_x 4y ago

Great name! Love hearing these guys when you get away from the cities.

rob74 4y ago
TIL about the Tokay gecko, and that its name is derived from its mating call - plus this little nugget: > The tokay gecko's call is also responsible for the name given to it by U.S. soldiers during the Vietnam War, the "Fuck-you lizard". My first thought was "why did they name it after a wine?" (https://en.wikipedia.org/wiki/Tokaji https://en.wikipedia.org/wiki/Tokaji, sometimes also spelled Tokay)

keithnz 4y ago

Really could do with some "Wow that's cool" examples.

heroHACK17 4y ago
Totally. I opened the site and thought "Ok this is cool," but didn't get much further.
themodelplumber 4y ago
It's like ack, so there's a good chance we'll hear the best examples anecdotally, in conversations about somebody running their job in Tokay on say, Haiku OS in a forgotten closet space, well pretty soon they just started forgetting to show up for work, and then one day somebody frantically called them and asked if they just crashed the company's stock, and it turns out that a wifi firmware upgrade had inadvertently turned off the closet's internet access. Or something like that.

stewbrew 4y ago

"You can exit the Tokay REPL with Ctrl+C." Other repls often use ctrl-d for end of Input Stream instead. Ctrl-c is rather expected to Interrupt the programme execution, not the repl.

mananaysiempre 4y ago
The Lua REPL also terminates on Ctrl-C, which can be infuriating when you’re accustomed to Readline’s interpretation of “abort history editing and clear input”.
- ihateolives 4y ago
  Ctrl+C for Elixir's iex as well. When i first used it I had to Google, it was that special "how to exit vim" moment for me.
  - darrenf 4y ago
    Perl's re.pl exits on ^C too (as well as ^D). Ditto Clojure.
  - square_usual 4y ago
    The even more annoying thing is that it's Ctrl+C twice.
    - weavie 4y ago
      Clozure common lisp is Ctrl+D three times.
    - klibertp 4y ago
      Well, at least there's a reason for this: Elixir REPL is actually a remote shell, and the first Ctrl+C is an escape sequence that drops you into local connection manager. You of course know this, but I thought I'll leave it here for people who don't :)
      - 4y ago

ianthehenry 4y ago

I am very interested in this project as a "better awk" is something I have often fantasized about. I read all of the documentation that's available on https://tokay.dev/tokay-docs/ https://tokay.dev/tokay-docs/, but unfortunately it never really... describes itself? Many sections, including the section on "parselets" are just unwritten. "Consumable" values are mentioned but never described (there is a "stub" section that doesn't really explain what the term means). It begins with a pretty detailed description of value "severity" but doesn't really motivate why the concept exists. (I think that it's (basically) a way to very concisely discard certain matches? When there are "more important" matches around them?) There are no examples of how I could use Tokay to "parse" something -- there are lots of examples dotted through the docs, but none of them demonstrate working with structured file formats, and they feel a little bit contrived. I'm not complaining here: this project is not making any false claims about its status, the docs are clearly and explicitly unfinished, it is very clear that Tokay is still under active development. But I want to learn more about it! I came away from this with a sense that it has the potential to be really useful to me, but without any concrete evidence to support that. I guess the next step is to download the source and start reading through the tests. All this to say: please highlight some examples showcasing situations where Tokay shines! (Parsing CSVs containing quoted strings was making the rounds recently, right? What does that look like in Tokay?) Oh, actually, the GitHub readme has an example that is more involved than any in the documentation: https://github.com/tokay-lang/tokay https://github.com/tokay-lang/tokay _ : [ \t]+ # redefine whitespace to just tab and space Factor : @{ Integer _ # built-in 64-bit signed integer token '(' _ Expr ')' _ } Term : @{ Term '*' _ Factor $1 * $4 Term '/' _ Factor $1 / $4 Factor } Expr : @{ Expr '+' _ Term $1 + $4 Expr '-' _ Term $1 - $4 Term } Expr _ print("= " + $1) # gives some neat result output

rob74 4y ago
Now you're doing it too ;) You copied the example, but didn't mention why it makes Tokay shine (it's a parser for mathematical expressions supporting the four basic operations and round brackets)
fjfaase 4y ago

Trufa 4y ago

I'm working on a side project that is slightly related to this, where you can basically use JS to quickly transform text any way you want. It would be really interesting to include many other languages other that JS and this one looks like a great fit (eventually). If anyone want's to give it spin you can but ONLY SOME FEATURES WORK HERE AND I WILL BREAK THE STUFF YOU DO! IT'S NOT EVEN ALPHA O BETA JUST BEING DEVELOPED. https://texttransform.io/ https://texttransform.io/ That said I am interested on HN's point of view of this idea even if very raw, specially people interested in text transformation.

samatman 4y ago

This might be really cool! The idea is solid. I must conclude that Tokay is too immature to use however. If it wasn't it would clearly display, within one click of the home page, the similarities and differences between awk and tokay, because that's what a developer would care about as opposed to anything else, and that feedback will be consistent so it must be very new. Good luck, look forward to seeing it again some day.

HelloNurse 4y ago

Is it good at parsing context sensitive languages, for example XML?

sshine 4y ago

A first impression reading the "Getting started": https://tokay.dev/getting-started/ https://tokay.dev/getting-started/ How does the Tokay executable know that some argument is a string vs. a file? # What if I have a file named "save the wales"? tokay program.tok -- file.txt tokay program.tok -- "save the wales" # What if I have a file named '.+'? tokay program.tok tokay '.+' This feels like someone thought this was neat because you don't need to specify `-e`, but it comes at the cost of ambiguity where your program does something different depending on the filesystem. This has a cost in environments where you're not in full control (e.g. when you distribute your executable for others to use).