8 ms·

Show HN: A complete Bash parser in JavaScript transpiled from Go

89 points by mvdan 8y ago

mvdan 8y ago
I'm not very familiar with JavaScript and npm - looking for suggestions on how to make the package easier to use.
- crooked-v 8y ago
  Some general thoughts, which may or may not be practical depending on how the transpiling works (I have no idea about go): - Instead of calling .Print(), have the class extend EventEmitter (node builtin, https://github.com/primus/eventemitter3 https://github.com/primus/eventemitter3 is a good browser shim for the same API), with an event that fires once for each line that terminates with a newline. - Hard mode of the above: Add streams functionality (node builtin, https://github.com/nodejs/readable-stream https://github.com/nodejs/readable-stream is an official browser shim for the same API), with output streams that operate on bytes instead of lines. - Combine parser and printer into a single ES6 class (e.g. using `new Parser()` syntax, rather than the indirect object generation). - In optional addition to the above, have .parse() return a Promise for the output (not the parsed program), with parsed programs stored on the Parser itself (and examinable via an additional class method). - Change all method names to be in camelCase instead of PascalCase. Only classes (as instantiated using `new`, not just methods that generate class objects) get PascalCase.
  - mvdan 8y ago
    Thanks for the suggestions! I'll need to read up on these js/node features. One reason that the API is a bit clunky is that it tries to mimic the Go API closely, so that the documentation and examples are reusable. However, your points on the string returns are very valid. I replaced all of Go's readers and writers (byte streams) with strings, simply because I didn't know of a better way. It definitely sounds like the features you suggested would be better. If the transpiler (gopherjs) supports them, I'll definitely give them a try.
    - crooked-v 8y ago
      To help, the short version on "why" for EventEmitter, streams, and Promises is that they're three different ways of doing things asynchronously, with different use cases and details. EventEmitter - You expect to intermittently get individually completed values that you do whatever with and then discard. These values may be grouped into useful sets by the EventEmitter (each separate event type). Streams - You expect to get raw partial data that you want to do your own buffering and handling with. You want to be able to feed this raw data to a different target (e.g. file writing) without handling the fine details yourself. Promise - You expect to get a single completed value when an action is completely finished, or (for a Promise without a return value) you just want to have something contingent on an action finishing. An example of each: Streams: A TelnetConnection class that handles connecting to a server, then supplies a stream with the bytes from the connection (with each chunk being whatever is convenient for buffering purposes), and ends the stream when the connection drops. EventEmitter: A TelnetHandler class that uses TelnetConnection and fires an event for each complete line (ending with \n). Promise: A TelnetHttp class that has a method get(hostname) that uses TelnetHandler and returns a Promise for the full text return value of doing a simple HTTP GET call.
      - mvdan 8y ago
        It sounds like I should use streams then. After all, that's what the Go API uses, Readers and Writers. Those simply pass chunks of bytes around. Thanks again for the help!
- z3t4 8y ago
  Thanks for the package! When it comes to JavaScript - the code is the documentation. You look up methods etc in the source code. But as this is 30k lines of generated code it's not very fun to read in order to figure out how something works. So in this case documentation, tutorials and examples becomes very important! For example how do you find all functions, or how do you find all variables, or all variables available (in scope) from position row/col. I know there are documentation for Go, but that is not super helpful for a JavaScript dev. We need samples ready to copy & paste. Think Stack Overflow (https://stackoverflow.com/ https://stackoverflow.com/) which is often the first result that comes up when you Google for something JavaScript related.
  - mvdan 8y ago
    That's a good point. I am adding more and more examples to the Go documentation these days. Ideally I'd just point the JS people at the Go docs and examples, but the translation is not exactly one-to-one. This is why the README file published with the JS package has a complete example. I'll try to add a few more.
    - z3t4 8y ago
      idea: Maybe the examples and documentation can be transpiled too !? Then you can have several tabs for each code snippet. Go, JavaScript, other. Some transpilations might end up weird, but you could fix those manually.
ClassAndBurn 8y ago
And now my head hurts.
- knodi 8y ago
  Thus is life in JS, perpetual headache.
- mvdan 8y ago
  This is hacky, but it beats having to write a shell parser from scratch again :) Once wasm support is shipped with Go, I hope to be able to accomplish the same without a transpiler. I'd also hope that the final package would be smaller and more performant.
  - JepZ 8y ago
    I didn't even know they were working on wasm for Go, but it looks like there was some progress during the last months: https://github.com/golang/go/issues/18892 https://github.com/golang/go/issues/18892
munificent 8y ago
This headline reads like a Markov chain trained to generate Hacker News posts. I love it.
- mvdan 8y ago
  I hope noone notices that most of my code was written by a Markov chain.
- alistproducer2 8y ago
  Hilarious because it's true. This is my favorite comment of the day.
- tlrobinson 8y ago
  Idea: a hackathon where you're required to implement a project with a title generated by a Markov chain. Bonus points if the code is written by a Markov chain.
  - gargravarr 8y ago
    I think it would still be more readable than half the source I work with on a daily basis...
ddtaylor 8y ago
Not that it's useful or desired, but could one transpile it into pure bash as well? It's Turing complete isn't it?
- mvdan 8y ago
  I'm not sure I understand what you're saying. If there was a Go->Bash transpiler (or a JS->Bash one), yes, we would theoretically end up with a Bash parser and interpreter in Bash. I imagine it would be quite the monster, though.
  - solarkraft 8y ago
    Luckily the current creation isn't already one.
    - mvdan 8y ago
      I'm going to go on assuming that you're not being sarcastic :) As I mentioned in another comment, once Go's wasm support is shipped and stable, I hope to make the JS package smaller and better. Until then, this is the best I can do without manually rewriting the Go code in JS.
monocasa 8y ago
> transpiled I still hate this term.
- mvdan 8y ago
  Is there a more appropriate word for source-to-source transformation? I understand that compilers aren't that different, but generating machine code is quite different.
  - fiddlerwoaroof 8y ago
    Machine code is just source code for a cpu. Anyways, how is a transpiler different from a bytecode compiler, except that the bytecode is human readable?
    - tlrobinson 8y ago
      > how is a transpiler different from a bytecode compiler, except that the bytecode is human readable? The bytecode is human readable.
      - monocasa 8y ago
        The assembly that comes out of GCC is human readable too (the actual assembling occurs via a different program from a different project than GCC). Is GCC a transpiler?
        tlrobinson 8y ago
        No, because assembly is not approximately the same level of abstraction as C.
        monocasa 8y ago
        How do you define that in a way that isn't a tautology?
Zalastax 8y ago
An important heads up: "Parsing Bash is Undecidable". https://www.oilshell.org/blog/2016/10/20.html https://www.oilshell.org/blog/2016/10/20.html
- mvdan 8y ago
  Yep, I've exchanged a few emails with Andy before, the oilshell author. I started the parser in early 2016, so we were both getting started around the same time. If you look at my README's caveats section, you'll see a very similar story - only accepting programs that can be parsed statically in a simple way.
hestefisk 8y ago
Love this. Ironic that it is BSD licensed though, considering it parses and runs the world’s probably most popular piece of GPL’ed software (probably second to Linux).
- mvdan 8y ago
  Most Go code out there is under permissive licenses, I assume because static linking is the norm. I hope I don't get an angry email from RMS at some point :)

89 points by mvdan 8y ago

mvdan 8y ago

I'm not very familiar with JavaScript and npm - looking for suggestions on how to make the package easier to use.

crooked-v 8y ago
Some general thoughts, which may or may not be practical depending on how the transpiling works (I have no idea about go): - Instead of calling .Print(), have the class extend EventEmitter (node builtin, https://github.com/primus/eventemitter3 https://github.com/primus/eventemitter3 is a good browser shim for the same API), with an event that fires once for each line that terminates with a newline. - Hard mode of the above: Add streams functionality (node builtin, https://github.com/nodejs/readable-stream https://github.com/nodejs/readable-stream is an official browser shim for the same API), with output streams that operate on bytes instead of lines. - Combine parser and printer into a single ES6 class (e.g. using `new Parser()` syntax, rather than the indirect object generation). - In optional addition to the above, have .parse() return a Promise for the output (not the parsed program), with parsed programs stored on the Parser itself (and examinable via an additional class method). - Change all method names to be in camelCase instead of PascalCase. Only classes (as instantiated using `new`, not just methods that generate class objects) get PascalCase.
- mvdan 8y ago
  Thanks for the suggestions! I'll need to read up on these js/node features. One reason that the API is a bit clunky is that it tries to mimic the Go API closely, so that the documentation and examples are reusable. However, your points on the string returns are very valid. I replaced all of Go's readers and writers (byte streams) with strings, simply because I didn't know of a better way. It definitely sounds like the features you suggested would be better. If the transpiler (gopherjs) supports them, I'll definitely give them a try.
  - crooked-v 8y ago
    To help, the short version on "why" for EventEmitter, streams, and Promises is that they're three different ways of doing things asynchronously, with different use cases and details. EventEmitter - You expect to intermittently get individually completed values that you do whatever with and then discard. These values may be grouped into useful sets by the EventEmitter (each separate event type). Streams - You expect to get raw partial data that you want to do your own buffering and handling with. You want to be able to feed this raw data to a different target (e.g. file writing) without handling the fine details yourself. Promise - You expect to get a single completed value when an action is completely finished, or (for a Promise without a return value) you just want to have something contingent on an action finishing. An example of each: Streams: A TelnetConnection class that handles connecting to a server, then supplies a stream with the bytes from the connection (with each chunk being whatever is convenient for buffering purposes), and ends the stream when the connection drops. EventEmitter: A TelnetHandler class that uses TelnetConnection and fires an event for each complete line (ending with \n). Promise: A TelnetHttp class that has a method get(hostname) that uses TelnetHandler and returns a Promise for the full text return value of doing a simple HTTP GET call.
    - mvdan 8y ago
      It sounds like I should use streams then. After all, that's what the Go API uses, Readers and Writers. Those simply pass chunks of bytes around. Thanks again for the help!
z3t4 8y ago
Thanks for the package! When it comes to JavaScript - the code is the documentation. You look up methods etc in the source code. But as this is 30k lines of generated code it's not very fun to read in order to figure out how something works. So in this case documentation, tutorials and examples becomes very important! For example how do you find all functions, or how do you find all variables, or all variables available (in scope) from position row/col. I know there are documentation for Go, but that is not super helpful for a JavaScript dev. We need samples ready to copy & paste. Think Stack Overflow (https://stackoverflow.com/ https://stackoverflow.com/) which is often the first result that comes up when you Google for something JavaScript related.
- mvdan 8y ago
  That's a good point. I am adding more and more examples to the Go documentation these days. Ideally I'd just point the JS people at the Go docs and examples, but the translation is not exactly one-to-one. This is why the README file published with the JS package has a complete example. I'll try to add a few more.
  - z3t4 8y ago
    idea: Maybe the examples and documentation can be transpiled too !? Then you can have several tabs for each code snippet. Go, JavaScript, other. Some transpilations might end up weird, but you could fix those manually.

ClassAndBurn 8y ago

And now my head hurts.

knodi 8y ago
Thus is life in JS, perpetual headache.
mvdan 8y ago
This is hacky, but it beats having to write a shell parser from scratch again :) Once wasm support is shipped with Go, I hope to be able to accomplish the same without a transpiler. I'd also hope that the final package would be smaller and more performant.
- JepZ 8y ago
  I didn't even know they were working on wasm for Go, but it looks like there was some progress during the last months: https://github.com/golang/go/issues/18892 https://github.com/golang/go/issues/18892

munificent 8y ago

This headline reads like a Markov chain trained to generate Hacker News posts. I love it.

mvdan 8y ago
I hope noone notices that most of my code was written by a Markov chain.
alistproducer2 8y ago
Hilarious because it's true. This is my favorite comment of the day.
tlrobinson 8y ago
Idea: a hackathon where you're required to implement a project with a title generated by a Markov chain. Bonus points if the code is written by a Markov chain.
- gargravarr 8y ago
  I think it would still be more readable than half the source I work with on a daily basis...

ddtaylor 8y ago

Not that it's useful or desired, but could one transpile it into pure bash as well? It's Turing complete isn't it?

mvdan 8y ago
I'm not sure I understand what you're saying. If there was a Go->Bash transpiler (or a JS->Bash one), yes, we would theoretically end up with a Bash parser and interpreter in Bash. I imagine it would be quite the monster, though.
- solarkraft 8y ago
  Luckily the current creation isn't already one.
  - mvdan 8y ago
    I'm going to go on assuming that you're not being sarcastic :) As I mentioned in another comment, once Go's wasm support is shipped and stable, I hope to make the JS package smaller and better. Until then, this is the best I can do without manually rewriting the Go code in JS.

monocasa 8y ago

> transpiled I still hate this term.

mvdan 8y ago
Is there a more appropriate word for source-to-source transformation? I understand that compilers aren't that different, but generating machine code is quite different.
- fiddlerwoaroof 8y ago
  Machine code is just source code for a cpu. Anyways, how is a transpiler different from a bytecode compiler, except that the bytecode is human readable?
  - tlrobinson 8y ago
    > how is a transpiler different from a bytecode compiler, except that the bytecode is human readable? The bytecode is human readable.
    - monocasa 8y ago
      The assembly that comes out of GCC is human readable too (the actual assembling occurs via a different program from a different project than GCC). Is GCC a transpiler?
      - tlrobinson 8y ago
        No, because assembly is not approximately the same level of abstraction as C.
        monocasa 8y ago
        How do you define that in a way that isn't a tautology?

Zalastax 8y ago

An important heads up: "Parsing Bash is Undecidable". https://www.oilshell.org/blog/2016/10/20.html https://www.oilshell.org/blog/2016/10/20.html

mvdan 8y ago
Yep, I've exchanged a few emails with Andy before, the oilshell author. I started the parser in early 2016, so we were both getting started around the same time. If you look at my README's caveats section, you'll see a very similar story - only accepting programs that can be parsed statically in a simple way.

hestefisk 8y ago

Love this. Ironic that it is BSD licensed though, considering it parses and runs the world’s probably most popular piece of GPL’ed software (probably second to Linux).

mvdan 8y ago
Most Go code out there is under permissive licenses, I assume because static linking is the norm. I hope I don't get an angry email from RMS at some point :)