9 ms·

Neugram – Scripting in Go

236 points by golangnews 9y ago

dom96 9y ago
This is really cool, but I can't help but suggest another language which works incredibly well for scripting: Nim[1]. Of course, I must put a disclaimer: I am incredibly biased as I am of Nim's core developers. But to help convince you, Nim has: * Shebang support * Error handling via exceptions * A somewhat limited REPL * Operator overloading While I applaud the author's efforts, I can't help but wonder if Nim would fit their needs (or be easier to adjust to their needs). 1 - https://nim-lang.org https://nim-lang.org
- crawshaw 9y ago
  Hi, I'm the author. I don't think Neugram is really ready for the full HN treatment, but I'm happy to answer questions. I kept trying to write a "Why Neugram" post and kept getting stuck, which is why I went with this incomplete enumeration of differences from Go. But I get a bit more into my motivation in section 4 of this post today: https://neugram.io/blog/design-principles https://neugram.io/blog/design-principles Basically, I write Go all the time. I occasionally write bash, perl, or python. Occasionally enough, that those languages fall out of my head. What I really want is a scripting language that reuses all my day-to-day knowledge from Go, only with some more scripting language friendliness. I suspect if I programmed in Nim every day, I would use it as a scripting language. Nim is neat.
- civility 9y ago
  Nim has become so good that no one else should develop languages as pet projects? If that's what you're saying (which is how I read it), it's a very obnoxious sentiment.
  - veidr 9y ago
    I took it more like "if you clicked on that, you may also be interested in this" (which I in fact was).
weberc2 9y ago
It seems like something like this would be useful for a Jupyter kernel. I don't think I'd use this as an embedded scripting language; it doesn't offer enough in terms of abstraction and conciseness to justify leaving Go. I'd probably try for a lisp.
- crawshaw 9y ago
  Jupyter is a good idea that was only recently suggested to me: https://github.com/neugram/ng/issues/16 https://github.com/neugram/ng/issues/16 Definitely something to try.
tedunangst 9y ago
It's not clear to me where I would go from this page to try it out, or install it, etc.
- crawshaw 9y ago
  Yes, that sounds like the kind of mistake I would make. I put a tiny link at the bottom of to the home page: https://neugram.io https://neugram.io. I'll do something more elaborate when I'm done with this other bug I'm working on. Thanks.
rileytg 9y ago
this reminds me of the effort to replace a lot of the intel management stuff with linux and have userland all written in go https://schd.ws/hosted_files/osseu17/84/Replace%20UEFI%20with%20Linux.pdf https://schd.ws/hosted_files/osseu17/84/Replace%20UEFI%20wit...
kornish 9y ago
"Go is syntactically a big language. Neugram’s front end has to match it, and that’s a lot of work, much of which is still to do." I appreciate the sentiment – that there's a lot of work left to do – but isn't the assertion about syntax size false? I thought Go's syntax was quite small compared to other popular languages? It only has 25 reserved words (Java has 50+) and was designed in part to be uniform and easy to read.
- crawshaw 9y ago
  I made this point poorly, because I agree with you that Go is quite easy to read and very predictable. With more words: I have written a parser and type checker for an ML-style language, with parametric types and several other neat tricks in it, and I've now written a parser and type checker for a good subset of Go. The latter has been far more work. I am not entirely sure how to explain the work. Go has lots of syntactic and type conveniences that are easy to use and read, but quite difficult to implement. As there are few implementers and many users, I think the original designers of Go picked well when they put the work on us implementers.
  - ansible 9y ago
    What stopped you from using the parsing in the standard library?
    - crawshaw 9y ago
      Because Neugram is a different language. It is so only in small ways, but they add up when you are in the parser, and especially in the type checker. First is the fact that the top level is statements, not declarations. That would mean heavy modification to the go/parser package to make the inner-function statement parser the top level. Second is the fact I need to parse incrementally, to implement a REPL. That would not require serious modification to go/parser, but it would to go/types. Third is I wanted to experiment with new syntax. It was quicker to get to that experimentation by working from scratch rather than adopting go/parser and go/types. Fourth is there are some fundamental things I would like to do differently, particularly around how comments are handled. I haven't got there yet. In retrospect, if the new Go parser inside cmd/compile existed when I started this, I probably would have started from there. (It does many of the things I wanted to do to go/parser.) I suspect I still would have ended up with my own type checker though, if for no other reason than incremental type checking.
nanodano 9y ago
You can already "go run program.go" from anywhere, even outside of a project structure, which is the same result as running an interpreted script. It's a novel idea but I don't really see it taking off. Is there really that much benefit to having a shebang at the top of the file versus executing "go run" and is it really worth having a REPL for a fairly verbose language? I certainly don't see myself wishing for a Java REPL, and I don't particular see myself reaching for a Go REPL either.
- crawshaw 9y ago
  The goal is to add other features that are also useful for scripting. The big one I've worked on so far is syntax for the os/exec package: https://github.com/neugram/ng/blob/master/docs/shell.md#a-shell-session https://github.com/neugram/ng/blob/master/docs/shell.md#a-sh... Another thing I'm interested in is operator overloading, for compact/pleasant matrix/table types.
  - dgacmu 9y ago
    Y'know, if you happened to accidentally create nice bindings for using TensorFlow in the process... :)
wybiral 9y ago
Sometimes switching between Go and C is weird (because of the type order or the semicolons) but I've never had any trouble switching between Go and Python. Go has an awesome ecosystem for writing servers and web applications but it's obvious that the libraries weren't built for scripting purposes and the language is more explicit (for good reasons). Python, on the other hand has an ecosystem that's great for scripting and language features that are more compact and terse.
- crawshaw 9y ago
  If you don't have trouble switching between Go and Python, then the concept of Neugram has much less to offer you than me. Possibly you don't because you use both languages more than me. My scripting needs are approximately once every other week, sometimes even longer. That's enough time for me to forget what python package has the date/time functions I need. I also wouldn't even begin to compare something in its infancy like Neugram with something as well-established as Python's huge collection of libraries. There are many years of work between here and there.
  - wybiral 9y ago
    > Possibly you don't because you use both languages more than me. It depends what the client or project demands. I've worked in Python, Javascript, PHP, VBA, Java, Go, and C (plus a few dialects of SQL). Spending months writing mostly one or two of those at a time. The only ones I enjoy enough to use in my hobby projects are Python, Javascript, and Go. But in my experience the hardest part of toggling between languages like that is the ecosystem (libraries and tools). In this case it seems like you're adding a new ecosystem which means I'd still have to stop and figure out "what's the right package to use for this?" For the most part, language is easy. It's the idioms, libraries, and tools that I find myself googling the most.
    - crawshaw 9y ago
      My goal is very much to reuse Go packages. It is somewhat working today. The underlying mechanism is when you type something like: import "github.com/pkg/errors" A .go wrapper file is generated, a Go plugin is built, and then loaded into the ng executable. You can then use the errors package in the Neugram evaluator. Under the hood the lifting is done by the reflect package. Very much my goal is when scripting, to use the os.Open and ioutil.ReadFile I know, along with filepath.Join, time.Now, and friends.
indescions_2017 9y ago
100% behind you on this, crawshaw. Although I've never missed having a Python style REPL for golang ("go run" usually fits the bill). You make a convincing argument for why it would indeed be a very powerful tool for any Linux admin's toolbox ;) It's premature for feature requests I know. But I'd love to be able to view the Stack Trace of a live running goroutine. Typically something I log to disk using the "runtime" calls. To be able to probe that info from a command shell would be neat, don't you think?
jeffinhat 9y ago
This is awesome. I've been trying to reach for go instead of bash more and more and this is exactly what I need for the small stuff. Any ideas how it could relate to the go toolchain? could I stick my Neugram files in a repo and go get them?
- crawshaw 9y ago
  The relation to the go toolchain right now is your GOPATH is used to find Go packages when you import them. (It uses the go tool to build the package as a plugin for loading.) You certainly could put your Neugram files in the same git repository as your Go code, and then "go get" would get them. I think there is something to be said for something like: import "github.com/crawshaw/foo/mypkg.ng" looking for the .ng file in your GOPATH. I need to think about that a bit more. (Note that a Neugram package is limited to a single file, unlike a Go package. This is for a couple of practical reasons, notably init order, and one philosophical reason: which is Neugram packages shouldn't get as big as Go packages.)
dragonfax 9y ago
I like this. Especially the error handling. Since the focus on a scripting language can be brevity (Ala perl) I don't want to write all that error handling boilerplate. But I don't want to ignore issues either. So this is a compromise. I can check errors, I can capture errors (in _) to ignore them. But if I don't do anything, they turn into panics.
ericfrederich 9y ago
No shebang support? What about this? Sure it's not "native" but it accomplishes the same thing. https://rosettacode.org/wiki/Multiline_shebang#Go https://rosettacode.org/wiki/Multiline_shebang#Go
lox 9y ago
This will be amazing for plugins!
bakul 9y ago
How does this compare with https://github.com/cosmos72/gomacro https://github.com/cosmos72/gomacro which does something very similar? It seems fairly complete, or at least much further along than neugram + it does lisp style macros! If you ignore the macros its language is pretty much go except that main(){ ... } is elided just like in ng. May be worth checking it out, at least for ideas/inspiration.
singularity2001 9y ago
even cleaner python: with open("hello.txt","w"): f.write("Hello, World!")
frou_dh 9y ago
Glad to see this using http://github.com/peterh/liner http://github.com/peterh/liner for interactive line editing. I used that package in an interpreter of my own and it works wonderfully to provide `readline` style keyboard interaction in a pure Go program.
- crawshaw 9y ago
  It really is an excellent package. Does exactly what it says on the box. I fear I may hit a wall with it eventually. For example, several interesting approaches to https://github.com/neugram/ng/issues/49 https://github.com/neugram/ng/issues/49 would step outside what liner can do. But I'm going to use it as long as I can.
  - frou_dh 9y ago
    Among other things, I actually modified liner to stop using/swallowing all control codes, and to instead surface some of them to the client program to decide how to respond. e.g. liner stops doing anything with Ctrl-X (0x18 ?) and instead ng could respond to Ctrl-X by writing a docstring to stderr for an identifier under the caret in the input buffer. https://github.com/frou/liner/blob/master/common.go#L208 https://github.com/frou/liner/blob/master/common.go#L208

236 points by golangnews 9y ago

This is really cool, but I can't help but suggest another language which works incredibly well for scripting: Nim[1]. Of course, I must put a disclaimer: I am incredibly biased as I am of Nim's core developers. But to help convince you, Nim has: * Shebang support * Error handling via exceptions * A somewhat limited REPL * Operator overloading While I applaud the author's efforts, I can't help but wonder if Nim would fit their needs (or be easier to adjust to their needs). 1 - https://nim-lang.org https://nim-lang.org

crawshaw 9y ago
Hi, I'm the author. I don't think Neugram is really ready for the full HN treatment, but I'm happy to answer questions. I kept trying to write a "Why Neugram" post and kept getting stuck, which is why I went with this incomplete enumeration of differences from Go. But I get a bit more into my motivation in section 4 of this post today: https://neugram.io/blog/design-principles https://neugram.io/blog/design-principles Basically, I write Go all the time. I occasionally write bash, perl, or python. Occasionally enough, that those languages fall out of my head. What I really want is a scripting language that reuses all my day-to-day knowledge from Go, only with some more scripting language friendliness. I suspect if I programmed in Nim every day, I would use it as a scripting language. Nim is neat.
civility 9y ago
Nim has become so good that no one else should develop languages as pet projects? If that's what you're saying (which is how I read it), it's a very obnoxious sentiment.
- veidr 9y ago
  I took it more like "if you clicked on that, you may also be interested in this" (which I in fact was).

weberc2 9y ago

It seems like something like this would be useful for a Jupyter kernel. I don't think I'd use this as an embedded scripting language; it doesn't offer enough in terms of abstraction and conciseness to justify leaving Go. I'd probably try for a lisp.

crawshaw 9y ago
Jupyter is a good idea that was only recently suggested to me: https://github.com/neugram/ng/issues/16 https://github.com/neugram/ng/issues/16 Definitely something to try.

tedunangst 9y ago

It's not clear to me where I would go from this page to try it out, or install it, etc.

crawshaw 9y ago
Yes, that sounds like the kind of mistake I would make. I put a tiny link at the bottom of to the home page: https://neugram.io https://neugram.io. I'll do something more elaborate when I'm done with this other bug I'm working on. Thanks.

rileytg 9y ago

this reminds me of the effort to replace a lot of the intel management stuff with linux and have userland all written in go https://schd.ws/hosted_files/osseu17/84/Replace%20UEFI%20with%20Linux.pdf https://schd.ws/hosted_files/osseu17/84/Replace%20UEFI%20wit...

kornish 9y ago

"Go is syntactically a big language. Neugram’s front end has to match it, and that’s a lot of work, much of which is still to do." I appreciate the sentiment – that there's a lot of work left to do – but isn't the assertion about syntax size false? I thought Go's syntax was quite small compared to other popular languages? It only has 25 reserved words (Java has 50+) and was designed in part to be uniform and easy to read.

crawshaw 9y ago
I made this point poorly, because I agree with you that Go is quite easy to read and very predictable. With more words: I have written a parser and type checker for an ML-style language, with parametric types and several other neat tricks in it, and I've now written a parser and type checker for a good subset of Go. The latter has been far more work. I am not entirely sure how to explain the work. Go has lots of syntactic and type conveniences that are easy to use and read, but quite difficult to implement. As there are few implementers and many users, I think the original designers of Go picked well when they put the work on us implementers.
- ansible 9y ago
  What stopped you from using the parsing in the standard library?
  - crawshaw 9y ago
    Because Neugram is a different language. It is so only in small ways, but they add up when you are in the parser, and especially in the type checker. First is the fact that the top level is statements, not declarations. That would mean heavy modification to the go/parser package to make the inner-function statement parser the top level. Second is the fact I need to parse incrementally, to implement a REPL. That would not require serious modification to go/parser, but it would to go/types. Third is I wanted to experiment with new syntax. It was quicker to get to that experimentation by working from scratch rather than adopting go/parser and go/types. Fourth is there are some fundamental things I would like to do differently, particularly around how comments are handled. I haven't got there yet. In retrospect, if the new Go parser inside cmd/compile existed when I started this, I probably would have started from there. (It does many of the things I wanted to do to go/parser.) I suspect I still would have ended up with my own type checker though, if for no other reason than incremental type checking.

nanodano 9y ago

You can already "go run program.go" from anywhere, even outside of a project structure, which is the same result as running an interpreted script. It's a novel idea but I don't really see it taking off. Is there really that much benefit to having a shebang at the top of the file versus executing "go run" and is it really worth having a REPL for a fairly verbose language? I certainly don't see myself wishing for a Java REPL, and I don't particular see myself reaching for a Go REPL either.

crawshaw 9y ago
The goal is to add other features that are also useful for scripting. The big one I've worked on so far is syntax for the os/exec package: https://github.com/neugram/ng/blob/master/docs/shell.md#a-shell-session https://github.com/neugram/ng/blob/master/docs/shell.md#a-sh... Another thing I'm interested in is operator overloading, for compact/pleasant matrix/table types.
- dgacmu 9y ago
  Y'know, if you happened to accidentally create nice bindings for using TensorFlow in the process... :)

wybiral 9y ago

Sometimes switching between Go and C is weird (because of the type order or the semicolons) but I've never had any trouble switching between Go and Python. Go has an awesome ecosystem for writing servers and web applications but it's obvious that the libraries weren't built for scripting purposes and the language is more explicit (for good reasons). Python, on the other hand has an ecosystem that's great for scripting and language features that are more compact and terse.

crawshaw 9y ago
If you don't have trouble switching between Go and Python, then the concept of Neugram has much less to offer you than me. Possibly you don't because you use both languages more than me. My scripting needs are approximately once every other week, sometimes even longer. That's enough time for me to forget what python package has the date/time functions I need. I also wouldn't even begin to compare something in its infancy like Neugram with something as well-established as Python's huge collection of libraries. There are many years of work between here and there.
- wybiral 9y ago
  > Possibly you don't because you use both languages more than me. It depends what the client or project demands. I've worked in Python, Javascript, PHP, VBA, Java, Go, and C (plus a few dialects of SQL). Spending months writing mostly one or two of those at a time. The only ones I enjoy enough to use in my hobby projects are Python, Javascript, and Go. But in my experience the hardest part of toggling between languages like that is the ecosystem (libraries and tools). In this case it seems like you're adding a new ecosystem which means I'd still have to stop and figure out "what's the right package to use for this?" For the most part, language is easy. It's the idioms, libraries, and tools that I find myself googling the most.
  - crawshaw 9y ago
    My goal is very much to reuse Go packages. It is somewhat working today. The underlying mechanism is when you type something like: import "github.com/pkg/errors" A .go wrapper file is generated, a Go plugin is built, and then loaded into the ng executable. You can then use the errors package in the Neugram evaluator. Under the hood the lifting is done by the reflect package. Very much my goal is when scripting, to use the os.Open and ioutil.ReadFile I know, along with filepath.Join, time.Now, and friends.

indescions_2017 9y ago

100% behind you on this, crawshaw. Although I've never missed having a Python style REPL for golang ("go run" usually fits the bill). You make a convincing argument for why it would indeed be a very powerful tool for any Linux admin's toolbox ;) It's premature for feature requests I know. But I'd love to be able to view the Stack Trace of a live running goroutine. Typically something I log to disk using the "runtime" calls. To be able to probe that info from a command shell would be neat, don't you think?

jeffinhat 9y ago

This is awesome. I've been trying to reach for go instead of bash more and more and this is exactly what I need for the small stuff. Any ideas how it could relate to the go toolchain? could I stick my Neugram files in a repo and go get them?

crawshaw 9y ago
The relation to the go toolchain right now is your GOPATH is used to find Go packages when you import them. (It uses the go tool to build the package as a plugin for loading.) You certainly could put your Neugram files in the same git repository as your Go code, and then "go get" would get them. I think there is something to be said for something like: import "github.com/crawshaw/foo/mypkg.ng" looking for the .ng file in your GOPATH. I need to think about that a bit more. (Note that a Neugram package is limited to a single file, unlike a Go package. This is for a couple of practical reasons, notably init order, and one philosophical reason: which is Neugram packages shouldn't get as big as Go packages.)

dragonfax 9y ago

I like this. Especially the error handling. Since the focus on a scripting language can be brevity (Ala perl) I don't want to write all that error handling boilerplate. But I don't want to ignore issues either. So this is a compromise. I can check errors, I can capture errors (in _) to ignore them. But if I don't do anything, they turn into panics.

ericfrederich 9y ago

No shebang support? What about this? Sure it's not "native" but it accomplishes the same thing. https://rosettacode.org/wiki/Multiline_shebang#Go https://rosettacode.org/wiki/Multiline_shebang#Go

lox 9y ago

This will be amazing for plugins!

bakul 9y ago

How does this compare with https://github.com/cosmos72/gomacro https://github.com/cosmos72/gomacro which does something very similar? It seems fairly complete, or at least much further along than neugram + it does lisp style macros! If you ignore the macros its language is pretty much go except that main(){ ... } is elided just like in ng. May be worth checking it out, at least for ideas/inspiration.

singularity2001 9y ago

even cleaner python: with open("hello.txt","w"): f.write("Hello, World!")

frou_dh 9y ago

Glad to see this using http://github.com/peterh/liner http://github.com/peterh/liner for interactive line editing. I used that package in an interpreter of my own and it works wonderfully to provide `readline` style keyboard interaction in a pure Go program.

crawshaw 9y ago
It really is an excellent package. Does exactly what it says on the box. I fear I may hit a wall with it eventually. For example, several interesting approaches to https://github.com/neugram/ng/issues/49 https://github.com/neugram/ng/issues/49 would step outside what liner can do. But I'm going to use it as long as I can.
- frou_dh 9y ago
  Among other things, I actually modified liner to stop using/swallowing all control codes, and to instead surface some of them to the client program to decide how to respond. e.g. liner stops doing anything with Ctrl-X (0x18 ?) and instead ng could respond to Ctrl-X by writing a docstring to stderr for an identifier under the caret in the input buffer. https://github.com/frou/liner/blob/master/common.go#L208 https://github.com/frou/liner/blob/master/common.go#L208