9 ms·

Scripting with Go

243 points by synergy20 5y ago

synergy20 5y ago
https://golangexample.com/making-it-easy-to-write-shell-like-scripts-in-golang/ https://golangexample.com/making-it-easy-to-write-shell-like... with quick examples, everything is pipe based just like bash|fish|zsh|csh
- 0xbadcafebee 5y ago
  Go script: p := Exec("man bogus") p.SetError(nil) output, err := p.String() fmt.Println(output) Shell script: man bogus ......I'm gonna stick with shell scripts.
mbreese 5y ago
If you’re willing to add a new binfmt to your system, here’s another method for using golang to write “scripts” (executed with gorun). It’s not the same as a shebang/hashbang, but it works. https://blog.cloudflare.com/using-go-as-a-scripting-language-in-linux/ https://blog.cloudflare.com/using-go-as-a-scripting-language...
mkdirp 5y ago
I'll have to check this out properly later. 10 years ago go devs rejected the idea of introducing support hashbangs and now we're left with having to use gorun[0], meaning, people are unlikely to use go for script. I hope the go devs will reconsider. I'd love to be able to use go for scripting. But as it stands, it's a sad state of affairs because you have to rely on hacks. [0] https://github.com/erning/gorun/ https://github.com/erning/gorun/
- throwaway894345 5y ago
  I'm not sure what definition you have for "scripts", but I'm fine with running `go run main.go` or whatever. I don't need my Go files to be executable (half the time I just run bash scripts via `bash script.sh` anyway because that always works). The biggest impediment is the ceremony for subprocessing out, but I'm sure that could be abstracted behind a library with a more pleasant veneer than os/exec, and even if not whatever you lose in subprocess ergonomics you make up for in static typing, not needing to Google/Stack Overflow everything, and general absence of weird quirks.
  - mkdirp 5y ago
    I mostly use executable scripts so I can put it in a dir in my PATH. That way I can just run globally. No need to know where it is, or how to run it. > The biggest impediment is the ceremony for subprocessing out Not sure I fully understand what you mean with this, but if I'm understanding it right, gorun attempts to solve this by doing `syscall.Exec()` on the compiled binary. Combined with slight variation on the "comment" hack (see examples in gorun's readme), signals get fully directed your binary. // 2>/dev/null ; exec gorun "$0" "$@" package main // rest of the code
  - lenkite 5y ago
    What about dependencies and libraries ? For tech that works very well for self-contained scripts, take a look at Deno. Supports she-bang. Also, you can have self-contained script with imports pointed to URL's or file system paths that are automatically resolved. This leads to easy management, maintenance and distribution.
- moondev 5y ago
  you can now directly "go run" any import path go run github.com/mikefarah/yq/v3@3.4.1 It also works fine in shebang that expects input of script filepath at $1. For example "test.yaml" #!/usr/bin/env -S go run github.com/mikefarah/yq/v3@3.4.1 r --- some: yaml: here chmod +x test.yaml then ./test.yaml some.yaml returns test
flakiness 5y ago
(Mostly trolling) You must be missing extension method that other modern languages have, let alone the pipe operator of Elixir, R, etc. ;-)
- jerf 5y ago
  The places I've "shell scripted with Go" involved also using some other code I already had in Go. The referenced page doesn't show it, but being able to do a bit of quick shell scripting-type manipulation on a file, then feed it to the Go JSON decoder to get Go objects, call a few methods or something, then maybe manipulate it a bit more on the way out, is sort of thing that is the real killer feature, IMHO. The space of "shell-like libraries", not least of which is shell itself, is so rich it's hard to beat out everything else on its own merits. But having something as easy as shell that integrates back into your Go ecosystem is very convenient. And I'm sure similar things are true for the other libraries in other languages. So I would personally not present this as "here's something awesome Go can uniquely do", but, "if you already have Go, be aware of this tool/technique".
  - flakiness 5y ago
    Yeah, I agree with your point and can see the usefulness. Sorry for trolling, just couldn't resist :-/
chubot 5y ago
FWIW a lot of other host languages are more DSL-like, and have similar libraries: https://github.com/oilshell/oil/wiki/Internal-DSLs-for-Shell https://github.com/oilshell/oil/wiki/Internal-DSLs-for-Shell It looks like this particular one relies on method chaining script.Args().Concat().Match("Error").First(10).Stdout()
kkfx 5y ago
Honestly I'm not convinced. I understand the need of a so-called "real" programming language ready available, that's what all classic system have had, from SmallTalk workstation to LispM, but for them there is not just the language, there is the complete ecosystem build with the same language, so an user made script have no difference than a system part of the user desktop. Unix decide to "simply" creating a system with a system language and an user system with an user language, the shell. I do not like much unix approach after having used Emacs a bit, but I do understand it. On contrary I always fails to "craft scripts" in "real" programming languages no matter what. I've tried in the past to "go Perl", "go Python", "go TCL", yes I can write scripts with them, I have written some etc but if I need something quick I go for the shell, zsh to be more precise (tried others, all failed at a certain point, from bash to elvish passing through oil shell and few others) or being in Emacs (EXWM is my WM/DE) Emacs itself depending on the case. I read various similar article for a language or another but in the long run see no colleagues really choose a programming language behind shell itself...
shimst3r 5y ago
While I see the benefit of this approach, I’m often baffled why people want to go either 100 % POSIX builtins or 100 % scripting language. The biggest benefit of the shell is its clearly defined input and output (and error) interfaces. Most programming languages can read from and write to stdin, stdout, and stderr. Why not use it and stick to KISS, replacing one cumbersome POSIX utility at a time, suites for the task? Then you don’t need to chain methods using less idiomatic code. But then you wouldn’t need these kind of libraries either.
- hnlmorg 5y ago
  I completely agree. This is the approach that I took with murex. It comes with a stack of builtins for json, jsonlines, yaml, toml, csv, sqlite3, and others. So you have all the power of data types and intelligent management of structured data but also works with regular POSIX CLI commands without any additional boilerplate code when swapping between murex code and coreutils (et al). So one could say I’ve spent a fair amount of time studying this point you’ve raised :)
  - cyberge99 5y ago
    Is murex open source? A quick search of murex shell presented me with beautiful seashells.
    - zto 5y ago
      A quick Google for murex cli -sea yields: https://murex.rocks https://murex.rocks and https://github.com/lmorg/murex https://github.com/lmorg/murex, looks like a promising tool
      - darrenf 5y ago
        Not to mention the URL in @hnlmorg’s HN profile!
  - klabb3 5y ago
    https://github.com/lmorg/murex https://github.com/lmorg/murex (Don't be afraid to self promote!) One obvious benefit over what's suggested in the article is that you can use it interactively first, with autocomplete and such goodies, and transition smoothly to a script later.
hankchinaski 5y ago
i am not sure i would personally use this library/approach when scripting, a few reasons: Using a third-party libraries as core part of your infrastructure (ci/cd scripts, provisioning, automation) implies a greater risk to potential security issues, "framework tax" ie. having to comply, learn, document, debug its custom APIs, having to deal with potential limitations, issues that either needs to be fixed upstream or result in the library being forked and therefore maintained in house. I would rather either put together a set of bash commands or - if the problem entails a more comprehensive endeavour with greater complexity - put together a in-house tool/library where i can make the right compromises from day one
mftb 5y ago
It took me a while to find the link to the library "script" and it's repo - https://github.com/bitfield/script https://github.com/bitfield/script
jniedrauer 5y ago
I may be in the minority here, but I personally prefer the "wrong answer" highlighted at the beginning of this article. Scripts are source code. They go in the repo, and they have the same standards applied to them as any other source code. I would much prefer that the code be explicit and rely on few if any third party libraries. I have go scripts that have been functioning in production for half a decade now without modification. They are as "self documenting" as any other go code, and I do not require esoteric knowledge about a third party library to re-familiarize myself with them.
- _adamb 5y ago
  I agree. It would be much more readable if it simply had a few comments too (something that can't really exist in a 1-liner) and is infinitely flexible (ex: for each match, also make an API call).
- fhood 5y ago
  I think you are missing the point. The overarching motivation here is "Normally one would do this in bash, but I want to use go instead. Is there any way for me to do that while retaining the aspects of bash that make it good for these tasks?" And the author provides a solution. Dunno why you would do that, but not really important.
vessenes 5y ago
I like me some bash scripting. And I do a lot of it. And I write a good amount of Go. I read the intent of the original package as a bit different than how it’s presented here; a pain point in using go for systems programming on Posix systems with GNU tooling is interacting with all the other really excellent tools there. There’s an awful lot of cruft to go spawn a shell script in Go without this lib. So, I cannot imagine using this to replace a short-ish bash script, it’s just too heavyweight to add go development into the workflow. But, I can easily imagine using this when I want to do some simple-to-medium-scale pipelined text processing from the middle of a go program.
- unfocussed_mike 5y ago
  > But, I can easily imagine using this when I want to do some simple-to-medium-scale pipelined text processing from the middle of a go program. Yeah, this is where I see it. I have a cross-platform utility in customer use that replaces a bash script or a .bat file that did a simple job with curl and a database client but was tricky for customers to configure and deploy; I replaced it with an interactively-configurable go command. In the middle of it is the replacement pipe; this library would help here. Might use it in the rewrite.
- GordonS 5y ago
  Thing is, bash is everywhere (or, at least a POSIX shell is) - for some use cases that matters, because you can't control what is available on the target machine. Even python isn't guaranteed - OK python is fairly ubiquitous, but what version? Shell scripting is far from perfect, and sometimes it does take a while to figure out how to do something that would be easy in another language - but very often, she'll scripting is "good enough".
  - oweiler 5y ago
    Bash yes, but commands like sed may or may not support certain flags depending in the platform. https://stackoverflow.com/questions/5694228/sed-in-place-flag-that-works-both-on-mac-bsd-and-linux https://stackoverflow.com/questions/5694228/sed-in-place-fla...
  - 5y ago
nunez 5y ago
There was a recent discussion in /r/golang about chaining methods like this. The OP had a hard time writing tests for their implementation since each part of the chain returned interfaces that were relied on by following links within. The top answer in the thread was that Golang wasn't really meant to make "Promise-" type workflows possible due to it leaning heavily on code being intentional and explicit. I agree with them. One of the problems I have with things like LINQ or Promises is that they are immensely powerful but can be really difficult to debug when things go wrong because of implicit errors within the pipeline bubble up the stack without providing location context. While the standard boilerplate around errors in Go is annoying, I actually prefer it for two reasons: 1. It provides exactly where things went wrong while looking at an exception trace, and 2. From a readability perspective, it is much easier to understand what the author of the code meant to do and, more importantly, what the follow-on error s mean. Sometimes, errors aren't actually errors. Either way, OCaml/Haskell/F#'s approach with match expressions and the `Result` type is the best way to deal with this, IMO. You get the best of both worlds: explicit declaration with a very expressive (triangular-like) "shape" to the code. So something like this: var data string = script.File("test.txt").Match("Error").CountLines() Can be expressed like this: var foundLines int32 = 0 var numLines int32 = match file.Open("test.txt") with | Error e -> return e | Ok f -> match f.Readlines() with | Error e -> return e | Ok lines -> match lines with | /Error.*/ -> foundLines++ | _ -> // do nothign return foundLines instead of: foundLines := 0 lines, err := ioutil.ReadFile("test.txt") if err != nil { return 0, err } for _, l := range lines { re, err := regexp.MustCompile(`Error.*`) if err != nil { return 0, err } matches := re.FindStringSubmatch(`Error.*`) if len(matches) > 0 { foundLines++ } } return foundLines, nil This way, you can see exactly where the error occurred but can still see that `numLines` is generated through a pipeline. As far as the library itself, I personally wouldn't use something like this, though I see the appeal. I turn to statically-typed languages like Golang when a Bash script becomes too kludgey to stand on its own (usually when I need to begin relying on mild data structures) and when I care about type safety and portability more than what Ruby or Python can give me. When I'm writing a Go program, I want something that's testable that I know can work anywhere. With Ruby or Python, unless I'm distributing the script as a Docker image, I have to worry about versions, environments, etc., none of which are pleasant. However, writing the error boilerplate is annoying, and I can see developers who want to write something quick but spend 99% of their time in Go using this to get something done with a tool they know well. I've seen similar things in the Java world; hell, that's the reason why Groovy and Kotlin exist! TL;DR: The "wrong thing" in the Bitfield article isn't necessarily wrong; their library is useful, but niche; all hail pattern-matching and the Result type.
qbasic_forever 5y ago
It's interesting that go text templates have shell-like pipelines but that never bled over into the rest of the language: https://pkg.go.dev/text/template#hdr-Pipelines https://pkg.go.dev/text/template#hdr-Pipelines It seems like a shame that this kind of power and expressiveness is reserved only for generating text.
gkfasdfasdf 5y ago
Something which the shell script has which the go implementation lacks (unless I missed it) is concurrency. When you have a shell pipeline like 'cmd1 | cmd2 | cmd3' those cmds are actually started at the same time and run in parallel. This is really great for performance - you get multiple cores working on the problem with very little effort.
shadowofneptune 5y ago
This article focused entirely on replacing the Unix shell. Does this library also work for the Windows shell or PowerShell (iirc PowerShell is POSIX)? I could see the value of having a single script in a Go repository which works for all build systems.
- Intermernet 5y ago
  The programs should be OS agnostic as I assume that the majority of the functionality is based on the go standard library. Haven't tested this, but I've written heaps of go code for Windows and Linux and have rarely had to add OS specific info. There are a few small differences, but it's easy to abstract them away, which I assume has been done in this case.
skybrian 5y ago
This Go library looks like a good one, with the caveats that it hasn't reached 1.0 yet, and does have a couple of dependencies that I didn't trace further. At one time jQuery was popular and this seems like a similar thing, but for files? It's small enough that if you're worried about "other people's code" you could fork it and maintain it yourself.
tedunangst 5y ago
Seems like this could pair well with something like yaegi.
hobobaggins 5y ago
It would be nice if Go could do this sort of chaining like jQuery makes so easy with Javascript: n, err := object .Process1(arg1, arg2) .Process2(true) .Process3("foo") That would make these sorts of chains much easier to read.
- svnpenn 5y ago
  You can do that now, just move the dots to end of line https://github.com/89z/googleplay/blob/v1.8.1/header.go#L126 https://github.com/89z/googleplay/blob/v1.8.1/header.go#L126
kodah 5y ago
While using the package seems nice I think what's missed here is that Go requires a module declaration and dependencies to be enumerated. That would mix your "scripts" with any production code written in Go. Bash and Python keep my scripts away from my app dependencies.
emeraldd 5y ago
I'm having a very hard time getting past the fact that the "wrong" code is wrapped very badly on both a mobile device and laptop. That inherently makes it harder to read ...
- ikiris 5y ago
  They had to really struggle to make that code not clearly readable, and the complete lack of any error handling and dependence on some random framework somehow more desirable to the reader in the second example.
bobnamob 5y ago
May be derailing discussion a little, but the nicest scripting language/environment I've ever used is Clojure's babashka[1]. Clojure's data first model + conveniences like the -> and ->> macros make simple data pipeline scripts a joy to write. babashka of course suffers from the "it isn't everywhere" problem that is mentioned regarding python. https://github.com/babashka/babashka https://github.com/babashka/babashka
simonw 5y ago
This is done absolutely superb API design - taking inspiration from shell pipes and turning it into a Go library with syntax this clean is really impressive: https://github.com/bitfield/script https://github.com/bitfield/script
jsz0 5y ago
I felt really fancy about a decade ago learning python and adopting it for even the most trivial tasks. Then I realized at least half the stuff I wrote doesn't even run now without modification. Meanwhile bash scripts I wrote that predated that all still work perfectly. Now I'm back on the bash big time.
mistrial9 5y ago
this is personally challenging to me, and gets me out of my comfort zone -- in the right ways. Nobody needs to care about my own learning trajectory for server-side software, but, I will say that I have been hoarding 'good' bash scripts for a bunch of years, after having a bad start with Perl for server-side things. Bash needs no comments here, but I use it everyday. I avoided GO-lang partly due to an insider vibe from Xooglers, and partly internal prioritization. This article strikes that nerve today. thanks for posting .. no promises but bookmarked.
birchb 5y ago
I had a look at this library today. Quote the author: "The script library is implemented entirely in Go, and does not require any external userland programs." Sorry but that's not scripting anymore. We all like using userland programs! it is The Way. Most programs tell you why they failed on stderr. Seems like stderr is lost when a pipe component fails. Strangely stderr and stdout are conflated in the pipe structure - there is no way to get stderr! You just get the numeric exit status. IO redirection appears to be missing. Difficult to use in production.

243 points by synergy20 5y ago

https://golangexample.com/making-it-easy-to-write-shell-like-scripts-in-golang/ https://golangexample.com/making-it-easy-to-write-shell-like... with quick examples, everything is pipe based just like bash|fish|zsh|csh

0xbadcafebee 5y ago
Go script: p := Exec("man bogus") p.SetError(nil) output, err := p.String() fmt.Println(output) Shell script: man bogus ......I'm gonna stick with shell scripts.

mbreese 5y ago

If you’re willing to add a new binfmt to your system, here’s another method for using golang to write “scripts” (executed with gorun). It’s not the same as a shebang/hashbang, but it works. https://blog.cloudflare.com/using-go-as-a-scripting-language-in-linux/ https://blog.cloudflare.com/using-go-as-a-scripting-language...

mkdirp 5y ago

I'll have to check this out properly later. 10 years ago go devs rejected the idea of introducing support hashbangs and now we're left with having to use gorun[0], meaning, people are unlikely to use go for script. I hope the go devs will reconsider. I'd love to be able to use go for scripting. But as it stands, it's a sad state of affairs because you have to rely on hacks. [0] https://github.com/erning/gorun/ https://github.com/erning/gorun/

throwaway894345 5y ago
I'm not sure what definition you have for "scripts", but I'm fine with running `go run main.go` or whatever. I don't need my Go files to be executable (half the time I just run bash scripts via `bash script.sh` anyway because that always works). The biggest impediment is the ceremony for subprocessing out, but I'm sure that could be abstracted behind a library with a more pleasant veneer than os/exec, and even if not whatever you lose in subprocess ergonomics you make up for in static typing, not needing to Google/Stack Overflow everything, and general absence of weird quirks.
- mkdirp 5y ago
  I mostly use executable scripts so I can put it in a dir in my PATH. That way I can just run globally. No need to know where it is, or how to run it. > The biggest impediment is the ceremony for subprocessing out Not sure I fully understand what you mean with this, but if I'm understanding it right, gorun attempts to solve this by doing `syscall.Exec()` on the compiled binary. Combined with slight variation on the "comment" hack (see examples in gorun's readme), signals get fully directed your binary. // 2>/dev/null ; exec gorun "$0" "$@" package main // rest of the code
- lenkite 5y ago
  What about dependencies and libraries ? For tech that works very well for self-contained scripts, take a look at Deno. Supports she-bang. Also, you can have self-contained script with imports pointed to URL's or file system paths that are automatically resolved. This leads to easy management, maintenance and distribution.
moondev 5y ago
you can now directly "go run" any import path go run github.com/mikefarah/yq/v3@3.4.1 It also works fine in shebang that expects input of script filepath at $1. For example "test.yaml" #!/usr/bin/env -S go run github.com/mikefarah/yq/v3@3.4.1 r --- some: yaml: here chmod +x test.yaml then ./test.yaml some.yaml returns test

flakiness 5y ago

(Mostly trolling) You must be missing extension method that other modern languages have, let alone the pipe operator of Elixir, R, etc. ;-)

jerf 5y ago
The places I've "shell scripted with Go" involved also using some other code I already had in Go. The referenced page doesn't show it, but being able to do a bit of quick shell scripting-type manipulation on a file, then feed it to the Go JSON decoder to get Go objects, call a few methods or something, then maybe manipulate it a bit more on the way out, is sort of thing that is the real killer feature, IMHO. The space of "shell-like libraries", not least of which is shell itself, is so rich it's hard to beat out everything else on its own merits. But having something as easy as shell that integrates back into your Go ecosystem is very convenient. And I'm sure similar things are true for the other libraries in other languages. So I would personally not present this as "here's something awesome Go can uniquely do", but, "if you already have Go, be aware of this tool/technique".
- flakiness 5y ago
  Yeah, I agree with your point and can see the usefulness. Sorry for trolling, just couldn't resist :-/

chubot 5y ago

FWIW a lot of other host languages are more DSL-like, and have similar libraries: https://github.com/oilshell/oil/wiki/Internal-DSLs-for-Shell https://github.com/oilshell/oil/wiki/Internal-DSLs-for-Shell It looks like this particular one relies on method chaining script.Args().Concat().Match("Error").First(10).Stdout()

kkfx 5y ago

Honestly I'm not convinced. I understand the need of a so-called "real" programming language ready available, that's what all classic system have had, from SmallTalk workstation to LispM, but for them there is not just the language, there is the complete ecosystem build with the same language, so an user made script have no difference than a system part of the user desktop. Unix decide to "simply" creating a system with a system language and an user system with an user language, the shell. I do not like much unix approach after having used Emacs a bit, but I do understand it. On contrary I always fails to "craft scripts" in "real" programming languages no matter what. I've tried in the past to "go Perl", "go Python", "go TCL", yes I can write scripts with them, I have written some etc but if I need something quick I go for the shell, zsh to be more precise (tried others, all failed at a certain point, from bash to elvish passing through oil shell and few others) or being in Emacs (EXWM is my WM/DE) Emacs itself depending on the case. I read various similar article for a language or another but in the long run see no colleagues really choose a programming language behind shell itself...

shimst3r 5y ago

While I see the benefit of this approach, I’m often baffled why people want to go either 100 % POSIX builtins or 100 % scripting language. The biggest benefit of the shell is its clearly defined input and output (and error) interfaces. Most programming languages can read from and write to stdin, stdout, and stderr. Why not use it and stick to KISS, replacing one cumbersome POSIX utility at a time, suites for the task? Then you don’t need to chain methods using less idiomatic code. But then you wouldn’t need these kind of libraries either.

hnlmorg 5y ago
I completely agree. This is the approach that I took with murex. It comes with a stack of builtins for json, jsonlines, yaml, toml, csv, sqlite3, and others. So you have all the power of data types and intelligent management of structured data but also works with regular POSIX CLI commands without any additional boilerplate code when swapping between murex code and coreutils (et al). So one could say I’ve spent a fair amount of time studying this point you’ve raised :)
- cyberge99 5y ago
  Is murex open source? A quick search of murex shell presented me with beautiful seashells.
  - zto 5y ago
    A quick Google for murex cli -sea yields: https://murex.rocks https://murex.rocks and https://github.com/lmorg/murex https://github.com/lmorg/murex, looks like a promising tool
    - darrenf 5y ago
      Not to mention the URL in @hnlmorg’s HN profile!
- klabb3 5y ago
  https://github.com/lmorg/murex https://github.com/lmorg/murex (Don't be afraid to self promote!) One obvious benefit over what's suggested in the article is that you can use it interactively first, with autocomplete and such goodies, and transition smoothly to a script later.

hankchinaski 5y ago

i am not sure i would personally use this library/approach when scripting, a few reasons: Using a third-party libraries as core part of your infrastructure (ci/cd scripts, provisioning, automation) implies a greater risk to potential security issues, "framework tax" ie. having to comply, learn, document, debug its custom APIs, having to deal with potential limitations, issues that either needs to be fixed upstream or result in the library being forked and therefore maintained in house. I would rather either put together a set of bash commands or - if the problem entails a more comprehensive endeavour with greater complexity - put together a in-house tool/library where i can make the right compromises from day one

mftb 5y ago

It took me a while to find the link to the library "script" and it's repo - https://github.com/bitfield/script https://github.com/bitfield/script

jniedrauer 5y ago

I may be in the minority here, but I personally prefer the "wrong answer" highlighted at the beginning of this article. Scripts are source code. They go in the repo, and they have the same standards applied to them as any other source code. I would much prefer that the code be explicit and rely on few if any third party libraries. I have go scripts that have been functioning in production for half a decade now without modification. They are as "self documenting" as any other go code, and I do not require esoteric knowledge about a third party library to re-familiarize myself with them.

_adamb 5y ago
I agree. It would be much more readable if it simply had a few comments too (something that can't really exist in a 1-liner) and is infinitely flexible (ex: for each match, also make an API call).
fhood 5y ago
I think you are missing the point. The overarching motivation here is "Normally one would do this in bash, but I want to use go instead. Is there any way for me to do that while retaining the aspects of bash that make it good for these tasks?" And the author provides a solution. Dunno why you would do that, but not really important.

vessenes 5y ago

I like me some bash scripting. And I do a lot of it. And I write a good amount of Go. I read the intent of the original package as a bit different than how it’s presented here; a pain point in using go for systems programming on Posix systems with GNU tooling is interacting with all the other really excellent tools there. There’s an awful lot of cruft to go spawn a shell script in Go without this lib. So, I cannot imagine using this to replace a short-ish bash script, it’s just too heavyweight to add go development into the workflow. But, I can easily imagine using this when I want to do some simple-to-medium-scale pipelined text processing from the middle of a go program.

unfocussed_mike 5y ago
> But, I can easily imagine using this when I want to do some simple-to-medium-scale pipelined text processing from the middle of a go program. Yeah, this is where I see it. I have a cross-platform utility in customer use that replaces a bash script or a .bat file that did a simple job with curl and a database client but was tricky for customers to configure and deploy; I replaced it with an interactively-configurable go command. In the middle of it is the replacement pipe; this library would help here. Might use it in the rewrite.
GordonS 5y ago
Thing is, bash is everywhere (or, at least a POSIX shell is) - for some use cases that matters, because you can't control what is available on the target machine. Even python isn't guaranteed - OK python is fairly ubiquitous, but what version? Shell scripting is far from perfect, and sometimes it does take a while to figure out how to do something that would be easy in another language - but very often, she'll scripting is "good enough".
- oweiler 5y ago
  Bash yes, but commands like sed may or may not support certain flags depending in the platform. https://stackoverflow.com/questions/5694228/sed-in-place-flag-that-works-both-on-mac-bsd-and-linux https://stackoverflow.com/questions/5694228/sed-in-place-fla...
- 5y ago

nunez 5y ago

There was a recent discussion in /r/golang about chaining methods like this. The OP had a hard time writing tests for their implementation since each part of the chain returned interfaces that were relied on by following links within. The top answer in the thread was that Golang wasn't really meant to make "Promise-" type workflows possible due to it leaning heavily on code being intentional and explicit. I agree with them. One of the problems I have with things like LINQ or Promises is that they are immensely powerful but can be really difficult to debug when things go wrong because of implicit errors within the pipeline bubble up the stack without providing location context. While the standard boilerplate around errors in Go is annoying, I actually prefer it for two reasons: 1. It provides exactly where things went wrong while looking at an exception trace, and 2. From a readability perspective, it is much easier to understand what the author of the code meant to do and, more importantly, what the follow-on error s mean. Sometimes, errors aren't actually errors. Either way, OCaml/Haskell/F#'s approach with match expressions and the `Result` type is the best way to deal with this, IMO. You get the best of both worlds: explicit declaration with a very expressive (triangular-like) "shape" to the code. So something like this: var data string = script.File("test.txt").Match("Error").CountLines() Can be expressed like this: var foundLines int32 = 0 var numLines int32 = match file.Open("test.txt") with | Error e -> return e | Ok f -> match f.Readlines() with | Error e -> return e | Ok lines -> match lines with | /Error.*/ -> foundLines++ | _ -> // do nothign return foundLines instead of: foundLines := 0 lines, err := ioutil.ReadFile("test.txt") if err != nil { return 0, err } for _, l := range lines { re, err := regexp.MustCompile(`Error.*`) if err != nil { return 0, err } matches := re.FindStringSubmatch(`Error.*`) if len(matches) > 0 { foundLines++ } } return foundLines, nil This way, you can see exactly where the error occurred but can still see that `numLines` is generated through a pipeline. As far as the library itself, I personally wouldn't use something like this, though I see the appeal. I turn to statically-typed languages like Golang when a Bash script becomes too kludgey to stand on its own (usually when I need to begin relying on mild data structures) and when I care about type safety and portability more than what Ruby or Python can give me. When I'm writing a Go program, I want something that's testable that I know can work anywhere. With Ruby or Python, unless I'm distributing the script as a Docker image, I have to worry about versions, environments, etc., none of which are pleasant. However, writing the error boilerplate is annoying, and I can see developers who want to write something quick but spend 99% of their time in Go using this to get something done with a tool they know well. I've seen similar things in the Java world; hell, that's the reason why Groovy and Kotlin exist! TL;DR: The "wrong thing" in the Bitfield article isn't necessarily wrong; their library is useful, but niche; all hail pattern-matching and the Result type.

qbasic_forever 5y ago

It's interesting that go text templates have shell-like pipelines but that never bled over into the rest of the language: https://pkg.go.dev/text/template#hdr-Pipelines https://pkg.go.dev/text/template#hdr-Pipelines It seems like a shame that this kind of power and expressiveness is reserved only for generating text.

gkfasdfasdf 5y ago

Something which the shell script has which the go implementation lacks (unless I missed it) is concurrency. When you have a shell pipeline like 'cmd1 | cmd2 | cmd3' those cmds are actually started at the same time and run in parallel. This is really great for performance - you get multiple cores working on the problem with very little effort.

shadowofneptune 5y ago

This article focused entirely on replacing the Unix shell. Does this library also work for the Windows shell or PowerShell (iirc PowerShell is POSIX)? I could see the value of having a single script in a Go repository which works for all build systems.

Intermernet 5y ago
The programs should be OS agnostic as I assume that the majority of the functionality is based on the go standard library. Haven't tested this, but I've written heaps of go code for Windows and Linux and have rarely had to add OS specific info. There are a few small differences, but it's easy to abstract them away, which I assume has been done in this case.

skybrian 5y ago

This Go library looks like a good one, with the caveats that it hasn't reached 1.0 yet, and does have a couple of dependencies that I didn't trace further. At one time jQuery was popular and this seems like a similar thing, but for files? It's small enough that if you're worried about "other people's code" you could fork it and maintain it yourself.

tedunangst 5y ago

Seems like this could pair well with something like yaegi.

hobobaggins 5y ago

It would be nice if Go could do this sort of chaining like jQuery makes so easy with Javascript: n, err := object .Process1(arg1, arg2) .Process2(true) .Process3("foo") That would make these sorts of chains much easier to read.

svnpenn 5y ago
You can do that now, just move the dots to end of line https://github.com/89z/googleplay/blob/v1.8.1/header.go#L126 https://github.com/89z/googleplay/blob/v1.8.1/header.go#L126

kodah 5y ago

While using the package seems nice I think what's missed here is that Go requires a module declaration and dependencies to be enumerated. That would mix your "scripts" with any production code written in Go. Bash and Python keep my scripts away from my app dependencies.

emeraldd 5y ago

I'm having a very hard time getting past the fact that the "wrong" code is wrapped very badly on both a mobile device and laptop. That inherently makes it harder to read ...

ikiris 5y ago
They had to really struggle to make that code not clearly readable, and the complete lack of any error handling and dependence on some random framework somehow more desirable to the reader in the second example.

bobnamob 5y ago

May be derailing discussion a little, but the nicest scripting language/environment I've ever used is Clojure's babashka[1]. Clojure's data first model + conveniences like the -> and ->> macros make simple data pipeline scripts a joy to write. babashka of course suffers from the "it isn't everywhere" problem that is mentioned regarding python. https://github.com/babashka/babashka https://github.com/babashka/babashka

simonw 5y ago

This is done absolutely superb API design - taking inspiration from shell pipes and turning it into a Go library with syntax this clean is really impressive: https://github.com/bitfield/script https://github.com/bitfield/script

jsz0 5y ago

I felt really fancy about a decade ago learning python and adopting it for even the most trivial tasks. Then I realized at least half the stuff I wrote doesn't even run now without modification. Meanwhile bash scripts I wrote that predated that all still work perfectly. Now I'm back on the bash big time.

mistrial9 5y ago

this is personally challenging to me, and gets me out of my comfort zone -- in the right ways. Nobody needs to care about my own learning trajectory for server-side software, but, I will say that I have been hoarding 'good' bash scripts for a bunch of years, after having a bad start with Perl for server-side things. Bash needs no comments here, but I use it everyday. I avoided GO-lang partly due to an insider vibe from Xooglers, and partly internal prioritization. This article strikes that nerve today. thanks for posting .. no promises but bookmarked.

birchb 5y ago

I had a look at this library today. Quote the author: "The script library is implemented entirely in Go, and does not require any external userland programs." Sorry but that's not scripting anymore. We all like using userland programs! it is The Way. Most programs tell you why they failed on stderr. Seems like stderr is lost when a pipe component fails. Strangely stderr and stdout are conflated in the pipe structure - there is no way to get stderr! You just get the numeric exit status. IO redirection appears to be missing. Difficult to use in production.