9 ms·
Scripting with Go (2022)
- kardianos 3y agoInteresting. I do something similar with my task https://github.com/kardianos/task https://github.com/kardianos/task package, which is in tern loosely based off of another package from 10-15 years ago.
- tgv 3y agoThat sounds interesting, but the package is unfortunately undocumented. I tried https://pkg.go.dev/github.com/kardianos/task https://pkg.go.dev/github.com/kardianos/task, but that doesn't help me understand it either. It's missing a high level explanation of what to use it for, its limits and some decent examples.
- nullwarp 3y agoOh very neat, thanks for posting I will definitely give this a try.
- perfmode 3y agoFrom Sanjay Ghemawat, 9 years ago https://github.com/ghemawat/stream https://github.com/ghemawat/stream
- earthboundkid 3y agoThis post is several years old fwiw.
- everybodyknows 3y agoThere's a cute little icon telling us "Feb 21" right at the top but omitting the year, which would have been ever so helpful.
- tyingq 3y agoIt's wrapped with a nice semantic tag with the actual date: <time class="dt-published date-callout" datetime="2022-02-21" pubdate=""> <div class="date-wrapper"> <span class="month">Feb </span><span class="day">21</span> </div> </time> Not sure if that's correct or not, though.
- tambourine_man 3y agoThe pipe like code with dot notation reminds me a lot of jQuery. That’s a compliment.
- coldtea 3y agoTry jq if you haven't already.
- tambourine_man 3y agoYes, but jq’s syntax is impossible to memorize for me. gron | rg FTW
- myzie 3y agoTake a look at Risor and its pipes capability. https://github.com/risor-io/risor#quick-example https://github.com/risor-io/risor#quick-example Stay in the Go ecosystem, but gain pipes, Python-like f-strings, and more. (I'm the author)
- KnobbleMcKnees 3y agoI agree with this, and also in a complimentary way, but it all seems very non-idiomatic for Go. But I am not a Go expert by any means.
- simonw 3y agoIf you're not familiar with Go there is one detail missing from this post (though it's in the script README) - what a complete program looks like. Here's the example from https://github.com/bitfield/script#a-realistic-use-case https://github.com/bitfield/script#a-realistic-use-case package main import ( "github.com/bitfield/script" ) func main() { script.Stdin().Column(1).Freq().First(10).Stdout() }
- jvictor118 3y agoIf one were actually going to use something like this, I’d think it’d be worth implementing a little shebang script that can wrap a single-file script in the necessary boilerplate and call go run!
- simonw 3y agoThat's a really fun idea. I got that working here: https://til.simonwillison.net/bash/go-script https://til.simonwillison.net/bash/go-script Now you can run this: cat file.txt | ./goscript.sh -c 'script.Stdin().Column(1).Freq().First(10).Stdout()' Or write scripts like this - call it 'top10.sh': #!/tmp/goscript.sh script.Stdin().Column(1).Freq().First(10).Stdout() Then run this: chmod 755 topten.sh echo "one\none\ntwo" | ./topten.sh
- moondev 3y agoThis is fantastic. Thanks for sharing and also including docs on your process and usage!
- jayd16 3y agoHmm, I wonder if this is Microsoft's real endgame with allowing the single line C# syntax.
- alexk307 3y agoThe whole point of using Go is to explicitly handle errors as they happen. All of these steps can fail, but it’s not clear how they fail and if the next steps should proceed or be skipped on previous failures. This is harder to reason about, debug, and write than grep and bash.
- nerdbaggy 3y agoI ended up using this for my cli scripting needs. https://github.com/google/zx https://github.com/google/zx
- Alifatisk 3y agoWhy is the Google project advertising Webpod?
- js2 3y agoPrevious discussion (March 11, 2022 | 243 points | 66 comments): https://news.ycombinator.com/item?id=30641883 https://news.ycombinator.com/item?id=30641883
- sgarland 3y agoEvery time I see things like this, I feel like the person must be unaware of awk. # the original one-liner to get unique IP addresses cut -d' ' -f 1 access.log | sort | uniq -c | sort -rn | head # turns into this with GNU awk gawk '{PROCINFO["sorted_in"] = "@val_num_desc"; a[$1]++} END {c=0; for (i in a) if (c++ < 10) print a[i], i}' access.log It's also far, far faster on larger files (base-spec M1 Air): $ wc -lc fake_log.txt 1000000 218433264 fake_log.txt $ hyperfine "gawk '{PROCINFO[\"sorted_in\"] = \"@val_num_desc\"; a[\$1]++} END {c=0; for (i in a) if (c++ <10) print a[i], i}' fake_log.txt" Benchmark 1: gawk '{PROCINFO["sorted_in"] = "@val_num_desc"; a[$1]++} END {c=0; for (i in a) if (c++ <10) print a[i], i}' fake_log.txt Time (mean ± σ): 1.250 s ± 0.003 s [User: 1.185 s, System: 0.061 s] Range (min … max): 1.246 s … 1.254 s 10 runs $ hyperfine "cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head" Benchmark 1: cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head Time (mean ± σ): 4.844 s ± 0.020 s [User: 5.367 s, System: 0.087 s] Range (min … max): 4.817 s … 4.873 s 10 runs Interestingly, GNU cut is significantly faster than BSD cut on the M1: $ hyperfine "gcut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head" Benchmark 1: gcut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head Time (mean ± σ): 3.622 s ± 0.004 s [User: 4.149 s, System: 0.078 s] Range (min … max): 3.616 s … 3.629 s 10 runs
- zer8k 3y agoI don't understand the downvotes. This is a fair criticism. The author even points out "programs as pipelines" which is literally the UNIX philosophy. There are tools that already exist on UNIX-likes more people should use instead of reaching for a script. I can sympathize with the author w.r.t wanting to use a single language you like for everything. However, after decades I've found this to be untenable. There are languages that are just simply better for one-off scripting (Perl, Python), and languages that aren't (anything compiled). Trying to bolt an interpreter onto a compiled language from the outside seems like a lot of work for questionable gain.
- karmakaze 3y agoThe 'scripting' vs 'compiled' language is a false dichotomy. Awk, Perl, Python are compiled programs. What makes a 'scripting' language special? Dynamic typing? Lack of compile step/delay? I could imagine a lifetime of collecting scripting macros/libs in lisp to be as good or better.
- wudangmonk 3y agoThe unix philosophy of having small programs that take in input, process it and return a result has proven to a success, I just never understood why the next logical step of having this program in library form never became a thing. I guess shells are a bit useful but not as useful as a decent repl (common-lisp or the jupyter repl) where these programs can be used as if they were a function.
- nicechianti 3y agoterrible idea
- geenat 3y agoWould love to use more golang- amazing build system and cross compiler built in. "All in one" binaries are the best thing ever. I adore most of the ideas in the language. .... but there are just soooo many little annoyances / inconveniences which turn me off. - No Optional Parameters. No Named Parameters. Throw us a bone Rob Pike, it's 2023. Type inferred composite literals may be an OK compromise.. if we ever see them: https://github.com/golang/go/issues/12854 https://github.com/golang/go/issues/12854 - Unused import = will not compile. Unused variable = Will not compile. Give us the ability to turn off the warning. - No null safe or nullish coalescing operator. (? in rust, ?? in php, etc.) - Verbosity of if err != nil { return err; } - A ternary operator would be nice, and could bring if err != nil to 1 line. - No double declarations. “no new variables on left side of :=” .. For some odd reason “err” is OK here... Would be highly convenient for pipelines, so each result doesn't need to be uniquely named. I'd describe Go as a "simple" language- Not an "easy" language. 1-2 lines in Python is going to be 5-10 lines in golang. Note: Nim has most of these..
- skybrian 3y agoHave you tried Deno?
- myzie 3y agoAgreed! Shameless plug: this is why I built Risor. https://github.com/risor-io/risor https://github.com/risor-io/risor Keep in the Go ecosystem, retain compatibility with the Go programs you already have, but have a much more concise scripting capability at your disposal.
- jerf 3y agoI don't do a lot of shell scripting type things in Go because it's not a great language for it, but when I do, I take another approach, which is just to panic. Generics offer a nice little func Must[T any](x T, err error) T { if err != nil { panic(err) } return x } which you can wrap around any standard "x, err :=" function to just make it panic, and even prior to generics you could wrap a "PanicOnErr(justReturnsErr())". In the event that you want to handle errors in some other manner, you trivially can, and you're not limited to just the pipeline design patterns, which are cool in some ways, but limiting when that's all you have. (It can also be tricky to ensure the pipeline is written in a way that doesn't generate a ton of memory traffic with intermediate arrays; I haven't checked to see what the library they show does.) Presumably if I'm writing this in Go I have some other reason for wanting to do that, like having some non-trivial concurrency desire (using concurrency to handle a newline-delimited JSON file was my major use case, doing non-trivial though not terribly extensive work on the JSON). While this may make some people freak, IMHO the real point of "errors as values" is not to force you to handle the errors in some very particular manner, but to make you think about the errors more deeply than a conventional exceptions-based program typically does. As such, it is perfectly legal and moral to think about your error handling and decide that what you really want is the entire program to terminate on the first error. Obviously this is not the correct solution for my API server blasting out tens of thousands of highly heterogeneous calls per second, but for a shell script it is quite often the correct answer. As something I have thought about and chosen deliberately, it's fine.
- booleandilemma 3y agoI like Go, but its insistence on not permitting unused imports and unused variables make it unsuitable for scripting, imo. For scripting I want something that I can be fast and messy in. Go is the opposite of that. It's ok, a language doesn't have to be good at everything.
- nprateem 3y agoThere should totally be a compiler flag to not require those
- saghm 3y agoMaybe there could be a whole class of things that are like errors, but not as severe, and flags that deal with them as a group! We could call them "warnings"
- tacticus 3y agoOh you mean things that are ignored in every situation ever
- saghm 3y agoIf you choose to do that with warnings, that's your prerogative. I pretty much take the opposite approach personally; before putting code into review (or merging, if it's a personal project with no reviewers), every warning is inspected and I decide whether to fix it by changing the code not to generate it or manually suppress it only in that specific location. 99% of the time the warning is a sign of something actually wrong, but the 1% where I know that I actually don't care (and the extremely frequent occurrences when I don't while in the development part of the cycle and not finished with the implementation), it's much better to not have it completely block me.
- nix-zarathustra 3y agoI agree. When I am scripting, I want to have quick feed back loop. You can't really do that with Go because it doesn't have as good introspection and debugging capabilities as a scripting language like Ruby and doesn't have exceptions, which means that error handling is more verbose than necessary. Also, I like being able to make modifications on the fly, so doing something in Ruby, I can just open the file make adjustments and I am done. With Go, I have to compile it and move it back into my path which is really tedious.
- ComputerGuru 3y agoTangentially related: I posted a shebang for scripting in rust some years ago, if anyone is interested: https://neosmart.net/blog/self-compiling-rust-code/ https://neosmart.net/blog/self-compiling-rust-code/
- simonw 3y agoInspired by comments in this thread, I threw together a Bash script that lets you do this: cat file.txt | ./goscript.sh -c 'script.Stdin().Column(1).Freq().First(10).Stdout()' You can also use it as a shebang line to write self-contained scripts. Details here: https://til.simonwillison.net/bash/go-script https://til.simonwillison.net/bash/go-script
- fsmv 3y agoI put together a go "sh-bang" line so you can just chmod +x your .go file and run it (and it works with go fmt unlike other options). /*usr/bin/env go run "$0" "$@"; exit $? #*/ It's fun try it out! Just make this the first line of the file.
- ilyt 3y agoPerl was literally made for that, just use it
- Hendrikto 3y agoThis is satire, right? I think commenters are completely missing the point. https://en.m.wikipedia.org/wiki/A_Modest_Proposal https://en.m.wikipedia.org/wiki/A_Modest_Proposal
- dang 3y agoThe submitted title was "Scripting with Go: A Modest Proposal" but the phrase "modest proposal" doesn't appear in the article, so I've taken it out. "Please use the original title, unless it is misleading or linkbait; don't editorialize." - https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- pdimitar 3y agoShell scripting is quite fine up until certain complexity (say 500-1000 lines), after which adding even a single small feature becomes a huge drag. We're talking hours for something that would take me 10 minutes in Golang and 15 in Rust. Many people love to smirk and say "just learn bash properly, duh" but that's missing the point that we never do big projects in bash so our muscle memory of bash is always kind of shallow. And by "we" I mean "a lot of programmers"; I am not stupid, but I have to learn bash's intricacies every time almost from scratch and that's not productive. It's very normal for things to slip up from your memory when you're not using them regularly. To make this even more annoying, nobody will pay me to work exclusively with bash for 3 months until it gets etched deep into my memory. So there's that too. I view OP as a good reminder that maybe universal-ish tools to get most of what we need from shell scripting exist even today but we aren't giving them enough attention and energy and we don't make them mainstream. Though it doesn't help that Golang doesn't automatically fetch dependencies when you just do `go run random_script.go`: https://github.com/golang/go/issues/36513 https://github.com/golang/go/issues/36513 I am not fixating on Golang in particular. But IMO next_bash_or_something should be due Soon™. It's not a huge problem to install a single program when provisioning a new VM or container either so I am not sure why are people so averse to it. So yeah, nice article. I like the direction. EDIT: I know about nushell, oilshell and fish but admittedly never gave them a chance.
- dang 3y agoDiscussed at the time: Scripting with Go - https://news.ycombinator.com/item?id=30641883 https://news.ycombinator.com/item?id=30641883 - March 2022 (66 comments)
- 1vuio0pswjnm7 3y agoexport LC_ALL=C awk '!a[$1]++' access.log|head If access.log is large enough, awk will fail. When this happens, one can split access.log into pieces, process separately then recombine. But that's more or less what sort(1) does with large files, creating temporary files in $TMPDIR or other user-specified directory after -T if using GNU sort. There was a way to eliminate duplicate lines from an unordered list using k/q, without using temporary files but I stopped using it after Kx, Inc. was sold off and I started using musl exclusively. q requires glibc. For example, something like #!/bin/sh # usage: $0 file echo "k).Q.fs[l:0::\`:$1];l:?:l;\`:$1 0:l"|exec q >null; Can this be done in ngn k. The other approach I use to avoid temporary files is to just put the list in an SQL database, add a UNIQUE constraint, and update the database.
- ramranch 3y agoOut of curiosity, what's your reasoning for using musl exclusively
- deleted 3y ago[deleted]
- 38 3y ago> cut -d' ' -f 1 access.log |sort |uniq -c |sort -rn |head FYI Go has Compact now: https://godocs.io/slices#Compact https://godocs.io/slices#Compact
- fuzztester 3y agoRelated: Ousterhout's dichotomy: https://en.m.wikipedia.org/wiki/Ousterhout%27s_dichotomy https://en.m.wikipedia.org/wiki/Ousterhout%27s_dichotomy
- DanielHB 3y agoI have been thinking that JS template literals could be a great replacement for shell programming, allowing you to make more powerful syntax to emulate a lot of bash useful things while still having a lot of a proper programming language power for example: import { jsh, cat, grep, PipeOutput } from 'jsh; // type PipeOutput = { stdout: ReadableStream, toString: () => Promise<string>, extra: Record<string,any> } function countLines(input: PipeOutput, argv: string[]): PipeOutput { // ... } const textToLookFor = process.argv[1] const output: PipeOutput = jsh`${cat} file.txt | ${grep} ${textToLookFor} | ${countLines}` console.log(output.toString())