7 ms·

Go for System Administrators

174 points by polvi 13y ago

JeremyMorgan 13y ago
While I'm too tired to read this thru thoroughly, I had to skim it and upvote it. This was actually my first thought when go was introduced. I don't see it taking over the web world, or the gaming world or anything of that sort, but I do see it being the next tool for multithreaded applications related to system work. Go may never "hit it big" with the mainstream but it very well could find it's place in administration. That's what I hope to use it for as I get more proficient.
- josephkern 13y ago
  It was built around software engineering principles (rather than experimental language constructs). The attention to small details (imports via git/bzr/hg, gofmt, go run, etc) deeply resonates with system admins (or at least myself and the OP).
  - dubbledidu 13y ago
    That's my impression too. The lone sysadmin who has to hack something up and moves on to other work likes Go; real developers who work in teams and have to maintain their codebase a month down the road stay the hell away from it.
    - xradionut 13y ago
      My coworkers whom aren't gurus can read and edit Python code. Go? I doubt it...
      - kinghajj 13y ago
        How is go so much more complex? The fact that struct types can be passed either by reference or by value?
        xradionut 13y ago
        It's not go. It's the fact I work with admins and analysts that are not formally trained developers. Python and PowerShell are their tools.
    - stevvooe 13y ago
      real developers who work in teams and have to maintain their codebase a month down the road stay the hell away from it. As a fake developer, our team has had little issue with code maintenance "a month down the road". I'm not sure what "real" developers" are using but their platitudes must be an effective resource.
      - cmccabe 13y ago
        Real Programmers wrote in machine code. Not FORTRAN. Not RATFOR. Not, even, assembly language. Machine Code. Raw, unadorned, inscrutable hexadecimal numbers. Directly. - The Story of Mel. Real Users hate Real Programmers - FreeBSD fortune
danieldk 13y ago
A long post that gives two arguments near the end in favor of Go over their old language of choice (Python): 1. No dependency issues. 2. Compiles with no issues on different platforms. And one sort of argument in between: 3. Some of the tools that they use were written in Go. While (1) is true for deployment, dependency handing is not a solved problem in Go. Since import statements do not allow you to specify versions, you have to keep track of which commits of a particular project you trust. I am pretty sure that this problem will be solved, but it currently isn't. (3) is perhaps an argument in favor of being able to read Go, but not to switch to Go. By those standards, everyone on Unix should be writing C ;).
- cmccabe 13y ago
  One thing that annoyed me about both Python and Rubyis that the way things in the standard library worked changed from release to release in subtle, incompatible ways. I remember getting burned by a change in how iterating over characters in a string worked in Ruby in the 1.8.x versions. Sorry-- I don't remember the details, it's been a few years. The point is that if you want to run in on a different PC than you developed the code on, or upgrade, these are buried landmines which are going to blow your leg off sooner or later. If you're working on an open source project, often the first sign you get that your Python code doesn't work on older or newer versions is a user reporting a mysterious RuntimeException. virtualenv can sort of solve a lot of these issues, if you have complete control of all computers running your software, and if you can find a Python version that works for all the packages you need. But for a lot of open source projects, these assumptions may not be true. And then you have the issue of stagnation. It sucks to be stuck on Python 2.4.x in 2013. But how do you make a business case for a potentially very risky upgrade? This is the real reason why initiatives like Python 3.x and Perl 6.x have been DOA, in my opinion.
  - philsnow 13y ago
    > One thing that annoyed me about both Python and Rubyis that the way things in the standard library worked changed from release to release in subtle, incompatible ways. I got bitten by one such implementation detail changing in between releases in python. It had something to do with the pickle machinery, like the 'set' builtin changing from using __reduce__ to __reduce_ex__ or vice versa, or something like that, and all the pickles that were serialized in 2.4 were deserializing incorrectly in 2.6. I can't remember the details now :\
shmageggy 13y ago
This is timely as I was just fiddling around with some CLI and subprocess communication in Go as I learn the language. Is there anything like envoy for Go? While interacting with processes wasn't too tough (once I found the bufio package -- doh!) there's still a lot of boilerplate involved. That's partially unavoidable given the static nature of the language, but it would be nice to have a library that streamlined some common patterns.
- cmccabe 13y ago
  I think what Go's subprocess stuff needs most is better examples. There aren't a lot of examples out there yet. For example, I saw one guy on StackOverflow use a complicated Reader thread to print a subprocess' output to stdout, when all he had to do was set Command#Stdout to os.stdout. I hope that I'm not assuming too much here (maybe you really did need bufio for your code), but in a lot of cases you don't need it. Like if you want to supply stdin to a process, just use Command#Stdin = strings.NewReader("what I want to send to stdin"). For getting the command output, it's often enough to just call Command#CombinedOutput and cast to a string-- no bufio required unless you're doing something fancy. I wrote some utility code that might or might not be helpful: https://github.com/cmccabe/test2/blob/master/subprocess.go https://github.com/cmccabe/test2/blob/master/subprocess.go
  - shmageggy 13y ago
    I definitely agree about the examples. There's really nothing out there except what you might run into on SO. In my case I wanted to programmatically communicate with an interactive command line app, so I ended up using StdinPipe and StdoutPipe along with a bufio.Writer and bufio.Scanner. This allowed me to easily send strings and read the lines that came back, which worked reasonably well. I agree that this would be overkill for simpler cases, in which your approach makes sense.
- krasin 13y ago
  Do you use os/exec [1] package or directly start processes with os.StartProcess? [1] http://golang.org/pkg/os/exec/ http://golang.org/pkg/os/exec/
flyt 13y ago
So, where should I Go to get started learning Go as a sysadmin?
- _ak 13y ago
  http://tour.golang.org/ http://tour.golang.org/
- minikomi 13y ago
  If you understand that they seem to be written as a learning exercises by the respective authors, I found that there's a good introduction to the nuts and bolts to be had browsing attempts at creating userspace tools in go: eg. https://bitbucket.org/telesto/goblin/ https://bitbucket.org/telesto/goblin/ or https://github.com/manveru/goblin https://github.com/manveru/goblin .. it seems to be a popular first project :)
- danieldk 13y ago
  I think Mark Summerfield's book is quite a good introduction: http://www.qtrac.eu/gobook.html http://www.qtrac.eu/gobook.html Many of his examples touch upon sysadmin work, such as the MP3 playlist conversion, implementing concurrent grep, and coverage of common file formats (XML, JSON, plain text) and archive formats. It's also a good introduction to Go. At times it drinks the kool-aid a bit to much, it sometimes takes quite mundane constructs as an example how stellar Go is (let's be honest, Go is quite a boring language, but boring can be good at times).
- josephkern 13y ago
  sysadmin here; I've been enjoying An Introduction to Programming in Go by Caleb Doxsey[1]. Short, sweet, and right to the point without being terse. There is nothing in this book that will directly relate to using go as an automation language, but I'd rather learn the fundamentals before learning specialized examples. [1]: http://www.golang-book.com/ http://www.golang-book.com/
- bgentry 13y ago
  https://gobyexample.com/ https://gobyexample.com/ is also a great resource
tibbit 13y ago
Anyone else able to build this? I ran into a brick wall: $ ./build # github.com/coreos/etcd/store src/github.com/coreos/etcd/store/store.go:633: method s.checkNode is not an expression, must be called # github.com/ccding/go-config-reader/config src/github.com/ccding/go-config-reader/config/config.go:35: undefined: bufio.NewScanner # github.com/coreos/go-raft src/github.com/coreos/go-raft/log.go:241: function ends without a return statement
- frou_dh 13y ago
  Looks like the code is using Go 1.1 features and you have an older install.
txutxu 13y ago
Compiling in production? so this is the lang to "go" in case you're such kind of sysadmin which allows a compiler in production. And have I to assume, it's the lang to "go", in case you're the kind of sysadmin which does not package things using the platform standards? Is "go" be the solution to "administer" backups, monitoring, deployment, staging, network devices, provisioning, on a heterogeneous and scalable environment ? For me, the "Zen" as sysadmin, is to impose as few toolchains and dependencies external to the _app_ to support, as possible. Going to "go" for me is as say that "java" is the lang to go if you're a sysadmin. Fact 1) Sysadmin should be polyglot. For her needed tasks, and for the apps to support. Fact 2) The right tool for the job. There is many many sub-jobs to administer a system properly (including security), there is no "one tool". Fact 3) The good sysadmin is invisible, and her job goes unnoticed. Even if it's not the same to support a single app in a few servers, than to administer a multinational company internal and external networks, I still think that adding "go" to any system planning, just to "administer it", is crazy. Fact 4) How many good sysadmins are good in "go"? you know, people moves, teams change, requirements change... is this choice really "helping" on the task of "administer" the infrastructure of the company in the long term? Anyway, I don't say it will not work. There are many environments, and different people. It may work. I simply think this is not a "general purpose" advise to follow. Congrats if it works in the case mentioned. Anything that always just works, is great, even if not advisable for other environments, teams or companies.
- ProblemFactory 13y ago
  > For me, the "Zen" as sysadmin, is to impose as few toolchains and dependencies external to the _app_ to support, as possible. Going to "go" for me is as say that "java" is the lang to go if you're a sysadmin. That seems to be the main advantage of Go that the author mentions. The Go compiler builds statically linked binaries. Once built, the single file can be copied and run on any common system, without installing the compiler/interpreter, packages or libraries onto that system. While I like Python, running Python scripts on servers requires considerable effort for deployment. You have to ensure that the right versions of the right packages are installed (which is even more difficult when it's someone else's server). Whereas with Go "deployment" would be just a file copy.
dhiraj86 13y ago
Interesting Post.
0xdeadbeefbabe 13y ago
didn't read the article, but the headline could benefit from s/Go/Golang/

174 points by polvi 13y ago

While I'm too tired to read this thru thoroughly, I had to skim it and upvote it. This was actually my first thought when go was introduced. I don't see it taking over the web world, or the gaming world or anything of that sort, but I do see it being the next tool for multithreaded applications related to system work. Go may never "hit it big" with the mainstream but it very well could find it's place in administration. That's what I hope to use it for as I get more proficient.

josephkern 13y ago
It was built around software engineering principles (rather than experimental language constructs). The attention to small details (imports via git/bzr/hg, gofmt, go run, etc) deeply resonates with system admins (or at least myself and the OP).
- dubbledidu 13y ago
  That's my impression too. The lone sysadmin who has to hack something up and moves on to other work likes Go; real developers who work in teams and have to maintain their codebase a month down the road stay the hell away from it.
  - xradionut 13y ago
    My coworkers whom aren't gurus can read and edit Python code. Go? I doubt it...
    - kinghajj 13y ago
      How is go so much more complex? The fact that struct types can be passed either by reference or by value?
      - xradionut 13y ago
        It's not go. It's the fact I work with admins and analysts that are not formally trained developers. Python and PowerShell are their tools.
  - stevvooe 13y ago
    real developers who work in teams and have to maintain their codebase a month down the road stay the hell away from it. As a fake developer, our team has had little issue with code maintenance "a month down the road". I'm not sure what "real" developers" are using but their platitudes must be an effective resource.
    - cmccabe 13y ago
      Real Programmers wrote in machine code. Not FORTRAN. Not RATFOR. Not, even, assembly language. Machine Code. Raw, unadorned, inscrutable hexadecimal numbers. Directly. - The Story of Mel. Real Users hate Real Programmers - FreeBSD fortune

danieldk 13y ago

A long post that gives two arguments near the end in favor of Go over their old language of choice (Python): 1. No dependency issues. 2. Compiles with no issues on different platforms. And one sort of argument in between: 3. Some of the tools that they use were written in Go. While (1) is true for deployment, dependency handing is not a solved problem in Go. Since import statements do not allow you to specify versions, you have to keep track of which commits of a particular project you trust. I am pretty sure that this problem will be solved, but it currently isn't. (3) is perhaps an argument in favor of being able to read Go, but not to switch to Go. By those standards, everyone on Unix should be writing C ;).

cmccabe 13y ago
One thing that annoyed me about both Python and Rubyis that the way things in the standard library worked changed from release to release in subtle, incompatible ways. I remember getting burned by a change in how iterating over characters in a string worked in Ruby in the 1.8.x versions. Sorry-- I don't remember the details, it's been a few years. The point is that if you want to run in on a different PC than you developed the code on, or upgrade, these are buried landmines which are going to blow your leg off sooner or later. If you're working on an open source project, often the first sign you get that your Python code doesn't work on older or newer versions is a user reporting a mysterious RuntimeException. virtualenv can sort of solve a lot of these issues, if you have complete control of all computers running your software, and if you can find a Python version that works for all the packages you need. But for a lot of open source projects, these assumptions may not be true. And then you have the issue of stagnation. It sucks to be stuck on Python 2.4.x in 2013. But how do you make a business case for a potentially very risky upgrade? This is the real reason why initiatives like Python 3.x and Perl 6.x have been DOA, in my opinion.
- philsnow 13y ago
  > One thing that annoyed me about both Python and Rubyis that the way things in the standard library worked changed from release to release in subtle, incompatible ways. I got bitten by one such implementation detail changing in between releases in python. It had something to do with the pickle machinery, like the 'set' builtin changing from using __reduce__ to __reduce_ex__ or vice versa, or something like that, and all the pickles that were serialized in 2.4 were deserializing incorrectly in 2.6. I can't remember the details now :\

shmageggy 13y ago

This is timely as I was just fiddling around with some CLI and subprocess communication in Go as I learn the language. Is there anything like envoy for Go? While interacting with processes wasn't too tough (once I found the bufio package -- doh!) there's still a lot of boilerplate involved. That's partially unavoidable given the static nature of the language, but it would be nice to have a library that streamlined some common patterns.

cmccabe 13y ago
I think what Go's subprocess stuff needs most is better examples. There aren't a lot of examples out there yet. For example, I saw one guy on StackOverflow use a complicated Reader thread to print a subprocess' output to stdout, when all he had to do was set Command#Stdout to os.stdout. I hope that I'm not assuming too much here (maybe you really did need bufio for your code), but in a lot of cases you don't need it. Like if you want to supply stdin to a process, just use Command#Stdin = strings.NewReader("what I want to send to stdin"). For getting the command output, it's often enough to just call Command#CombinedOutput and cast to a string-- no bufio required unless you're doing something fancy. I wrote some utility code that might or might not be helpful: https://github.com/cmccabe/test2/blob/master/subprocess.go https://github.com/cmccabe/test2/blob/master/subprocess.go
- shmageggy 13y ago
  I definitely agree about the examples. There's really nothing out there except what you might run into on SO. In my case I wanted to programmatically communicate with an interactive command line app, so I ended up using StdinPipe and StdoutPipe along with a bufio.Writer and bufio.Scanner. This allowed me to easily send strings and read the lines that came back, which worked reasonably well. I agree that this would be overkill for simpler cases, in which your approach makes sense.
krasin 13y ago
Do you use os/exec [1] package or directly start processes with os.StartProcess? [1] http://golang.org/pkg/os/exec/ http://golang.org/pkg/os/exec/

flyt 13y ago

So, where should I Go to get started learning Go as a sysadmin?

_ak 13y ago
http://tour.golang.org/ http://tour.golang.org/
minikomi 13y ago
If you understand that they seem to be written as a learning exercises by the respective authors, I found that there's a good introduction to the nuts and bolts to be had browsing attempts at creating userspace tools in go: eg. https://bitbucket.org/telesto/goblin/ https://bitbucket.org/telesto/goblin/ or https://github.com/manveru/goblin https://github.com/manveru/goblin .. it seems to be a popular first project :)
danieldk 13y ago
I think Mark Summerfield's book is quite a good introduction: http://www.qtrac.eu/gobook.html http://www.qtrac.eu/gobook.html Many of his examples touch upon sysadmin work, such as the MP3 playlist conversion, implementing concurrent grep, and coverage of common file formats (XML, JSON, plain text) and archive formats. It's also a good introduction to Go. At times it drinks the kool-aid a bit to much, it sometimes takes quite mundane constructs as an example how stellar Go is (let's be honest, Go is quite a boring language, but boring can be good at times).
josephkern 13y ago
sysadmin here; I've been enjoying An Introduction to Programming in Go by Caleb Doxsey[1]. Short, sweet, and right to the point without being terse. There is nothing in this book that will directly relate to using go as an automation language, but I'd rather learn the fundamentals before learning specialized examples. [1]: http://www.golang-book.com/ http://www.golang-book.com/
bgentry 13y ago
https://gobyexample.com/ https://gobyexample.com/ is also a great resource

tibbit 13y ago

Anyone else able to build this? I ran into a brick wall: $ ./build # github.com/coreos/etcd/store src/github.com/coreos/etcd/store/store.go:633: method s.checkNode is not an expression, must be called # github.com/ccding/go-config-reader/config src/github.com/ccding/go-config-reader/config/config.go:35: undefined: bufio.NewScanner # github.com/coreos/go-raft src/github.com/coreos/go-raft/log.go:241: function ends without a return statement

frou_dh 13y ago
Looks like the code is using Go 1.1 features and you have an older install.

txutxu 13y ago

Compiling in production? so this is the lang to "go" in case you're such kind of sysadmin which allows a compiler in production. And have I to assume, it's the lang to "go", in case you're the kind of sysadmin which does not package things using the platform standards? Is "go" be the solution to "administer" backups, monitoring, deployment, staging, network devices, provisioning, on a heterogeneous and scalable environment ? For me, the "Zen" as sysadmin, is to impose as few toolchains and dependencies external to the _app_ to support, as possible. Going to "go" for me is as say that "java" is the lang to go if you're a sysadmin. Fact 1) Sysadmin should be polyglot. For her needed tasks, and for the apps to support. Fact 2) The right tool for the job. There is many many sub-jobs to administer a system properly (including security), there is no "one tool". Fact 3) The good sysadmin is invisible, and her job goes unnoticed. Even if it's not the same to support a single app in a few servers, than to administer a multinational company internal and external networks, I still think that adding "go" to any system planning, just to "administer it", is crazy. Fact 4) How many good sysadmins are good in "go"? you know, people moves, teams change, requirements change... is this choice really "helping" on the task of "administer" the infrastructure of the company in the long term? Anyway, I don't say it will not work. There are many environments, and different people. It may work. I simply think this is not a "general purpose" advise to follow. Congrats if it works in the case mentioned. Anything that always just works, is great, even if not advisable for other environments, teams or companies.

ProblemFactory 13y ago
> For me, the "Zen" as sysadmin, is to impose as few toolchains and dependencies external to the _app_ to support, as possible. Going to "go" for me is as say that "java" is the lang to go if you're a sysadmin. That seems to be the main advantage of Go that the author mentions. The Go compiler builds statically linked binaries. Once built, the single file can be copied and run on any common system, without installing the compiler/interpreter, packages or libraries onto that system. While I like Python, running Python scripts on servers requires considerable effort for deployment. You have to ensure that the right versions of the right packages are installed (which is even more difficult when it's someone else's server). Whereas with Go "deployment" would be just a file copy.

dhiraj86 13y ago

Interesting Post.

0xdeadbeefbabe 13y ago

didn't read the article, but the headline could benefit from s/Go/Golang/