8 ms·

Coming Soon – AWS SDK for Go

209 points by urlgrey 12y ago

enahs-sf 12y ago
This is great! I can now unify a lot of my workflow and not have to use ruby as a glue layer between applications and operations. It's great to see the Go ecosystem improving! I think in a few years, it may become the defacto tool of systems programmers.
- vezzy-fnord 12y ago
  It's not going to be displacing low-level programming any time soon, unless radical changes are in store for 2.x and beyond, hypothetically. That said, I can see it becoming a standard for daemons, utilities and low-level userspace in general due to its rather excellent POSIX and syscall support. I really find the (present) lack of this to be Rust's main deficiency, though the third-party crates are slowly picking up with bindings for things like prctl(), epoll() and so on. EDIT: Why the downvotes? Rob Pike has stated that he regrets attaching the "system programming" label to Go, since he never intended it for OS kernels and the like. If it's about me being wrong on Rust's POSIX bindings or Go's suitability for daemons, do prove me wrong.
  - bacongobbler 12y ago
    I believe this is why you are getting downvoted: > It's not going to be displacing low-level programming any time soon, unless radical changes are in store for 2.x and beyond, hypothetically What radical changes are you referring to, and why are they needed in order for Go to be a de-facto language for system programmers?
    - eropple 12y ago
      Drop the garbage collector. Systems programming cannot stop the world.
      - jscheel 12y ago
        You can already do that: http://golang.org/pkg/runtime/ http://golang.org/pkg/runtime/
        shanemhansen 12y ago
        That's not useful for a kernel. You can disable garbage collection in go, but go doesn't provide the tools to disable creating garbage entirely. You can get really close to creating zero garbage through profiling and helping the compiler's escape analysis, but afaik the go spec makes no guarantees about stack vs heap allocation.
        pjmlp 12y ago
        > That's not useful for a kernel. Oberon Kernel.Mod, initial version 11.4.86 http://www.inf.ethz.ch/personal/wirth/ProjectOberon/Sources/Kernel.Mod.txt http://www.inf.ethz.ch/personal/wirth/ProjectOberon/Sources/...
        shanemhansen 12y ago
        I failed to see any references to the go runtime package in that link which leads me to believe you misunderstood my statement. Using the APIs documented in the go runtime package to disable garbage collection would not be an effective memory management strategy for a kernel.
    - vezzy-fnord 12y ago
      The GC is more of a cultural issue than anything (what with Oberon being perfectly fine), but it will have an impact as a result. The runtime will be an overhead, though it does have a malleable library interface, apparently.
  - pjmlp 12y ago
    Once upon a time that was a programming language called Cedar that was used at Xerox PARC to build an environment for strong typed languages similar to what Smalltalk and Interlisp-D environments offered. It used reference counting with GC for collecting cycles. A certain swiss guy decided to create a graphical workstation based on the NS32000 processor, using a revised version of Modula-2, with a GC for systems programming, named Oberon. The Oberon OS managed to provide a fully working graphics workstation capable of fulfilling many tasks at the ETHZ systems programming department. Specially the latest version, System 3 with the GUI Gadgets framework. It had as descendents EthOS, BlueBottle AOS and its design influenced Plan9's ACME and Rio. But the world was busy looking into UNIX being adopted by the industry and Oberon is no more.
    - vezzy-fnord 12y ago
      As I noted regarding Oberon, but you'll have a hard time trying to break the cultural bias against GC.
      - pjmlp 12y ago
        I learned long time ago that you just need to wait a few programmer generations, than the bias against whatever technology is solved by virtue of nature. When I started programming, the bias was against any language higher level than Assembly for home computers.
- shanemhansen 12y ago
  Frankly go isn't my favorite glue language (and I'm one of the relatively few people who's been doing go full time for a couple years). I currently prefer python for that kind of stuff, or sometimes bash. However it's certainly possible to use as a glue language. I've talked to some folks at DO who just install go on machines and `go run` scripts.
  - bryanl 12y ago
    I'm going to go out on a limb and say that probably isn't true. All go at DigitalOcean is deployed using one of two mechanisms. Nothing is `go run` outside of a developer's workstation.
  - enahs-sf 12y ago
    For small internal projects, I typically find that I end up writing the program and compiling it; It just builds up my developer toolchain from the command line to do my bidding. Also, these are all managed under source control, so my co-workers can also take advantage.
kid0m4n 12y ago
This is great news indeed. Why could this not have come 8 months ago? Back then, it was quite disheartening to see the replies from AWS team about Go support.
bkeroack 12y ago
There exists a third party library called goamz which has been forked all over the place when people have needed to add random features not found in the original version. It works well but hopefully you don't need a set of features that require you to import multiple versions. Hopefully Amazon's version will be comprehensive and idiomatic.
- Rapzid 12y ago
  True. The "big" players that forked it couldn't be explicatived to work together so the project became seriously fractured. I'm hoping this will move swiftly through experimental to release, and that the generated nature doesn't produce yucky to use implementations.
- f2f 12y ago
  the problem with goamz, when i forked it, was that merging changes into canonical's bazaar cvs was very hard to do. that was before github and pull requests became de-rigeur, so i'm a bit dated. still, very few of canonical's go projects are bzr-based anymore. that's the reason you're seeing all those forks: we all added what we needed, couldn't get it in, and gave up.
bgentry 12y ago
It's awesome that they reached out to the authors of a promising community project[1] and adopted that, rather than building their own project from scratch. AWS' first-party client libraries are generally awful, but I think this one stands a chance to be very good. Also, the current situation with goamz is truly horrific. There are multiple independent forks of the original Canonical Launchpad-hosted version, each with their own subtle differences in usage, interfaces and feature support. None of them feel like idiomatic Go. It really would be unlikely for this now AWS-supported project to end up doing worse than the status quo. [1] mainly Coda Hale from Stripe, where the project originated: https://github.com/stripe/aws-go https://github.com/stripe/aws-go
- jeffbarr 12y ago
  > AWS' first-party client libraries are generally awful Do you have some feedback that I can pass along to the AWS SDK team?
  - jbergknoff 12y ago
    The node.js library [1] is incredibly well done. It's easily the best-documented node.js library I've ever seen. [1] https://github.com/aws/aws-sdk-js https://github.com/aws/aws-sdk-js
  - omarforgotpwd 12y ago
    I like the first party libraries personally, because I find them to be very full featured.
  - hoodoof 12y ago
    > AWS' first-party client libraries are generally awful Not a reasonable comment.
  - bgentry 12y ago
    Sure, but going from memory on this :) As far as the Ruby SDK, my opinions are mostly based on how things were 2-3 years ago, when I wrote it off entirely after inheriting several large projects that made heavy use of it. In the past, it was difficult (impossible?) to figure out how it worked by reading the code. That's highly unusual for a Ruby library. It felt like a generated library, not like Ruby. When something wasn't working as expected (which happened often), it was difficult to figure out what was going wrong and why. As a heavy AWS user, I've felt much more comfortable using Fog, even though I only really use its AWS provider. That's true even though I didn't even get to take advantage of Fog's multi-provider abstractions (basically just using it to make raw AWS API calls). Hopefully that's all in the past and I should give it another look the next time I'm doing that stuff in Ruby. Fog also gave much better control over a lot of the details that are important when doing stuff in production, like timeouts and keep-alive settings. The 2nd example that comes to mind is the AWS Flow framework for Ruby: https://github.com/aws/aws-flow-ruby https://github.com/aws/aws-flow-ruby Flow does things like create global methods ( https://github.com/aws/aws-flow-ruby/blob/master/aws-flow/lib/aws/decider.rb#L60 https://github.com/aws/aws-flow-ruby/blob/master/aws-flow/li... ) and its DSL abstraction just doesn't feel right. It tries too hard to make it feel like you're just running Ruby code, and it's tough to determine how it uses SWF under the hood. It's such a large abstraction on top of SWF that it basically means you can't write parts of your app in anything other than the Flow frameworks for Java or Ruby. But then, the other major SWF framework I've seen (https://github.com/sclasen/swf4go https://github.com/sclasen/swf4go) also locks you in. So maybe it's just a shortcoming of SWF that it's not usable without a big abstraction layered on top of it? From what others are saying, it sounds like the newer stuff is in much better shape. That's very encouraging! Regardless, I think you've made a great decision in taking over aws-go. I have very high hopes for its future :)
marcosnils 12y ago
aws golang sdk's have been around for a while and I've found several other which are well tested and maintained such as https://github.com/goamz/goamz https://github.com/goamz/goamz. It's awesome that aws finally decide which library they're officially adopting but I found frustrating that the library is poorly tested and it will take while until it becomes the official. It'd be great if all the other sdk's contributors start working on the same project.
- djhworld 12y ago
  The goamz project is an embarrassing mess, so many forks.
  - Bjartr 12y ago
    But anyone can fork any project on github willy-nilly, for whatever reason they want, it's not something that's really under the control of original project.
    - djhworld 12y ago
      True but they're all diverging away from one another, all under the "goamz" name
      - Bjartr 12y ago
        like a great many projects on github. Popular/mainstream projects on github have thousands of forks.
mrKlin 12y ago
This is very promising for the Go future as more mature packages will be available from big companies
namelezz 12y ago
Recently I have heard so many good stories about Go although my impression of Go gradually decreases due to unstable development tools and unfriendly debugging tools. FYI, I use IntelliJ to write Go.
- zik 12y ago
  I used to use IntelliJ with the Go plugin but I've recently switched to LiteIDE and I find it a significant improvement.
- wtf_is_up 12y ago
  Sublime Text + GoSublime plugin is my favorite setup.
  - namelezz 12y ago
    Sublime Text + GoSublime was my first try but then I kept getting //GsDoc: no docs found. At that time, my solution was using Sublime Text + GoSublime + Godef. I still could not find a good debugging tool though. Godef: https://github.com/buaazp/Godef https://github.com/buaazp/Godef Others also have this issue with GoSublime http://stackoverflow.com/questions/21832288/gosublime-go-to-definition http://stackoverflow.com/questions/21832288/gosublime-go-to-...
  - sanatgersappa 12y ago
    I used that. These days, I use Atom + Go-Plus.
fideloper 12y ago
I'd love to see their CLI tool converted to Go - not because I have anything against Python, but because installing Go binaries is so easy. (Would there need to be any dependencies to install?)
- artursapek 12y ago
  If you're just installing a binary, there shouldn't be.
- artifaxx 12y ago
  Good point, I look forward to seeing how this fits together. It would definitely add to the value of AWS for me.
djhworld 12y ago
Wow that moved quick, I remember reading about this library a month or so back. Glad to hear Amazon have formally adopted it as canonical!
moorage 12y ago
I maintain https://github.com/AdRoll/goamz https://github.com/AdRoll/goamz FKA https://github.com/crowdmob/goamz https://github.com/crowdmob/goamz , if anyone has any questions.
sciurus 12y ago
If this has better coverage of AWS features than their goamz fork, I hope Hashicorp will switch to it in Terraform. I'd love to use Terraform but too much of what I use Cloudformation for isn't there. https://github.com/hashicorp/terraform/issues/28 https://github.com/hashicorp/terraform/issues/28
andrewstuart 12y ago
How about a Rust SDK for AWS?

209 points by urlgrey 12y ago

This is great! I can now unify a lot of my workflow and not have to use ruby as a glue layer between applications and operations. It's great to see the Go ecosystem improving! I think in a few years, it may become the defacto tool of systems programmers.

vezzy-fnord 12y ago
It's not going to be displacing low-level programming any time soon, unless radical changes are in store for 2.x and beyond, hypothetically. That said, I can see it becoming a standard for daemons, utilities and low-level userspace in general due to its rather excellent POSIX and syscall support. I really find the (present) lack of this to be Rust's main deficiency, though the third-party crates are slowly picking up with bindings for things like prctl(), epoll() and so on. EDIT: Why the downvotes? Rob Pike has stated that he regrets attaching the "system programming" label to Go, since he never intended it for OS kernels and the like. If it's about me being wrong on Rust's POSIX bindings or Go's suitability for daemons, do prove me wrong.
- bacongobbler 12y ago
  I believe this is why you are getting downvoted: > It's not going to be displacing low-level programming any time soon, unless radical changes are in store for 2.x and beyond, hypothetically What radical changes are you referring to, and why are they needed in order for Go to be a de-facto language for system programmers?
  - eropple 12y ago
    Drop the garbage collector. Systems programming cannot stop the world.
    - jscheel 12y ago
      You can already do that: http://golang.org/pkg/runtime/ http://golang.org/pkg/runtime/
      - shanemhansen 12y ago
        That's not useful for a kernel. You can disable garbage collection in go, but go doesn't provide the tools to disable creating garbage entirely. You can get really close to creating zero garbage through profiling and helping the compiler's escape analysis, but afaik the go spec makes no guarantees about stack vs heap allocation.
        pjmlp 12y ago
        > That's not useful for a kernel. Oberon Kernel.Mod, initial version 11.4.86 http://www.inf.ethz.ch/personal/wirth/ProjectOberon/Sources/Kernel.Mod.txt http://www.inf.ethz.ch/personal/wirth/ProjectOberon/Sources/...
        shanemhansen 12y ago
        I failed to see any references to the go runtime package in that link which leads me to believe you misunderstood my statement. Using the APIs documented in the go runtime package to disable garbage collection would not be an effective memory management strategy for a kernel.
  - vezzy-fnord 12y ago
    The GC is more of a cultural issue than anything (what with Oberon being perfectly fine), but it will have an impact as a result. The runtime will be an overhead, though it does have a malleable library interface, apparently.
- pjmlp 12y ago
  Once upon a time that was a programming language called Cedar that was used at Xerox PARC to build an environment for strong typed languages similar to what Smalltalk and Interlisp-D environments offered. It used reference counting with GC for collecting cycles. A certain swiss guy decided to create a graphical workstation based on the NS32000 processor, using a revised version of Modula-2, with a GC for systems programming, named Oberon. The Oberon OS managed to provide a fully working graphics workstation capable of fulfilling many tasks at the ETHZ systems programming department. Specially the latest version, System 3 with the GUI Gadgets framework. It had as descendents EthOS, BlueBottle AOS and its design influenced Plan9's ACME and Rio. But the world was busy looking into UNIX being adopted by the industry and Oberon is no more.
  - vezzy-fnord 12y ago
    As I noted regarding Oberon, but you'll have a hard time trying to break the cultural bias against GC.
    - pjmlp 12y ago
      I learned long time ago that you just need to wait a few programmer generations, than the bias against whatever technology is solved by virtue of nature. When I started programming, the bias was against any language higher level than Assembly for home computers.
shanemhansen 12y ago
Frankly go isn't my favorite glue language (and I'm one of the relatively few people who's been doing go full time for a couple years). I currently prefer python for that kind of stuff, or sometimes bash. However it's certainly possible to use as a glue language. I've talked to some folks at DO who just install go on machines and `go run` scripts.
- bryanl 12y ago
  I'm going to go out on a limb and say that probably isn't true. All go at DigitalOcean is deployed using one of two mechanisms. Nothing is `go run` outside of a developer's workstation.
- enahs-sf 12y ago
  For small internal projects, I typically find that I end up writing the program and compiling it; It just builds up my developer toolchain from the command line to do my bidding. Also, these are all managed under source control, so my co-workers can also take advantage.

kid0m4n 12y ago

This is great news indeed. Why could this not have come 8 months ago? Back then, it was quite disheartening to see the replies from AWS team about Go support.

bkeroack 12y ago

There exists a third party library called goamz which has been forked all over the place when people have needed to add random features not found in the original version. It works well but hopefully you don't need a set of features that require you to import multiple versions. Hopefully Amazon's version will be comprehensive and idiomatic.

Rapzid 12y ago
True. The "big" players that forked it couldn't be explicatived to work together so the project became seriously fractured. I'm hoping this will move swiftly through experimental to release, and that the generated nature doesn't produce yucky to use implementations.
f2f 12y ago
the problem with goamz, when i forked it, was that merging changes into canonical's bazaar cvs was very hard to do. that was before github and pull requests became de-rigeur, so i'm a bit dated. still, very few of canonical's go projects are bzr-based anymore. that's the reason you're seeing all those forks: we all added what we needed, couldn't get it in, and gave up.

bgentry 12y ago

It's awesome that they reached out to the authors of a promising community project[1] and adopted that, rather than building their own project from scratch. AWS' first-party client libraries are generally awful, but I think this one stands a chance to be very good. Also, the current situation with goamz is truly horrific. There are multiple independent forks of the original Canonical Launchpad-hosted version, each with their own subtle differences in usage, interfaces and feature support. None of them feel like idiomatic Go. It really would be unlikely for this now AWS-supported project to end up doing worse than the status quo. [1] mainly Coda Hale from Stripe, where the project originated: https://github.com/stripe/aws-go https://github.com/stripe/aws-go

jeffbarr 12y ago
> AWS' first-party client libraries are generally awful Do you have some feedback that I can pass along to the AWS SDK team?
- jbergknoff 12y ago
  The node.js library [1] is incredibly well done. It's easily the best-documented node.js library I've ever seen. [1] https://github.com/aws/aws-sdk-js https://github.com/aws/aws-sdk-js
- omarforgotpwd 12y ago
  I like the first party libraries personally, because I find them to be very full featured.
- hoodoof 12y ago
  > AWS' first-party client libraries are generally awful Not a reasonable comment.
- bgentry 12y ago
  Sure, but going from memory on this :) As far as the Ruby SDK, my opinions are mostly based on how things were 2-3 years ago, when I wrote it off entirely after inheriting several large projects that made heavy use of it. In the past, it was difficult (impossible?) to figure out how it worked by reading the code. That's highly unusual for a Ruby library. It felt like a generated library, not like Ruby. When something wasn't working as expected (which happened often), it was difficult to figure out what was going wrong and why. As a heavy AWS user, I've felt much more comfortable using Fog, even though I only really use its AWS provider. That's true even though I didn't even get to take advantage of Fog's multi-provider abstractions (basically just using it to make raw AWS API calls). Hopefully that's all in the past and I should give it another look the next time I'm doing that stuff in Ruby. Fog also gave much better control over a lot of the details that are important when doing stuff in production, like timeouts and keep-alive settings. The 2nd example that comes to mind is the AWS Flow framework for Ruby: https://github.com/aws/aws-flow-ruby https://github.com/aws/aws-flow-ruby Flow does things like create global methods ( https://github.com/aws/aws-flow-ruby/blob/master/aws-flow/lib/aws/decider.rb#L60 https://github.com/aws/aws-flow-ruby/blob/master/aws-flow/li... ) and its DSL abstraction just doesn't feel right. It tries too hard to make it feel like you're just running Ruby code, and it's tough to determine how it uses SWF under the hood. It's such a large abstraction on top of SWF that it basically means you can't write parts of your app in anything other than the Flow frameworks for Java or Ruby. But then, the other major SWF framework I've seen (https://github.com/sclasen/swf4go https://github.com/sclasen/swf4go) also locks you in. So maybe it's just a shortcoming of SWF that it's not usable without a big abstraction layered on top of it? From what others are saying, it sounds like the newer stuff is in much better shape. That's very encouraging! Regardless, I think you've made a great decision in taking over aws-go. I have very high hopes for its future :)

marcosnils 12y ago

aws golang sdk's have been around for a while and I've found several other which are well tested and maintained such as https://github.com/goamz/goamz https://github.com/goamz/goamz. It's awesome that aws finally decide which library they're officially adopting but I found frustrating that the library is poorly tested and it will take while until it becomes the official. It'd be great if all the other sdk's contributors start working on the same project.

djhworld 12y ago
The goamz project is an embarrassing mess, so many forks.
- Bjartr 12y ago
  But anyone can fork any project on github willy-nilly, for whatever reason they want, it's not something that's really under the control of original project.
  - djhworld 12y ago
    True but they're all diverging away from one another, all under the "goamz" name
    - Bjartr 12y ago
      like a great many projects on github. Popular/mainstream projects on github have thousands of forks.

mrKlin 12y ago

This is very promising for the Go future as more mature packages will be available from big companies

namelezz 12y ago

Recently I have heard so many good stories about Go although my impression of Go gradually decreases due to unstable development tools and unfriendly debugging tools. FYI, I use IntelliJ to write Go.

zik 12y ago
I used to use IntelliJ with the Go plugin but I've recently switched to LiteIDE and I find it a significant improvement.
wtf_is_up 12y ago
Sublime Text + GoSublime plugin is my favorite setup.
- namelezz 12y ago
  Sublime Text + GoSublime was my first try but then I kept getting //GsDoc: no docs found. At that time, my solution was using Sublime Text + GoSublime + Godef. I still could not find a good debugging tool though. Godef: https://github.com/buaazp/Godef https://github.com/buaazp/Godef Others also have this issue with GoSublime http://stackoverflow.com/questions/21832288/gosublime-go-to-definition http://stackoverflow.com/questions/21832288/gosublime-go-to-...
- sanatgersappa 12y ago
  I used that. These days, I use Atom + Go-Plus.

fideloper 12y ago

I'd love to see their CLI tool converted to Go - not because I have anything against Python, but because installing Go binaries is so easy. (Would there need to be any dependencies to install?)

artursapek 12y ago
If you're just installing a binary, there shouldn't be.
artifaxx 12y ago
Good point, I look forward to seeing how this fits together. It would definitely add to the value of AWS for me.

djhworld 12y ago

Wow that moved quick, I remember reading about this library a month or so back. Glad to hear Amazon have formally adopted it as canonical!

moorage 12y ago

I maintain https://github.com/AdRoll/goamz https://github.com/AdRoll/goamz FKA https://github.com/crowdmob/goamz https://github.com/crowdmob/goamz , if anyone has any questions.

sciurus 12y ago

If this has better coverage of AWS features than their goamz fork, I hope Hashicorp will switch to it in Terraform. I'd love to use Terraform but too much of what I use Cloudformation for isn't there. https://github.com/hashicorp/terraform/issues/28 https://github.com/hashicorp/terraform/issues/28

andrewstuart 12y ago

How about a Rust SDK for AWS?