8 ms·

Gophercloud: A Multi-cloud Software Development Kit for Go

61 points by jnoller 13y ago

howeyc 13y ago
Interesting. Browsing the code it looks like they are using closures[1] in a way I haven't seen other Go code use them. Similar to the "using" keyword in .NET or the "with-" macros (with-open-file) in Common Lisp. [1] https://github.com/rackspace/gophercloud/blob/master/acceptance/04-create-server.go https://github.com/rackspace/gophercloud/blob/master/accepta...
- zemo 13y ago
  You don't see it because it's not really idiomatic Go code. Another way to do this would be to define a type that is a function type, and then to put a method on that function type. Yes, functions can have methods in Go. This seems a bit odd at first, and it may not seem particularly important, but because a function can have a method, and an interface is just a method set, you can use a function to satisfy an interface. A more idiomatic way to accomplish the same thing would be to define the argument function as it's own type: type AccessFn func(gophercloud.AccessProvider) so now AccessFn describes any function that takes an AccessProvider as its sole parameter, and returns nothing. On this function type, define WithIdentity as a method (edited for brevity): func (fn AccessFn) WithIdentity(allowReauth bool) { acc, _ := gophercloud.Authenticate(...) fn(acc) } Inside of the function's method, you can invoke the function itself. Then you would define the actual AccessFn as some variable: fn := AccessFn(func(gophercloud.AccessProvider ) { // ... }) and then call its WithIdentity method: fn.WithIdentity(ar) And then, if you were so inclined, you could define an interface over your WithIdentity method, so that in whatever context you're in, anything that has a WithIdentity method will do. This often (but not always) leads to code that's a bit easier to read. Nested callbacks... not always the easiest to wrap your head around. shameless plug: I've written about this in more detail here, if this sort of thing interests you: http://jordanorelli.tumblr.com/post/42369331748/function-types-in-go-golang http://jordanorelli.tumblr.com/post/42369331748/function-typ...
  - jnoller 13y ago
    I shot this to Samuel the current dev lead - thank you!
  - kc5tja 13y ago
    Goodness, I'm confused. I'm going to need a lot of time to digest this. That said, much discussion about ideal way of writing Gophercloud happened on the #go-nuts IRC channel, and nobody objected to what I have now as being unidiomatic. You're the first! Deeply nested closures are definitely a problem, but I usually solve that by factoring closures out into top-level functions of their own. That said, one of Gophercloud's goals is to be as idiomatic as possible. Are you available on #go-nuts if I have questions? I'd like to play with this technique some more and get a feel for it before applying it to (and risk breaking) gophercloud's existing API.
    - kc5tja 13y ago
      Oh, I'm thick! I see what's happening now! The problem is it doesn't produce any gain in clarity for the purposes I'm using it for. I think for the acceptance tests, it's use is fine. I can see if it were part of the public API, there's an argument that an interface could be used, but even there, I'm not sure it'd aid readability. Semantically, however, they're of course identical.
      - zemo 13y ago
        yeah, to be honest, it may have been eggregious of me to call how I did it idiomatic. There's a lot we just don't have idioms for. Anyway, callback chains tend to be uncommon; that's just one way to avoid it. There's a handful of ways you could write this. Generally if you can expose functionality through methods, things tend to work out better in the end since you can transition into interfaces more readily. I'm jordanorelli in #go-nuts.
kapilvt 13y ago
First there's a huge caveat to calling this multi-cloud, ie. its a library for openstack clouds only from what i gather of the src and documentation. Second, there's already a nice go library for openstack clouds that's seen multiple cloud usage (rackspace, hpcloud, openstack grizzly/folsom) in the form of http://launchpad.net/goose http://launchpad.net/goose It has significantly more coverage of available openstack services (image, compute, storage) and more realworld usage and better testing afaics.
- jnoller 13y ago
  No, there is no caveat; we're working on openstack first, then rackspace, then others. It's just started out obviously. We examined goose/juju and preferred to approach this via the Apache 2 license ala all of the other tooling/SDKs out there we (Rackspace) support and assign developers to. Yes, it is currently focused on openstack clouds, but the goal is to reach parity with Fog (ruby), jclouds (Java), Libcloud (Python), pkgcloud (node.js), and so on. The focus is a series of clean abstractions for users to be able to build tools and programs which can talk across any cloud provider easily using a common set of names / abstractions.
  - kapilvt 13y ago
    ic. Probably worth updating the readme then atm its got "Gophercloud provides a Go binding to OpenStack cloud APIs." fwiw. goose is LGPL. I'm at a bit of a toss up re LCD between clouds vs exposing the underlying functionality of the clouds. The LCD apis (libcloud from experience) break down with real world usage even for simple things like net security groups, or zones spec into provider specific usage. It seems like most cross cloud apps ends up needing to redefine the abstraction layer within itself rather than attempting to reuse the same LCD code across clouds at which point its not clear that the LCD layer isn't a limitation to taking advantage of a native cloud functionality.
    - jnoller 13y ago
      Thanks! I'll take care of it or file an issue. As for GPL vs LGPL - I am aware, but given the goals of not focusing on openstack solely and also that all of our other work is Apache 2 for maximum flexibility and freedom, we currently prefer this approach. The LCD issue is being resolve in libcloud - we have multiple FT devs working on it right now, and does not come up as an issue with Fog, jclouds, or pkgcloud. It's possible to make an abstraction that is simple enough to start but powerful enough to access the advanced features of each individual cloud host. I hope users and consumers will keep us honest - as we said in the post, our interest is pretty much no lock-in on an API level (hence: openstack) and no lock-in on the code level. We really want to make tools that allow our users and others to fluidly move between hosts.
danieldk 13y ago
Go also shares some attributes with Erlang; in particular, its “goroutine” construct makes writing concurrent software much easier than managing threads by hand, That's a lot of handwaving. The overlap is that both Go and Erlang have lightweight threads. But Erlang uses the (fairly) different actor model, which usages messaging between actors, supervision trees to control and monitor the life cycles of actors, remote actors, etc. Go has lightweight threads and channels to communicate between threads. Go (concurrency) shares very little with Erlang. It's also funny that they move on to claim that Go has better concurrency support than Java, when in fact, you can use real Erlang-like actor networks in Java by adding one dependency to your Maven file. Akka actors are lightweight concurrent entities, since you can easily create hundreds of thousands or even millions of actors. That said, it's interesting to see one API for multiple clouds. [1] http://akka.io/ http://akka.io/
- kc5tja 13y ago
  Both Erlang and Go's concurrency models are strongly influenced by Hoare's work in Communicating Sequential Processes. That certainly qualifies. You appear to be interpreting "some attributes" as a deep-dive analysis into the unique traits of what makes Erlang, well, Erlang. That was never the intention, and I'm not sure what motivated this. Actor model libraries exist in various states of maturity for Go, as well.
  - danieldk 13y ago
    Both Erlang and Go's concurrency models are strongly influenced by Hoare's work in Communicating Sequential Processes. That certainly qualifies. Erlang uses the actor model, while Go uses (a modified version of) CSP. Earlier versions of CSP were more like the actor model in that they did not introduce a level of indirection (the channel). But it's not the same concurrency model. You appear to be interpreting "some attributes" as a deep-dive analysis into the unique traits of what makes Erlang, What else am I supposed to do? Otherwise, it would be a statement as meaningless as 'Haskell shares some properties with Algol 68'. I am pretty sure it does. That was never the intention, and I'm not sure what motivated this. Erlang is well-known for its concurrency features and the reliability provided by the actor model (including supervision, etc.) (e.g. in telephone systems). It is quite easy to read the quoted sentence as an attempt to give a good reputation to Go by comparing it to Erlang. I only question if such a comparison is warranted. Actor model libraries exist in various states of maturity for Go, as well. Well, if they are available in various states of maturity, I am interested to know which one is the most mature. Does it cover most of the functionality of Erlang and Akka? (The most interesting package that I found some time ago was a binding against the Erlang C nodes functionality.)
bebna 13y ago
And I hoped for cheap gopher:// hosting.
- jnoller 13y ago
  Will pass to product management. Could be possible.
  - ggoodale 13y ago
    Seems like you've got a WAIS to go.
gtaylor 13y ago
I think it's a noble effort, but I'd be strongly concerned with the ease of use/digestion/maintainability of something that strives to cover most major cloud APIs. This stuff moves fast, and I can't imagine having to try to keep up with multiple larger platforms. While we were getting Pathwright launched, we contributed a whole lot to boto, which is primarily (but not exclusively) catered to AWS. It grew Eucalyptus (not a huge jump) and Google Compute support, but they were badly documented/tested and still seem to see much less maintenance than the AWS stuff. The situation has improved slightly as Amazon has hired a few more guys to help the maintainer make boto awesome, but they're still playing catchup with the core competency (AWS). This isn't to talk badly of boto, it's more to illustrate the magnitude of the task of keeping up with even one aggressively developing vendor. I worry that these "mega-cloud-libs" are the wrong approach. The OpenStack stuff will probably end up with excellent support, but the others are likely to see much less attention, documentation, maintenance, help. The neglected areas of the codebase get to be embarrassing and not up to the quality standards of the primary focus (OpenStack in this case). Perhaps they end up falling behind in API versions for some vendors, limiting their utility. This is definitely not to poo-poo the effort, I think it's a really neat ideal. Personally, I'd love to see them focus on making a kick ass OpenStack Go API with top notch documentation and tests, and leave the other services to their respective communities. That alone would be noteworthy.
- jnoller 13y ago
  Thank you for the feedback - there is definitely a risk of lowest common denominator, and some truth that a generalized toolkit won't be as "good" as a domain-specific tool. For openstack specifically there's goose/juju right now. We (rackspace) have had great success with Fog, jclouds, pkgcloud, libcloud, and other multi-cloud toolkits. For openstack / Rackspace cloud we also have php-opencloud, openstack.net and pyrax (which uses the python-XXXX clients from openstack core). It definitely is a fine line to walk; a good abstraction though means that you can use the most common features easily and float through providers, but you have access to the advanced features if you need or want them. Commonly those advanced features (on a provider level) mean a greater level of vendor lock-in which is why they may not make sense to expose in the higher level abstraction.
- 13y ago
icedog 13y ago
Why do I have to read six paragraphs before I find out what Gophercloud is?
- jnoller 13y ago
  We wanted to do a quick introduction to Go ideas so that those of us "not in the know" would know the why, what and additional context.
  - icedog 13y ago
    Yes, obviously you're doing an introduction to Go. But you may want to re-evaluate your description of quick.
vanderZwan 13y ago
> by Rob Pike and Ken Thompson Why does everyone always forget Robert Griesemer? He might not be as involved with the public as Rob Pike, but he's the guy that made gofmt and godoc (and I'm pretty sure he's also at least partially to thank for the language being specified in such a way that it is "gofmt"-able in the first place, considering he maintains the spec). That alone counts as a pretty big positive influence on the language.
- gecampbell 13y ago
  Thanks for that. I'm new to Go myself, and that's some helpful info.

61 points by jnoller 13y ago

Interesting. Browsing the code it looks like they are using closures[1] in a way I haven't seen other Go code use them. Similar to the "using" keyword in .NET or the "with-" macros (with-open-file) in Common Lisp. [1] https://github.com/rackspace/gophercloud/blob/master/acceptance/04-create-server.go https://github.com/rackspace/gophercloud/blob/master/accepta...

zemo 13y ago
You don't see it because it's not really idiomatic Go code. Another way to do this would be to define a type that is a function type, and then to put a method on that function type. Yes, functions can have methods in Go. This seems a bit odd at first, and it may not seem particularly important, but because a function can have a method, and an interface is just a method set, you can use a function to satisfy an interface. A more idiomatic way to accomplish the same thing would be to define the argument function as it's own type: type AccessFn func(gophercloud.AccessProvider) so now AccessFn describes any function that takes an AccessProvider as its sole parameter, and returns nothing. On this function type, define WithIdentity as a method (edited for brevity): func (fn AccessFn) WithIdentity(allowReauth bool) { acc, _ := gophercloud.Authenticate(...) fn(acc) } Inside of the function's method, you can invoke the function itself. Then you would define the actual AccessFn as some variable: fn := AccessFn(func(gophercloud.AccessProvider ) { // ... }) and then call its WithIdentity method: fn.WithIdentity(ar) And then, if you were so inclined, you could define an interface over your WithIdentity method, so that in whatever context you're in, anything that has a WithIdentity method will do. This often (but not always) leads to code that's a bit easier to read. Nested callbacks... not always the easiest to wrap your head around. shameless plug: I've written about this in more detail here, if this sort of thing interests you: http://jordanorelli.tumblr.com/post/42369331748/function-types-in-go-golang http://jordanorelli.tumblr.com/post/42369331748/function-typ...
- jnoller 13y ago
  I shot this to Samuel the current dev lead - thank you!
- kc5tja 13y ago
  Goodness, I'm confused. I'm going to need a lot of time to digest this. That said, much discussion about ideal way of writing Gophercloud happened on the #go-nuts IRC channel, and nobody objected to what I have now as being unidiomatic. You're the first! Deeply nested closures are definitely a problem, but I usually solve that by factoring closures out into top-level functions of their own. That said, one of Gophercloud's goals is to be as idiomatic as possible. Are you available on #go-nuts if I have questions? I'd like to play with this technique some more and get a feel for it before applying it to (and risk breaking) gophercloud's existing API.
  - kc5tja 13y ago
    Oh, I'm thick! I see what's happening now! The problem is it doesn't produce any gain in clarity for the purposes I'm using it for. I think for the acceptance tests, it's use is fine. I can see if it were part of the public API, there's an argument that an interface could be used, but even there, I'm not sure it'd aid readability. Semantically, however, they're of course identical.
    - zemo 13y ago
      yeah, to be honest, it may have been eggregious of me to call how I did it idiomatic. There's a lot we just don't have idioms for. Anyway, callback chains tend to be uncommon; that's just one way to avoid it. There's a handful of ways you could write this. Generally if you can expose functionality through methods, things tend to work out better in the end since you can transition into interfaces more readily. I'm jordanorelli in #go-nuts.

kapilvt 13y ago

First there's a huge caveat to calling this multi-cloud, ie. its a library for openstack clouds only from what i gather of the src and documentation. Second, there's already a nice go library for openstack clouds that's seen multiple cloud usage (rackspace, hpcloud, openstack grizzly/folsom) in the form of http://launchpad.net/goose http://launchpad.net/goose It has significantly more coverage of available openstack services (image, compute, storage) and more realworld usage and better testing afaics.

jnoller 13y ago
No, there is no caveat; we're working on openstack first, then rackspace, then others. It's just started out obviously. We examined goose/juju and preferred to approach this via the Apache 2 license ala all of the other tooling/SDKs out there we (Rackspace) support and assign developers to. Yes, it is currently focused on openstack clouds, but the goal is to reach parity with Fog (ruby), jclouds (Java), Libcloud (Python), pkgcloud (node.js), and so on. The focus is a series of clean abstractions for users to be able to build tools and programs which can talk across any cloud provider easily using a common set of names / abstractions.
- kapilvt 13y ago
  ic. Probably worth updating the readme then atm its got "Gophercloud provides a Go binding to OpenStack cloud APIs." fwiw. goose is LGPL. I'm at a bit of a toss up re LCD between clouds vs exposing the underlying functionality of the clouds. The LCD apis (libcloud from experience) break down with real world usage even for simple things like net security groups, or zones spec into provider specific usage. It seems like most cross cloud apps ends up needing to redefine the abstraction layer within itself rather than attempting to reuse the same LCD code across clouds at which point its not clear that the LCD layer isn't a limitation to taking advantage of a native cloud functionality.
  - jnoller 13y ago
    Thanks! I'll take care of it or file an issue. As for GPL vs LGPL - I am aware, but given the goals of not focusing on openstack solely and also that all of our other work is Apache 2 for maximum flexibility and freedom, we currently prefer this approach. The LCD issue is being resolve in libcloud - we have multiple FT devs working on it right now, and does not come up as an issue with Fog, jclouds, or pkgcloud. It's possible to make an abstraction that is simple enough to start but powerful enough to access the advanced features of each individual cloud host. I hope users and consumers will keep us honest - as we said in the post, our interest is pretty much no lock-in on an API level (hence: openstack) and no lock-in on the code level. We really want to make tools that allow our users and others to fluidly move between hosts.

danieldk 13y ago

Go also shares some attributes with Erlang; in particular, its “goroutine” construct makes writing concurrent software much easier than managing threads by hand, That's a lot of handwaving. The overlap is that both Go and Erlang have lightweight threads. But Erlang uses the (fairly) different actor model, which usages messaging between actors, supervision trees to control and monitor the life cycles of actors, remote actors, etc. Go has lightweight threads and channels to communicate between threads. Go (concurrency) shares very little with Erlang. It's also funny that they move on to claim that Go has better concurrency support than Java, when in fact, you can use real Erlang-like actor networks in Java by adding one dependency to your Maven file. Akka actors are lightweight concurrent entities, since you can easily create hundreds of thousands or even millions of actors. That said, it's interesting to see one API for multiple clouds. [1] http://akka.io/ http://akka.io/

kc5tja 13y ago
Both Erlang and Go's concurrency models are strongly influenced by Hoare's work in Communicating Sequential Processes. That certainly qualifies. You appear to be interpreting "some attributes" as a deep-dive analysis into the unique traits of what makes Erlang, well, Erlang. That was never the intention, and I'm not sure what motivated this. Actor model libraries exist in various states of maturity for Go, as well.
- danieldk 13y ago
  Both Erlang and Go's concurrency models are strongly influenced by Hoare's work in Communicating Sequential Processes. That certainly qualifies. Erlang uses the actor model, while Go uses (a modified version of) CSP. Earlier versions of CSP were more like the actor model in that they did not introduce a level of indirection (the channel). But it's not the same concurrency model. You appear to be interpreting "some attributes" as a deep-dive analysis into the unique traits of what makes Erlang, What else am I supposed to do? Otherwise, it would be a statement as meaningless as 'Haskell shares some properties with Algol 68'. I am pretty sure it does. That was never the intention, and I'm not sure what motivated this. Erlang is well-known for its concurrency features and the reliability provided by the actor model (including supervision, etc.) (e.g. in telephone systems). It is quite easy to read the quoted sentence as an attempt to give a good reputation to Go by comparing it to Erlang. I only question if such a comparison is warranted. Actor model libraries exist in various states of maturity for Go, as well. Well, if they are available in various states of maturity, I am interested to know which one is the most mature. Does it cover most of the functionality of Erlang and Akka? (The most interesting package that I found some time ago was a binding against the Erlang C nodes functionality.)

bebna 13y ago

And I hoped for cheap gopher:// hosting.

jnoller 13y ago
Will pass to product management. Could be possible.
- ggoodale 13y ago
  Seems like you've got a WAIS to go.

gtaylor 13y ago

I think it's a noble effort, but I'd be strongly concerned with the ease of use/digestion/maintainability of something that strives to cover most major cloud APIs. This stuff moves fast, and I can't imagine having to try to keep up with multiple larger platforms. While we were getting Pathwright launched, we contributed a whole lot to boto, which is primarily (but not exclusively) catered to AWS. It grew Eucalyptus (not a huge jump) and Google Compute support, but they were badly documented/tested and still seem to see much less maintenance than the AWS stuff. The situation has improved slightly as Amazon has hired a few more guys to help the maintainer make boto awesome, but they're still playing catchup with the core competency (AWS). This isn't to talk badly of boto, it's more to illustrate the magnitude of the task of keeping up with even one aggressively developing vendor. I worry that these "mega-cloud-libs" are the wrong approach. The OpenStack stuff will probably end up with excellent support, but the others are likely to see much less attention, documentation, maintenance, help. The neglected areas of the codebase get to be embarrassing and not up to the quality standards of the primary focus (OpenStack in this case). Perhaps they end up falling behind in API versions for some vendors, limiting their utility. This is definitely not to poo-poo the effort, I think it's a really neat ideal. Personally, I'd love to see them focus on making a kick ass OpenStack Go API with top notch documentation and tests, and leave the other services to their respective communities. That alone would be noteworthy.

jnoller 13y ago
Thank you for the feedback - there is definitely a risk of lowest common denominator, and some truth that a generalized toolkit won't be as "good" as a domain-specific tool. For openstack specifically there's goose/juju right now. We (rackspace) have had great success with Fog, jclouds, pkgcloud, libcloud, and other multi-cloud toolkits. For openstack / Rackspace cloud we also have php-opencloud, openstack.net and pyrax (which uses the python-XXXX clients from openstack core). It definitely is a fine line to walk; a good abstraction though means that you can use the most common features easily and float through providers, but you have access to the advanced features if you need or want them. Commonly those advanced features (on a provider level) mean a greater level of vendor lock-in which is why they may not make sense to expose in the higher level abstraction.
13y ago

icedog 13y ago

Why do I have to read six paragraphs before I find out what Gophercloud is?

jnoller 13y ago
We wanted to do a quick introduction to Go ideas so that those of us "not in the know" would know the why, what and additional context.
- icedog 13y ago
  Yes, obviously you're doing an introduction to Go. But you may want to re-evaluate your description of quick.

vanderZwan 13y ago

> by Rob Pike and Ken Thompson Why does everyone always forget Robert Griesemer? He might not be as involved with the public as Rob Pike, but he's the guy that made gofmt and godoc (and I'm pretty sure he's also at least partially to thank for the language being specified in such a way that it is "gofmt"-able in the first place, considering he maintains the spec). That alone counts as a pretty big positive influence on the language.

gecampbell 13y ago
Thanks for that. I'm new to Go myself, and that's some helpful info.