9 ms·

Libcloud: One Interface To Rule Them All

136 points by shrikar 13y ago

1qaz2wsx3edc 13y ago
In the ruby ecosystem Fog has existed for a long time which offers a similar service: http://fog.io/ http://fog.io/ Edit: I should also say it's neat to see an apache supported project in this space, more tooling around these services are very useful.
- khc 13y ago
  jcloud recently became part of apache: https://jclouds.apache.org/ https://jclouds.apache.org/
  - moondowner 13y ago
    Link to GitHub https://github.com/jclouds https://github.com/jclouds jclouds is good for Java and Clojure projects.
- deleted 13y ago
  [deleted]
- diminish 13y ago
  I'm curious how fog stands feature-wise in comparison to Apache libcloud.
retrack 13y ago
With this new website, the libcloud community has done a great job of better documenting. The list of supported providers has also increased. we are glad to be part of the adventure with exoscale: https://libcloud.apache.org/blog/2014/01/27/libcloud-0-14-and-the-new-exoscale-driver.html https://libcloud.apache.org/blog/2014/01/27/libcloud-0-14-an...
geertj 13y ago
Each cloud has its own semantics. Unless you want to unify the semantics, you'll either end up with lowest-common denominator APIs, or cloud specific methods. What is needed is a standard for discoverable RESTful interfaces. With that, there could be a single client library for each platform/programming language to access any compliant API. The metaphor is here that i don't need a different browser to connect to Citibank's and Bank of America's online banking services. A single browser works just fine. This is the deeper meaning of RESTful, something which is overlooked by many people.
- odonnellryan 13y ago
  Some banks work only with IE...
  - ams6110 13y ago
    Hm really? I can't imagine a bank in 2014 getting away with this, at least for consumer-facing services. They are just going to write off all their mobile customers? If you're talking about B2B or other more back-end type stuff, yeah some of this is annoyingly outdated. My bank's "business bill pay" website looks like it was written in 1998 and not changed since.... it does work in FF and Chrome though, since it's just bare-bones html forms and almost no javascript.
    - odonnellryan 13y ago
      No, consumer facing websites. I work closely with a Service Desk department. I've dealt with having to get major bank websites up and going one too many times!
- angrybits 13y ago
  I have seen this example before, but to me this neglects the aspect of using human intuition to understand how to, for example, send a wire through Citibank's site vs BofA's site. Without being argumentative, I think want to understand how this applies to APIs? (i.e. How could Citibank's hypothetical RESTful API teach my code how to send a wire transfer through their system if I had been using BofA for all previous transactions?) Or am I just flat-out missing a bigger point here?
- pfraze 13y ago
  The Link header [1] works very well for this. It exports an index of links with the same semantics as the <link> element. <link rel="stylesheet" href="/styles.css"> But in the header, which means API consumers don't have to search the response body for links, and can specifically request the index with a HEAD request. Link: </styles.css>; rel="stylesheet" The reltype is pretty powerful - it enables semantic standards and lets a client make assumptions about endpoints. Since the Link header is a standard format of, you know, links, it's also very discoverable. 1. http://tools.ietf.org/html/rfc5988 http://tools.ietf.org/html/rfc5988 EDIT: hm. No idea why I got downvoted, but I tightened it up to get more to the OP's point.
ikusalic 13y ago
I hope libcloud gets some traction. The problem is, currently it's not really mature enough. The whole idea is to use one interface so you don't end up with writing special cases for different IaaS. The problem is that it simply isn't fully-featured and that the interface for different platform differs even when it's not really necessary. I've been using it to interact with AWS and OpenStack, but still end up with lots of special cases and even falling back to boto for some AWS features (e.g. CloudWatch). I really hope they fix the interface issues and cover more features of specific platforms in future.
andyzweb 13y ago
and in the darkness ...
computer 13y ago
> "Latest stable version: 0.13.2" Does that mean that they suggest I can use this in production (since it's "stable") or that I can't, since it's not at version 1 yet?
- foxhop 13y ago
  It is just a version schema. A version is dictated "stable" when tests pass or confidence in the code at that release is high. They could have easily started with version 1.0.0 and currently be on version 1.13.2. The version number doesn't directly dictate stability.
- tbarbugli 13y ago
  lots of projects have 0.x stable for years (then all of the sudden they go from 0.6.7 to 7.0 which is kind of funny!)
- briancurtin 13y ago
  We just released 0.14, and the next will move to 1.0. The general feeling is that it is stable, and people really like seeing "1.0".
foxhop 13y ago
I'm using salt-cloud which uses apache-libcloud. I've blogged a few times about it here - This post shows how to stand up fleet of servers with Digital Ocean and salt-cloud: http://russell.ballestrini.net/create-your-own-fleet-of-servers-with-digital-ocean-and-salt-cloud/ http://russell.ballestrini.net/create-your-own-fleet-of-serv... This post shows how I quickly setup a Sensu monitoring test environment: http://russell.ballestrini.net/sensu-salt/ http://russell.ballestrini.net/sensu-salt/
pothibo 13y ago
Not sure I'm a fan of wrapping libraries like that one. It tried to abstract things from you, but on the the other hand, you need to know the implementation details of each provider to understand the behavior of the library. Maybe I misunderstood what it does?
- kordless 13y ago
  Most wrappers exist to provide a fast and easy way to execute common methods. They likely make it trivial for most people to implement something quickly. Those who need functionality beyond what they provide probably don't care about it being easy to implement and probably won't use it. It's a choice.
nbevans 13y ago
No Azure support? Really? But has support for various minnow providers...
- briancurtin 13y ago
  Someone working on or using Azure would be more than welcomed to implement support for Azure drivers. Contributions typically come from the providers themselves, or the users of those providers.
- DamnYuppie 13y ago
  Yeah....I mean Apache and MS have always been so tight, hard to imagine them getting left out ;)
  - gtirloni 13y ago
    Microsoft is a platinum sponsor of the Apache foundation nowadays: http://www.apache.org/foundation/thanks.html http://www.apache.org/foundation/thanks.html http://www.apache.org/foundation/sponsorship.html http://www.apache.org/foundation/sponsorship.html
  - brianm 13y ago
    Oddly enough, Apache and MS have a pretty good relationship nowadays. There was definitely tension in the past, but MS is a very different company today then it was in the 90's, early 00's. I wouldn't be surprised, now that it is highlighted, to see MS submit a patch to libcloud with what is needed to support Azure.
- ogrisel 13y ago
  The Azure API is much more complex that any other providers: implementing it without introducing a dependency on the Azure Python SDK [1] is going to be complex. Just have a look at its source code (e.g. [2] and [3]) to get an idea. All drivers available in Apache Libcloud have been implemented without introducing a dependency on a custom SDK. Why make an exception for Azure? Note that the complexity does not come from the Azure Python client API but rather from the Azure REST API that is in my opinion much more complicated that any other provider's. Azure has some specific idiosyncrasies (such as deployments, certificates management, VM images that first need to be transferred to a blob container) that make it harder to fit in the simple and unified Libcloud API. [1] https://github.com/WindowsAzure/azure-sdk-for-python https://github.com/WindowsAzure/azure-sdk-for-python [2] https://github.com/WindowsAzure/azure-sdk-for-python/blob/master/azure/servicemanagement/__init__.py https://github.com/WindowsAzure/azure-sdk-for-python/blob/ma... [3] https://github.com/WindowsAzure/azure-sdk-for-python/blob/master/azure/servicemanagement/servicemanagementservice.py https://github.com/WindowsAzure/azure-sdk-for-python/blob/ma...
tbarbugli 13y ago
sounds like a really ambitious project to me, I cant do anything else then wish the dev team good luck! (and use boto meanwhile the product matures :))
dmourati 13y ago
Honestly the one interface should be the AWS API. Eucalyptus has already embraced it and set the tone. OpenStack is partially there but political battles in the community have forced a stalemate. If you standardize on the lowest common denominator, you lose many of the key features of AWS such as VPC and autoscaling.
- comice 13y ago
  If you standardize on AWS you lose many key features of other systems, and you put AWS in control of your future.
- justinsb 13y ago
  This is a long-standing argument in OpenStack. The root problem is that the AWS API exposes only what AWS exposes (trivial example: no live-migration.) The AWS API precludes features better than AWS, which is a big driver for some private clouds. But there's no stalemate in OpenStack: it supports both APIs, although the EC2 API definitely gets less love. If you improve the EC2 API implementation, those improvements will get merged subject to the normal code review. If you want to do an ecosystem project that proxies EC2 calls to OpenStack, there are obvious questions over code duplication, but there's not profound opposition. And if you do want to do a proxy, go right ahead; you don't need OpenStack's buy in at all: build it, make it better than what is there already, and you'll win the argument. The root problem (as with all open source) is that there are are a lot more people saying they want others to build better EC2 compatability, than there are people saying they want to build it.
- kordless 13y ago
  The AWS API is an INTERFACE to a proprietary system which runs closed source code. The code that runs the API endpoints is also closed source. Whether or not the SDKs for the AWS API are open or not is irrelevant to this discussion (not that you said something to the contrary, but I'm covering my bases here). Eucalyptus has no more or less 'embraced' the AWS API methods than OpenStack has - save for a bare few methods that have different connotations in OpenStack land than they do with Euca: https://wiki.openstack.org/wiki/Nova/APIFeatureComparison https://wiki.openstack.org/wiki/Nova/APIFeatureComparison. The 'political battles' you are citing in OpenStack are actually a bunch of cage rattlers who are promoting their own interests in their respective service offerings. Going into a company that uses AWS for infrastructure and trying to pitch them on OpenStack is a long, drawn out process. Selling new technology is hard. The lowest common denominator benefit that OpenStack provides is an open and transparent code base that can be modified and utilized by anyone. It's that openness that allows the few people using features like VPC and autoscaling to write some simple code to do deployments on whatever they like. Honestly, it's a handful of lines of code at the most for any given method. Saying you 'lose' feature functionality is simply a rationalization for attacking the OpenStack movement.
deleted 13y ago
[deleted]
stackcollision 13y ago
Did we learn nothing from LOTR?
bachback 13y ago
cool. how does this relate to packer & vagrant? this is basically the trend to an abstract machine interface. so you can boot an image at several place and you want notice. this will matter in terms of price and consumer power.
bachback 13y ago
repo: https://github.com/apache/libcloud https://github.com/apache/libcloud
contingencies 13y ago
I agree that the result of these APIs can be a dumbing down of your cloud infrastructure. IMHO, the major challenges inherent in having multiple cloud providers with differing APIs is absolutely not writing code to use them, rather: - infrastructure management - workflow process security - credential security - offline workflow (local emulation) support - cost comparisons - performance observation / guarantees - SLA - recognition of disparate legal jurisdictions where required - managing nontrivial inter-service build and live dependencies - complex or non-standard network topology support (ie. not "single default route to the internet" or "single default route to the internet, secondary route to cloud provider specific renamed VLAN concept") - embedding real time failover Keeping all of the above out of the way of regular programmers who just want to write cloud provider neutral services is the real challenge. None of these libcloud/libvirt type solutions ever target the above, which mostly border on operations concerns. I am now working on the second functional prototype of a system that I believe goes a long way toward resolving these issues by taking a broader, more operations-centric perspective on the evolving norm of multi-provider cloud infrastructure within companies that may include remote developers, require offline development support, and still need to maintain higher standards of trust and security. Those curious can browse the presently rather obtuse documentation at http://stani.sh/walter/cims/ http://stani.sh/walter/cims/ .. I am considering asking my company to allow me to open source it. Right now we have storage providers for LVM and ZFS, and support for an internal, high availability corosync/pacemaker cluster + LXC based cloud provider in addition to a range of popular commercial ones. Very interested in feedback and/or experienced/motivated help.

136 points by shrikar 13y ago

In the ruby ecosystem Fog has existed for a long time which offers a similar service: http://fog.io/ http://fog.io/ Edit: I should also say it's neat to see an apache supported project in this space, more tooling around these services are very useful.

khc 13y ago
jcloud recently became part of apache: https://jclouds.apache.org/ https://jclouds.apache.org/
- moondowner 13y ago
  Link to GitHub https://github.com/jclouds https://github.com/jclouds jclouds is good for Java and Clojure projects.
deleted 13y ago
[deleted]
diminish 13y ago
I'm curious how fog stands feature-wise in comparison to Apache libcloud.

retrack 13y ago

With this new website, the libcloud community has done a great job of better documenting. The list of supported providers has also increased. we are glad to be part of the adventure with exoscale: https://libcloud.apache.org/blog/2014/01/27/libcloud-0-14-and-the-new-exoscale-driver.html https://libcloud.apache.org/blog/2014/01/27/libcloud-0-14-an...

geertj 13y ago

Each cloud has its own semantics. Unless you want to unify the semantics, you'll either end up with lowest-common denominator APIs, or cloud specific methods. What is needed is a standard for discoverable RESTful interfaces. With that, there could be a single client library for each platform/programming language to access any compliant API. The metaphor is here that i don't need a different browser to connect to Citibank's and Bank of America's online banking services. A single browser works just fine. This is the deeper meaning of RESTful, something which is overlooked by many people.

odonnellryan 13y ago
Some banks work only with IE...
- ams6110 13y ago
  Hm really? I can't imagine a bank in 2014 getting away with this, at least for consumer-facing services. They are just going to write off all their mobile customers? If you're talking about B2B or other more back-end type stuff, yeah some of this is annoyingly outdated. My bank's "business bill pay" website looks like it was written in 1998 and not changed since.... it does work in FF and Chrome though, since it's just bare-bones html forms and almost no javascript.
  - odonnellryan 13y ago
    No, consumer facing websites. I work closely with a Service Desk department. I've dealt with having to get major bank websites up and going one too many times!
angrybits 13y ago
I have seen this example before, but to me this neglects the aspect of using human intuition to understand how to, for example, send a wire through Citibank's site vs BofA's site. Without being argumentative, I think want to understand how this applies to APIs? (i.e. How could Citibank's hypothetical RESTful API teach my code how to send a wire transfer through their system if I had been using BofA for all previous transactions?) Or am I just flat-out missing a bigger point here?
pfraze 13y ago
The Link header [1] works very well for this. It exports an index of links with the same semantics as the <link> element. <link rel="stylesheet" href="/styles.css"> But in the header, which means API consumers don't have to search the response body for links, and can specifically request the index with a HEAD request. Link: </styles.css>; rel="stylesheet" The reltype is pretty powerful - it enables semantic standards and lets a client make assumptions about endpoints. Since the Link header is a standard format of, you know, links, it's also very discoverable. 1. http://tools.ietf.org/html/rfc5988 http://tools.ietf.org/html/rfc5988 EDIT: hm. No idea why I got downvoted, but I tightened it up to get more to the OP's point.

ikusalic 13y ago

I hope libcloud gets some traction. The problem is, currently it's not really mature enough. The whole idea is to use one interface so you don't end up with writing special cases for different IaaS. The problem is that it simply isn't fully-featured and that the interface for different platform differs even when it's not really necessary. I've been using it to interact with AWS and OpenStack, but still end up with lots of special cases and even falling back to boto for some AWS features (e.g. CloudWatch). I really hope they fix the interface issues and cover more features of specific platforms in future.

andyzweb 13y ago

and in the darkness ...

computer 13y ago

> "Latest stable version: 0.13.2" Does that mean that they suggest I can use this in production (since it's "stable") or that I can't, since it's not at version 1 yet?

foxhop 13y ago
It is just a version schema. A version is dictated "stable" when tests pass or confidence in the code at that release is high. They could have easily started with version 1.0.0 and currently be on version 1.13.2. The version number doesn't directly dictate stability.
tbarbugli 13y ago
lots of projects have 0.x stable for years (then all of the sudden they go from 0.6.7 to 7.0 which is kind of funny!)
briancurtin 13y ago
We just released 0.14, and the next will move to 1.0. The general feeling is that it is stable, and people really like seeing "1.0".

foxhop 13y ago

I'm using salt-cloud which uses apache-libcloud. I've blogged a few times about it here - This post shows how to stand up fleet of servers with Digital Ocean and salt-cloud: http://russell.ballestrini.net/create-your-own-fleet-of-servers-with-digital-ocean-and-salt-cloud/ http://russell.ballestrini.net/create-your-own-fleet-of-serv... This post shows how I quickly setup a Sensu monitoring test environment: http://russell.ballestrini.net/sensu-salt/ http://russell.ballestrini.net/sensu-salt/

pothibo 13y ago

Not sure I'm a fan of wrapping libraries like that one. It tried to abstract things from you, but on the the other hand, you need to know the implementation details of each provider to understand the behavior of the library. Maybe I misunderstood what it does?

kordless 13y ago
Most wrappers exist to provide a fast and easy way to execute common methods. They likely make it trivial for most people to implement something quickly. Those who need functionality beyond what they provide probably don't care about it being easy to implement and probably won't use it. It's a choice.

nbevans 13y ago

No Azure support? Really? But has support for various minnow providers...

briancurtin 13y ago
Someone working on or using Azure would be more than welcomed to implement support for Azure drivers. Contributions typically come from the providers themselves, or the users of those providers.
DamnYuppie 13y ago
Yeah....I mean Apache and MS have always been so tight, hard to imagine them getting left out ;)
- gtirloni 13y ago
  Microsoft is a platinum sponsor of the Apache foundation nowadays: http://www.apache.org/foundation/thanks.html http://www.apache.org/foundation/thanks.html http://www.apache.org/foundation/sponsorship.html http://www.apache.org/foundation/sponsorship.html
- brianm 13y ago
  Oddly enough, Apache and MS have a pretty good relationship nowadays. There was definitely tension in the past, but MS is a very different company today then it was in the 90's, early 00's. I wouldn't be surprised, now that it is highlighted, to see MS submit a patch to libcloud with what is needed to support Azure.
ogrisel 13y ago
The Azure API is much more complex that any other providers: implementing it without introducing a dependency on the Azure Python SDK [1] is going to be complex. Just have a look at its source code (e.g. [2] and [3]) to get an idea. All drivers available in Apache Libcloud have been implemented without introducing a dependency on a custom SDK. Why make an exception for Azure? Note that the complexity does not come from the Azure Python client API but rather from the Azure REST API that is in my opinion much more complicated that any other provider's. Azure has some specific idiosyncrasies (such as deployments, certificates management, VM images that first need to be transferred to a blob container) that make it harder to fit in the simple and unified Libcloud API. [1] https://github.com/WindowsAzure/azure-sdk-for-python https://github.com/WindowsAzure/azure-sdk-for-python [2] https://github.com/WindowsAzure/azure-sdk-for-python/blob/master/azure/servicemanagement/__init__.py https://github.com/WindowsAzure/azure-sdk-for-python/blob/ma... [3] https://github.com/WindowsAzure/azure-sdk-for-python/blob/master/azure/servicemanagement/servicemanagementservice.py https://github.com/WindowsAzure/azure-sdk-for-python/blob/ma...

tbarbugli 13y ago

sounds like a really ambitious project to me, I cant do anything else then wish the dev team good luck! (and use boto meanwhile the product matures :))

dmourati 13y ago

Honestly the one interface should be the AWS API. Eucalyptus has already embraced it and set the tone. OpenStack is partially there but political battles in the community have forced a stalemate. If you standardize on the lowest common denominator, you lose many of the key features of AWS such as VPC and autoscaling.

comice 13y ago
If you standardize on AWS you lose many key features of other systems, and you put AWS in control of your future.
justinsb 13y ago
This is a long-standing argument in OpenStack. The root problem is that the AWS API exposes only what AWS exposes (trivial example: no live-migration.) The AWS API precludes features better than AWS, which is a big driver for some private clouds. But there's no stalemate in OpenStack: it supports both APIs, although the EC2 API definitely gets less love. If you improve the EC2 API implementation, those improvements will get merged subject to the normal code review. If you want to do an ecosystem project that proxies EC2 calls to OpenStack, there are obvious questions over code duplication, but there's not profound opposition. And if you do want to do a proxy, go right ahead; you don't need OpenStack's buy in at all: build it, make it better than what is there already, and you'll win the argument. The root problem (as with all open source) is that there are are a lot more people saying they want others to build better EC2 compatability, than there are people saying they want to build it.
kordless 13y ago
The AWS API is an INTERFACE to a proprietary system which runs closed source code. The code that runs the API endpoints is also closed source. Whether or not the SDKs for the AWS API are open or not is irrelevant to this discussion (not that you said something to the contrary, but I'm covering my bases here). Eucalyptus has no more or less 'embraced' the AWS API methods than OpenStack has - save for a bare few methods that have different connotations in OpenStack land than they do with Euca: https://wiki.openstack.org/wiki/Nova/APIFeatureComparison https://wiki.openstack.org/wiki/Nova/APIFeatureComparison. The 'political battles' you are citing in OpenStack are actually a bunch of cage rattlers who are promoting their own interests in their respective service offerings. Going into a company that uses AWS for infrastructure and trying to pitch them on OpenStack is a long, drawn out process. Selling new technology is hard. The lowest common denominator benefit that OpenStack provides is an open and transparent code base that can be modified and utilized by anyone. It's that openness that allows the few people using features like VPC and autoscaling to write some simple code to do deployments on whatever they like. Honestly, it's a handful of lines of code at the most for any given method. Saying you 'lose' feature functionality is simply a rationalization for attacking the OpenStack movement.

deleted 13y ago

[deleted]

stackcollision 13y ago

Did we learn nothing from LOTR?

bachback 13y ago

cool. how does this relate to packer & vagrant? this is basically the trend to an abstract machine interface. so you can boot an image at several place and you want notice. this will matter in terms of price and consumer power.

bachback 13y ago

repo: https://github.com/apache/libcloud https://github.com/apache/libcloud

contingencies 13y ago

I agree that the result of these APIs can be a dumbing down of your cloud infrastructure. IMHO, the major challenges inherent in having multiple cloud providers with differing APIs is absolutely not writing code to use them, rather: - infrastructure management - workflow process security - credential security - offline workflow (local emulation) support - cost comparisons - performance observation / guarantees - SLA - recognition of disparate legal jurisdictions where required - managing nontrivial inter-service build and live dependencies - complex or non-standard network topology support (ie. not "single default route to the internet" or "single default route to the internet, secondary route to cloud provider specific renamed VLAN concept") - embedding real time failover Keeping all of the above out of the way of regular programmers who just want to write cloud provider neutral services is the real challenge. None of these libcloud/libvirt type solutions ever target the above, which mostly border on operations concerns. I am now working on the second functional prototype of a system that I believe goes a long way toward resolving these issues by taking a broader, more operations-centric perspective on the evolving norm of multi-provider cloud infrastructure within companies that may include remote developers, require offline development support, and still need to maintain higher standards of trust and security. Those curious can browse the presently rather obtuse documentation at http://stani.sh/walter/cims/ http://stani.sh/walter/cims/ .. I am considering asking my company to allow me to open source it. Right now we have storage providers for LVM and ZFS, and support for an internal, high availability corosync/pacemaker cluster + LXC based cloud provider in addition to a range of popular commercial ones. Very interested in feedback and/or experienced/motivated help.