4 ms·

Fabric Python with Cleaner API and Parallel Deployment

183 points by tav 16y ago

jedsmith 16y ago
I've just started playing with Fabric, and I like your take on its functionality better. If they're resistant to merge you, why not fork into something new? You definitely have something very useful there, from a casual read.
- tav 16y ago
  Whilst forking is always a possibility, I'd rather contribute back to the project and help improve Fabric collectively. The community is made up of really nice guys and it'd be great if we could figure out a way of getting the various ideas to work together. Thanks for checking out the docs and your encouraging words!
  - _Fil_ 16y ago
    Jeff mentioned on twitter that he's keen on merging some of the stuff post 1.0.
    - tswicegood 16y ago
      Yes. Those of us who have been in the Fabric community working with one another are aware of this. From my conversations on IRC with @tav, I thought he was as well. It's always encouraging to see new people get involved. It's quite another to see someone relatively new to the community try to go around that community by announcing "new features" for a project they've barely started work on. Sorry if that comes off a little bitter, but seeing this as the top post on HN today rubbed me the wrong way.
rch 16y ago
Plus one for the task decorator. Very nice work all around.
tswicegood 16y ago
Just a quick note: these features are definitely experimental. The @task decorator has been in stable forks of Fabric for over a year at this point (this is the 3rd, if not 4th independent implementation of it). Likewise, parallel tasks have also been relatively stable for quite a long time.
- jsdalton 16y ago
  Good to hear, hopefully the task decorator will make it in soon because I think it's a much superior API. The little hack on Fabric I use currently takes the opposite approach -- I use an @internal decorator that hides certain functions from the task list but keeps the default Fabric behavior around for anything else. It's just a few lines of codes that mess with a private list Fabric uses to track internal functions: from fabric.main import _internals def internal(fn): _internals.append(fn) return fn _internals.append(internal)
  - tswicegood 16y ago
    That's an interesting idea. I like the idea of turning things off too. In some cases, that's more useful than explicitly defining a @task. Care to open a ticket on code.fabfile.org with it? Maybe we can work that in as an option in 1.1 (when @task is set to hit).
    - jsdalton 16y ago
      Sure, I'll try to file one later so it can at least be considered. What I was trying to say in my earlier comment was that I would actually prefer the explicit @task decorator be part of the official Fabric API because it's more clear and, well, explicit. I prefer my @internal decorator as a hack in the meantime because I imagine it's less surprising to someone who might stumble upon my fabfile without any explanation from me.
    - bitprophet_ 16y ago
      Yea, definitely add a link to that in http://code.fabfile.org/issues/show/76 http://code.fabfile.org/issues/show/76 please! Might be nice to give people even more control. I think @task is "enough" but if we can have a richer API while not overburdening things, could be a plus.
sophacles 16y ago
While I like a lot of the stuff you have incorporated into your fabric, it seems to me that you are trying to take credit for things that have existed as branches to the main codebase of fabric for quite a while. Just off the top of my head: task decorators, parallel execution, better host management and command-line task listing all exist in other people's patches. If you have redone the code, good for you, but at least mention that the ideas and prototype implementations aren't yours. (I don't think a claim to ignorance of such is really that valid either, since the Redmine for the project has tickets for most of the things you've done, and has links to the branches containing them). As to getting stuff into fabric: a very valid complaint (and one I have myself, since like a lot of people I run my own hacked version of fabric) is that fabric development moves very very slowly. There are several different feature branches in several people's code-bases that could have been released 6 or more months ago. Perhaps if you start pulling other patches that have a lot of functionality complementary to yours, you could have a product good enough to push development on the mainline too (ala gcc/egcs of old...)
- bitprophet_ 16y ago
  At this point I'm pretty sure I've learned my lesson about release timing (I've also signed on with a job which should give me more time to work on Fabric, since they use it daily). Definitely deserve all the flak I get for being slow, though.
- tav 16y ago
  Let me get this right. Due to convergent needs, dozens of Fabric users post various ideas to the issue tracker and mailing lists. And over the years, some people hack together personal branches solving some of these issues and, despite the incredible amount of time that's passed, they fail to get their work into the mainline branch. Eventually, one hacker comes along — who, having been a Fabric user for a long time, has independently experienced similar needs and finding no decent solution, decides to take on the task of implementing the various features himself — making sure that it's clean, backwards compatible and fully documented. And your response is to accuse him of trying to take credit for other people's work?! I can understand your frustrations with the pace of Fabric development, but accusing another hacker of taking credit for other people's work simply beggars belief. Even more incredible is that so many people decided to upvote you! For the record, I'm not taking credit for anyone else's work. I came up with the various ideas myself and you can look at the code to see that it's not based on anyone else's work. Thank you.
_Fil_ 16y ago
There is definitely some cool stuff in there. I particularly like the @task decorator, the directory support, the context and of course the parallel tasks. Nice work tav.
askedrelic 16y ago
Cool ideas. We're just now moving from a 1 machine deployment to a 4 (and probably more) machine deployment at work and I can already see the value in this.
polvi 16y ago
Another cool trick: Integrating Cloudkick tags with Fabric. Using the Cloudkick API - you can express things like "run certain_func on everything tagged 'loadbalancer'". https://support.cloudkick.com/Integrating_Fabric_with_Cloudkick https://support.cloudkick.com/Integrating_Fabric_with_Cloudk... We dog food this for all of our production deployments. For example, when we ship a new version of cloudkick.com, we just run "fab ship", which looks for all servers tagged "cloudkick.com" and runs the appropriate deployment script on each of them. Tagging can be done through the CK UI, automatically with the agent, or with things like puppet or chef. This allows you to fully automate deployment and other tasks around groups of servers.
- askedrelic 16y ago
  I figured out something like that using the Python Linode API last week: http://asktherelic.com/2011/02/17/dynamic-fabric-commands-for-managing-cloud-servers/ http://asktherelic.com/2011/02/17/dynamic-fabric-commands-fo...
bitprophet_ 16y ago
Fabric maintainer here -- Tav's been quite active on our IRC channel while developing much of this, and it's great to have more people excited about improving the tool! As tswicegood says in another comment, some of these changes (@task, parallelism, etc) aren't unique to Tav's work, so when it comes time to merge we'll draw on the various implementations to see what feels best. The features unique to his fork will be evaluated like any other contribution and merged in if they fit well and are generally useful (which many seem to be). The plan right now is post-1.0 (1.0 is scheduled for Pycon or earlier) to round up a lot of major contributions and get them into core, so people like tswicegood don't have to work off of their own forks quite as much :)
simonz05 16y ago
I cant see how adding such a wast amount of features makes the API cleaner. You made the library more complex than what it already was. Fixing Fabric should in my opinion be done, not in a backwards compatible way, but by forking it and taking away all the crap you don't want, then build a cleaner API. Adding more features to something which already is bad doesn't make it better.
ghc 16y ago
This looks nice, but I have to be honest...I'm not going to switch from the version of fabric I use to this. Why? Even though I do have to deploy to more than 30 hosts at once, the thought of the time that would need to be spent verifying that there are no bugs in this would make it impractical to switch. I have to imagine there are others like me who feel the same way. It's a shame that the people this would most benefit are probably the least likely to switch.
- bitprophet_ 16y ago
  I don't believe Tav posted this as an actual fork, but simply highlighting his branches/patches, in the same way that many other Fabric users have code waiting to be merged into core. When the time comes to merge/incorporate them, we will definitely be wanting peoples' opinions on whether they feel the changes would fit into core Fabric :)
cagenut 16y ago
I admit to substantial ignorance here, but can anyone who's using fabric, or for whom these patches to fabric are an important improvement, speak to any kind of pro/con comparison they did with func (https://fedorahosted.org/func/ https://fedorahosted.org/func/)? I've been using func for parallel deployment to 15 - 20 servers for about a year now, but I have to admit I didn't really do any research into other options at the time. Its written by the guys who wrote yum and cobbler, which are already core to my setup, so it seemed natural to just stay in their ecosystem. Thing is I never hear about anyone else using it, so I'm curious if anyone did more diligence than I and had specific reasons of choosing one or the other.
- mccutchen 16y ago
  I had not heard of func until this thread. I chose fabric because it was the only Python-based option I was aware of.
- foljs 16y ago
  Em, func seems Fedora-only...
traviscline 16y ago
Did some filter-branching and rebasing to get tav's work on top of current master: https://github.com/traviscline/fabric/tree/tav https://github.com/traviscline/fabric/tree/tav
encbladexp 16y ago
The @task decorator is a real good idea. Many good changes in 1.0. The only thing i miss is a virtualenv context manager (with virtualenv(blupp):...).
carlhu 16y ago
Beautiful work. I've always felt that Fabric strikes a effective balance between simplicity and flexibility and love how you're extending it without breaking its spirit.

183 points by tav 16y ago

I've just started playing with Fabric, and I like your take on its functionality better. If they're resistant to merge you, why not fork into something new? You definitely have something very useful there, from a casual read.

tav 16y ago
Whilst forking is always a possibility, I'd rather contribute back to the project and help improve Fabric collectively. The community is made up of really nice guys and it'd be great if we could figure out a way of getting the various ideas to work together. Thanks for checking out the docs and your encouraging words!
- _Fil_ 16y ago
  Jeff mentioned on twitter that he's keen on merging some of the stuff post 1.0.
  - tswicegood 16y ago
    Yes. Those of us who have been in the Fabric community working with one another are aware of this. From my conversations on IRC with @tav, I thought he was as well. It's always encouraging to see new people get involved. It's quite another to see someone relatively new to the community try to go around that community by announcing "new features" for a project they've barely started work on. Sorry if that comes off a little bitter, but seeing this as the top post on HN today rubbed me the wrong way.

rch 16y ago

Plus one for the task decorator. Very nice work all around.

tswicegood 16y ago

Just a quick note: these features are definitely experimental. The @task decorator has been in stable forks of Fabric for over a year at this point (this is the 3rd, if not 4th independent implementation of it). Likewise, parallel tasks have also been relatively stable for quite a long time.

jsdalton 16y ago
Good to hear, hopefully the task decorator will make it in soon because I think it's a much superior API. The little hack on Fabric I use currently takes the opposite approach -- I use an @internal decorator that hides certain functions from the task list but keeps the default Fabric behavior around for anything else. It's just a few lines of codes that mess with a private list Fabric uses to track internal functions: from fabric.main import _internals def internal(fn): _internals.append(fn) return fn _internals.append(internal)
- tswicegood 16y ago
  That's an interesting idea. I like the idea of turning things off too. In some cases, that's more useful than explicitly defining a @task. Care to open a ticket on code.fabfile.org with it? Maybe we can work that in as an option in 1.1 (when @task is set to hit).
  - jsdalton 16y ago
    Sure, I'll try to file one later so it can at least be considered. What I was trying to say in my earlier comment was that I would actually prefer the explicit @task decorator be part of the official Fabric API because it's more clear and, well, explicit. I prefer my @internal decorator as a hack in the meantime because I imagine it's less surprising to someone who might stumble upon my fabfile without any explanation from me.
  - bitprophet_ 16y ago
    Yea, definitely add a link to that in http://code.fabfile.org/issues/show/76 http://code.fabfile.org/issues/show/76 please! Might be nice to give people even more control. I think @task is "enough" but if we can have a richer API while not overburdening things, could be a plus.

sophacles 16y ago

While I like a lot of the stuff you have incorporated into your fabric, it seems to me that you are trying to take credit for things that have existed as branches to the main codebase of fabric for quite a while. Just off the top of my head: task decorators, parallel execution, better host management and command-line task listing all exist in other people's patches. If you have redone the code, good for you, but at least mention that the ideas and prototype implementations aren't yours. (I don't think a claim to ignorance of such is really that valid either, since the Redmine for the project has tickets for most of the things you've done, and has links to the branches containing them). As to getting stuff into fabric: a very valid complaint (and one I have myself, since like a lot of people I run my own hacked version of fabric) is that fabric development moves very very slowly. There are several different feature branches in several people's code-bases that could have been released 6 or more months ago. Perhaps if you start pulling other patches that have a lot of functionality complementary to yours, you could have a product good enough to push development on the mainline too (ala gcc/egcs of old...)

bitprophet_ 16y ago
At this point I'm pretty sure I've learned my lesson about release timing (I've also signed on with a job which should give me more time to work on Fabric, since they use it daily). Definitely deserve all the flak I get for being slow, though.
tav 16y ago
Let me get this right. Due to convergent needs, dozens of Fabric users post various ideas to the issue tracker and mailing lists. And over the years, some people hack together personal branches solving some of these issues and, despite the incredible amount of time that's passed, they fail to get their work into the mainline branch. Eventually, one hacker comes along — who, having been a Fabric user for a long time, has independently experienced similar needs and finding no decent solution, decides to take on the task of implementing the various features himself — making sure that it's clean, backwards compatible and fully documented. And your response is to accuse him of trying to take credit for other people's work?! I can understand your frustrations with the pace of Fabric development, but accusing another hacker of taking credit for other people's work simply beggars belief. Even more incredible is that so many people decided to upvote you! For the record, I'm not taking credit for anyone else's work. I came up with the various ideas myself and you can look at the code to see that it's not based on anyone else's work. Thank you.

_Fil_ 16y ago

There is definitely some cool stuff in there. I particularly like the @task decorator, the directory support, the context and of course the parallel tasks. Nice work tav.

askedrelic 16y ago

Cool ideas. We're just now moving from a 1 machine deployment to a 4 (and probably more) machine deployment at work and I can already see the value in this.

polvi 16y ago

Another cool trick: Integrating Cloudkick tags with Fabric. Using the Cloudkick API - you can express things like "run certain_func on everything tagged 'loadbalancer'". https://support.cloudkick.com/Integrating_Fabric_with_Cloudkick https://support.cloudkick.com/Integrating_Fabric_with_Cloudk... We dog food this for all of our production deployments. For example, when we ship a new version of cloudkick.com, we just run "fab ship", which looks for all servers tagged "cloudkick.com" and runs the appropriate deployment script on each of them. Tagging can be done through the CK UI, automatically with the agent, or with things like puppet or chef. This allows you to fully automate deployment and other tasks around groups of servers.

askedrelic 16y ago
I figured out something like that using the Python Linode API last week: http://asktherelic.com/2011/02/17/dynamic-fabric-commands-for-managing-cloud-servers/ http://asktherelic.com/2011/02/17/dynamic-fabric-commands-fo...

bitprophet_ 16y ago

Fabric maintainer here -- Tav's been quite active on our IRC channel while developing much of this, and it's great to have more people excited about improving the tool! As tswicegood says in another comment, some of these changes (@task, parallelism, etc) aren't unique to Tav's work, so when it comes time to merge we'll draw on the various implementations to see what feels best. The features unique to his fork will be evaluated like any other contribution and merged in if they fit well and are generally useful (which many seem to be). The plan right now is post-1.0 (1.0 is scheduled for Pycon or earlier) to round up a lot of major contributions and get them into core, so people like tswicegood don't have to work off of their own forks quite as much :)

simonz05 16y ago

I cant see how adding such a wast amount of features makes the API cleaner. You made the library more complex than what it already was. Fixing Fabric should in my opinion be done, not in a backwards compatible way, but by forking it and taking away all the crap you don't want, then build a cleaner API. Adding more features to something which already is bad doesn't make it better.

ghc 16y ago

This looks nice, but I have to be honest...I'm not going to switch from the version of fabric I use to this. Why? Even though I do have to deploy to more than 30 hosts at once, the thought of the time that would need to be spent verifying that there are no bugs in this would make it impractical to switch. I have to imagine there are others like me who feel the same way. It's a shame that the people this would most benefit are probably the least likely to switch.

bitprophet_ 16y ago
I don't believe Tav posted this as an actual fork, but simply highlighting his branches/patches, in the same way that many other Fabric users have code waiting to be merged into core. When the time comes to merge/incorporate them, we will definitely be wanting peoples' opinions on whether they feel the changes would fit into core Fabric :)

cagenut 16y ago

I admit to substantial ignorance here, but can anyone who's using fabric, or for whom these patches to fabric are an important improvement, speak to any kind of pro/con comparison they did with func (https://fedorahosted.org/func/ https://fedorahosted.org/func/)? I've been using func for parallel deployment to 15 - 20 servers for about a year now, but I have to admit I didn't really do any research into other options at the time. Its written by the guys who wrote yum and cobbler, which are already core to my setup, so it seemed natural to just stay in their ecosystem. Thing is I never hear about anyone else using it, so I'm curious if anyone did more diligence than I and had specific reasons of choosing one or the other.

mccutchen 16y ago
I had not heard of func until this thread. I chose fabric because it was the only Python-based option I was aware of.
foljs 16y ago
Em, func seems Fedora-only...

traviscline 16y ago

Did some filter-branching and rebasing to get tav's work on top of current master: https://github.com/traviscline/fabric/tree/tav https://github.com/traviscline/fabric/tree/tav

encbladexp 16y ago

The @task decorator is a real good idea. Many good changes in 1.0. The only thing i miss is a virtualenv context manager (with virtualenv(blupp):...).

carlhu 16y ago

Beautiful work. I've always felt that Fabric strikes a effective balance between simplicity and flexibility and love how you're extending it without breaking its spirit.