6 ms·

Spinnaker: Global Continuous Delivery by Netflix

104 points by kalehrishi 11y ago

zobzu 11y ago
Looks cool albeit i've a rant: It's "funny" that for a deployment system, its own deployment follows all this crappy standards people have been running by lately. Config in $HOME, startup with a random shell script, install is a compilation of python/bash/etc scripts that do magic, ... I mean look at this: https://github.com/spinnaker/spinnaker/blob/master/gradlew#L149 https://github.com/spinnaker/spinnaker/blob/master/gradlew#L... (or actually, read the whole script, be scared) Software nowadays.. a bunch of shell scripts with hacks all over which few actually knows how to write :/ (writing bash scripts well does take quite a bit of knowledge) /rant over, send me your downvotes.
- predakanga 11y ago
  While I agree with you generally, the linked script isn't really a fair criticism of Spinnaker. The linked script is just a standard bootstrap script, used by Gradle - I won't defend the logic behind it (they have a page for that themselves: https://docs.gradle.org/current/userguide/gradle_wrapper.html https://docs.gradle.org/current/userguide/gradle_wrapper.htm...), but you can see the same thing in just about any project's Autotools files (see https://github.com/ruby/ruby/blob/trunk/aclocal.m4 https://github.com/ruby/ruby/blob/trunk/aclocal.m4)
- techdragon 11y ago
  You get my comment and up vote instead. I'm increasingly frustrated by the discovery of major projects where this sort of "spaghetti code" is considered acceptable. Few of them deem it a priority to remediate or even just document the mess. Of particular note is the lineage of build system scripts used by first Gentoo then mutated by the chromeos team, finally used by and dramatically simplified by the coreos team who forked off from the chromeos fully and are cleaning up the mess left by the chromeos team hacking away on these bash scripts. There are dozens of places where simple hash bang python scripts would have cleaned up hundreds of lines of bash boilerplate. I'm incredibly frustrated by the lack of willingness to decide that the outcome is key and the build scripts should be a testable part of the release process and a growing number of places I find have wound up considering the build scripts part of their release tests therefore they should change as infrequently as possible. Not even going to start talking about programs that don't have stand alone build/install instructions because they always generate builds inside some kind of harness like a Debian or RPM source package.
- optimusclimb 11y ago
  While I certainly wouldn't let it be the thing I judge the project on, to your point, why is step 3 here: http://spinnaker.io/documentation/getting_started.html http://spinnaker.io/documentation/getting_started.html install the thing that's going to manage your whole deployment pipeline by curling a bash script from the 'net?
  - tomaslin 11y ago
    This is not the way it is deployed inside of Netflix. The scripts were collaborated by Kenzan and Google to make it easy to try and install Spinnaker. There is also a docker-compose configuration to make it easy to try. Internally, we deploy each microservice to a separate aws intance that is heavily monitored.
- fs111 11y ago
  That comes from the build tool and is auto generated: https://docs.gradle.org/current/userguide/gradle_wrapper.html https://docs.gradle.org/current/userguide/gradle_wrapper.htm...
- annnnd 11y ago
  Lines 140-149... I have mixed feelings about that. On one hand, the software developer in me keeps screaming "DRY! DRY! DRY!". On the other hand I find it really concise and understandable. I mean, can you look at this code and NOT know what it does? Now try to make the code better - do you still get the same effect? Probably not. (disclaimer: I am arguing about this specific case, not about such code smells in general) UPDATE: ...and if I missed something besides that the code is repeated, please correct me. Bash is not my strong point.
catshirt 11y ago
at a glance... seems nearly identical to OpsWorks. am i right? why would i use one or the other? that said, and maybe i can answer my own question... OpsWorks has been a grand pain in the ass, so new options even built on AWS come welcome.
- yeukhon 11y ago
  My reading is that this project aims to do more than just OpsWork. It basically manages your deployment and release cycle, including releasing your infrastructure changes, if you make it so. You can basically do all of this in Jenkins today, but you don't have the fancy GUI, that's my impression so far. http://spinnaker.io/online_docs/overview/overview.html http://spinnaker.io/online_docs/overview/overview.html
  - daviddumenil 11y ago
    A replacement for Jenkins as the CI Pipeline management tool is my understanding. It now makes sense why they stepped away from the Job DSL Jenkins plugin that they helped create a while back (though there's still support for Jenkins steps in there which'll be a great help with porting existing pipelines).
    - stuff4ben 11y ago
      JobDSL was more along the lines of a templating engine with groovy customization that tends to break Jenkins masters. The Workflow plugin would be more akin to Spinnaker than JobDSL.
      - daviddumenil 11y ago
        I'm not saying it's a like for like replacement, I'm saying they stopped working on JobDSL because they were working on Spinnaker. This feels like a Jenkins replacement rather than a Jenkins plugin replacement or at least something that sits atop Jenkins in driving your build and deployment activities (it's got support to call and watch Jenkins jobs so it's sensible to assume they use both). I'm interested to hear that JobDSL is painful for Jenkins masters do you know what the pain points are?
d2xdy2 11y ago
Jeeze; for a second I thought Netflix had somehow made a sentient CD platform... > http://www.artificialbrains.com/spinnaker http://www.artificialbrains.com/spinnaker
- randomsearch 11y ago
  Indeed. It's not a good sign that Netflix weren't aware of this very high-profile long-standing project with the same name. It suggests that they aren't much in touch with the academic CS community.
  - biot 11y ago
    Should names be globally unique? Spinnaker Software [0] was around in the early 1980s and can claim the throne for longest use of the name in the computer space. [0] https://en.wikipedia.org/wiki/Spinnaker_Software https://en.wikipedia.org/wiki/Spinnaker_Software
    - randomsearch 11y ago
      > Should names be globally unique? I didn't say that. I just think it's unlikely they would have chosen the name if they were aware of Manchester's project, which means they didn't know about it, which means they don't read much about CS research.
- jguimont 11y ago
  A Spinnaker is a type sail first and foremost.
  - js2 11y ago
    And also the first and foremost sail. :-)
- trextrex 11y ago
  There is also a SpiNNaker computer platform [1] [1] http://apt.cs.manchester.ac.uk/projects/SpiNNaker/ http://apt.cs.manchester.ac.uk/projects/SpiNNaker/
parasubvert 11y ago
This is Netflix's replacement for their Asgard platform: http://techblog.netflix.com/2015/09/moving-from-asgard-to-spinnaker.html http://techblog.netflix.com/2015/09/moving-from-asgard-to-sp... Also largely written in Groovy / Spring Boot. http://www.groovy-lang.org http://www.groovy-lang.org
mgreg 11y ago
Anyone know why this wasn't published in github.com/neflix as the rest of the NetFlix OSS tools are but rather in a new account at github.com/spinnaker? They've also gone through the trouble of doing more PR and setting up spinnaker.io for this.
- andyfleming 11y ago
  There was mention of partnering with Google, Microsoft, and Pivotal (in their blog post http://techblog.netflix.com/2015/11/global-continuous-delivery-with.html http://techblog.netflix.com/2015/11/global-continuous-delive... ). Maybe they wanted to have a separate github organization for easier management, etc.
andyfleming 11y ago
Also, the blog post here: http://techblog.netflix.com/2015/11/global-continuous-delivery-with.html http://techblog.netflix.com/2015/11/global-continuous-delive...
odiroot 11y ago
I really like they used Python for this sort of "system tool". Didn't know Netflix actually uses Python at all, I was expecting all of it to be Java.
- fletcherr 11y ago
  Er. It is almost all Java (well, mostly Groovy). Netflix uses a lot of Python but there's virtually none in this project.
  - pluma 11y ago
    odiroot was likely talking about this: https://github.com/spinnaker/spinnaker/search?l=python https://github.com/spinnaker/spinnaker/search?l=python
    - odiroot 11y ago
      Exactly, Github says 84.1% Python.
merb 11y ago
I have a mixed feeling about this. I mean how do you Deliver Spinnaker. The project is fucking complex, if everything goes wrong you are pretty much done..
stephen 11y ago
We had looked at Asgard, but didn't use it because it used dedicated load balancers (e.g. separate ec2 instances) instead of ELBs (IIRC). Our deployments were already using ELBs and we wanted to stay with that. Does anyone know if this has changed with Spinnaker? I would not be surprised if Spinnaker's canary/etc. features require finer-grained control over the traffic than an ELB with just course-grained "add/remove instance" gives you, so it probably makes sense.
stuff4ben 11y ago
Curious how this compares to Jenkins Workflow plugin suite. At first glance Spinnaker seems to be simpler and has a prettier UI. Workflow has a lot more customization possible and the crappy Jenkins UI to go along with it. I might look at Spinnaker anyways just to keep from gouging my eyes out on the Jenkins "UI".
mrbonner 11y ago
Is this mainly written in Groovy? I tried to look at the github repo source but only saw a bunch of install shell scripts.
- fletcherr 11y ago
  It's a bunch of microservices, look at the other projects in the org.
mikecb 11y ago
I'm interested in what this says about Netflix's plans to stay on AWS, move off of it, or simply diversify their infrastructure. Maybe' we'll see a Chaos (Whatever's bigger than Kong), that models knocking out an entire provider: i.e., don't have to rely on Amazon's WAN anymore because they also have Cloud Platform/Azure to supply things.

104 points by kalehrishi 11y ago

zobzu 11y ago

Looks cool albeit i've a rant: It's "funny" that for a deployment system, its own deployment follows all this crappy standards people have been running by lately. Config in $HOME, startup with a random shell script, install is a compilation of python/bash/etc scripts that do magic, ... I mean look at this: https://github.com/spinnaker/spinnaker/blob/master/gradlew#L149 https://github.com/spinnaker/spinnaker/blob/master/gradlew#L... (or actually, read the whole script, be scared) Software nowadays.. a bunch of shell scripts with hacks all over which few actually knows how to write :/ (writing bash scripts well does take quite a bit of knowledge) /rant over, send me your downvotes.

predakanga 11y ago
While I agree with you generally, the linked script isn't really a fair criticism of Spinnaker. The linked script is just a standard bootstrap script, used by Gradle - I won't defend the logic behind it (they have a page for that themselves: https://docs.gradle.org/current/userguide/gradle_wrapper.html https://docs.gradle.org/current/userguide/gradle_wrapper.htm...), but you can see the same thing in just about any project's Autotools files (see https://github.com/ruby/ruby/blob/trunk/aclocal.m4 https://github.com/ruby/ruby/blob/trunk/aclocal.m4)
techdragon 11y ago
You get my comment and up vote instead. I'm increasingly frustrated by the discovery of major projects where this sort of "spaghetti code" is considered acceptable. Few of them deem it a priority to remediate or even just document the mess. Of particular note is the lineage of build system scripts used by first Gentoo then mutated by the chromeos team, finally used by and dramatically simplified by the coreos team who forked off from the chromeos fully and are cleaning up the mess left by the chromeos team hacking away on these bash scripts. There are dozens of places where simple hash bang python scripts would have cleaned up hundreds of lines of bash boilerplate. I'm incredibly frustrated by the lack of willingness to decide that the outcome is key and the build scripts should be a testable part of the release process and a growing number of places I find have wound up considering the build scripts part of their release tests therefore they should change as infrequently as possible. Not even going to start talking about programs that don't have stand alone build/install instructions because they always generate builds inside some kind of harness like a Debian or RPM source package.
optimusclimb 11y ago
While I certainly wouldn't let it be the thing I judge the project on, to your point, why is step 3 here: http://spinnaker.io/documentation/getting_started.html http://spinnaker.io/documentation/getting_started.html install the thing that's going to manage your whole deployment pipeline by curling a bash script from the 'net?
- tomaslin 11y ago
  This is not the way it is deployed inside of Netflix. The scripts were collaborated by Kenzan and Google to make it easy to try and install Spinnaker. There is also a docker-compose configuration to make it easy to try. Internally, we deploy each microservice to a separate aws intance that is heavily monitored.
fs111 11y ago
That comes from the build tool and is auto generated: https://docs.gradle.org/current/userguide/gradle_wrapper.html https://docs.gradle.org/current/userguide/gradle_wrapper.htm...
annnnd 11y ago
Lines 140-149... I have mixed feelings about that. On one hand, the software developer in me keeps screaming "DRY! DRY! DRY!". On the other hand I find it really concise and understandable. I mean, can you look at this code and NOT know what it does? Now try to make the code better - do you still get the same effect? Probably not. (disclaimer: I am arguing about this specific case, not about such code smells in general) UPDATE: ...and if I missed something besides that the code is repeated, please correct me. Bash is not my strong point.

catshirt 11y ago

at a glance... seems nearly identical to OpsWorks. am i right? why would i use one or the other? that said, and maybe i can answer my own question... OpsWorks has been a grand pain in the ass, so new options even built on AWS come welcome.

yeukhon 11y ago
My reading is that this project aims to do more than just OpsWork. It basically manages your deployment and release cycle, including releasing your infrastructure changes, if you make it so. You can basically do all of this in Jenkins today, but you don't have the fancy GUI, that's my impression so far. http://spinnaker.io/online_docs/overview/overview.html http://spinnaker.io/online_docs/overview/overview.html
- daviddumenil 11y ago
  A replacement for Jenkins as the CI Pipeline management tool is my understanding. It now makes sense why they stepped away from the Job DSL Jenkins plugin that they helped create a while back (though there's still support for Jenkins steps in there which'll be a great help with porting existing pipelines).
  - stuff4ben 11y ago
    JobDSL was more along the lines of a templating engine with groovy customization that tends to break Jenkins masters. The Workflow plugin would be more akin to Spinnaker than JobDSL.
    - daviddumenil 11y ago
      I'm not saying it's a like for like replacement, I'm saying they stopped working on JobDSL because they were working on Spinnaker. This feels like a Jenkins replacement rather than a Jenkins plugin replacement or at least something that sits atop Jenkins in driving your build and deployment activities (it's got support to call and watch Jenkins jobs so it's sensible to assume they use both). I'm interested to hear that JobDSL is painful for Jenkins masters do you know what the pain points are?

d2xdy2 11y ago

Jeeze; for a second I thought Netflix had somehow made a sentient CD platform... > http://www.artificialbrains.com/spinnaker http://www.artificialbrains.com/spinnaker

randomsearch 11y ago
Indeed. It's not a good sign that Netflix weren't aware of this very high-profile long-standing project with the same name. It suggests that they aren't much in touch with the academic CS community.
- biot 11y ago
  Should names be globally unique? Spinnaker Software [0] was around in the early 1980s and can claim the throne for longest use of the name in the computer space. [0] https://en.wikipedia.org/wiki/Spinnaker_Software https://en.wikipedia.org/wiki/Spinnaker_Software
  - randomsearch 11y ago
    > Should names be globally unique? I didn't say that. I just think it's unlikely they would have chosen the name if they were aware of Manchester's project, which means they didn't know about it, which means they don't read much about CS research.
jguimont 11y ago
A Spinnaker is a type sail first and foremost.
- js2 11y ago
  And also the first and foremost sail. :-)
trextrex 11y ago
There is also a SpiNNaker computer platform [1] [1] http://apt.cs.manchester.ac.uk/projects/SpiNNaker/ http://apt.cs.manchester.ac.uk/projects/SpiNNaker/

parasubvert 11y ago

This is Netflix's replacement for their Asgard platform: http://techblog.netflix.com/2015/09/moving-from-asgard-to-spinnaker.html http://techblog.netflix.com/2015/09/moving-from-asgard-to-sp... Also largely written in Groovy / Spring Boot. http://www.groovy-lang.org http://www.groovy-lang.org

mgreg 11y ago

Anyone know why this wasn't published in github.com/neflix as the rest of the NetFlix OSS tools are but rather in a new account at github.com/spinnaker? They've also gone through the trouble of doing more PR and setting up spinnaker.io for this.

andyfleming 11y ago
There was mention of partnering with Google, Microsoft, and Pivotal (in their blog post http://techblog.netflix.com/2015/11/global-continuous-delivery-with.html http://techblog.netflix.com/2015/11/global-continuous-delive... ). Maybe they wanted to have a separate github organization for easier management, etc.

andyfleming 11y ago

Also, the blog post here: http://techblog.netflix.com/2015/11/global-continuous-delivery-with.html http://techblog.netflix.com/2015/11/global-continuous-delive...

odiroot 11y ago

I really like they used Python for this sort of "system tool". Didn't know Netflix actually uses Python at all, I was expecting all of it to be Java.

fletcherr 11y ago
Er. It is almost all Java (well, mostly Groovy). Netflix uses a lot of Python but there's virtually none in this project.
- pluma 11y ago
  odiroot was likely talking about this: https://github.com/spinnaker/spinnaker/search?l=python https://github.com/spinnaker/spinnaker/search?l=python
  - odiroot 11y ago
    Exactly, Github says 84.1% Python.

merb 11y ago

I have a mixed feeling about this. I mean how do you Deliver Spinnaker. The project is fucking complex, if everything goes wrong you are pretty much done..

stephen 11y ago

We had looked at Asgard, but didn't use it because it used dedicated load balancers (e.g. separate ec2 instances) instead of ELBs (IIRC). Our deployments were already using ELBs and we wanted to stay with that. Does anyone know if this has changed with Spinnaker? I would not be surprised if Spinnaker's canary/etc. features require finer-grained control over the traffic than an ELB with just course-grained "add/remove instance" gives you, so it probably makes sense.

stuff4ben 11y ago

Curious how this compares to Jenkins Workflow plugin suite. At first glance Spinnaker seems to be simpler and has a prettier UI. Workflow has a lot more customization possible and the crappy Jenkins UI to go along with it. I might look at Spinnaker anyways just to keep from gouging my eyes out on the Jenkins "UI".

mrbonner 11y ago

Is this mainly written in Groovy? I tried to look at the github repo source but only saw a bunch of install shell scripts.

fletcherr 11y ago
It's a bunch of microservices, look at the other projects in the org.

mikecb 11y ago

I'm interested in what this says about Netflix's plans to stay on AWS, move off of it, or simply diversify their infrastructure. Maybe' we'll see a Chaos (Whatever's bigger than Kong), that models knocking out an entire provider: i.e., don't have to rely on Amazon's WAN anymore because they also have Cloud Platform/Azure to supply things.