8 ms·

Show HN: Vespene – My new Python CI/CD and automation server written in Django

180 points by mpdehaan2 8y ago

mpdehaan2 8y ago
Hey folks, this is my new project. You might possibly remember me from creating Cobbler and Ansible. I thought I'd share and if you have any questions, let me know!
- jweir 8y ago
  What was the inspiration? What problem(s) did you have with other tools that Vespene solves?
  - mpdehaan2 8y ago
    So back around 2003 I was working at Adaptec (an IBM spinoff) and we were trying to put together a build system, I think I ended up front-ending it around DamageControl, which was a Ruby port of CruiseControl, except I had to hand-code a fake "SCM" module to run timed (versus commit-based builds). Our builds took like 8 hours! At that time, I really wanted to help people with build environments but never really did it. Then, what sort of happened is I kept running into overly complex Jenkins installs. The typical miles of config checkboxes was overwhelming for me, but a larger problem was that organizations would have like 200 microservices projects, and all the build scripts would be slightly different. The idea was then to take the templating system from Cobbler - which allows lots of variables merged in at various levels, and Jinja2, and allow reuse of build scripts through variables and snippets. There was some other frustration with Jenkins - the idea that it was a ginormous codebase and still not database based, and you had to hunt for good plugins. Ansible was pretty successful for getting everybody together to contribute to common modules (it maybe took on too many though) and make sure you didn't have to go on too much of a plugin hunt. So could possibly we start a "batteries included" build system? Finally, I wanted to try out some opportunity to merge the functionality of a build system with something like Rundeck, so you didn't need two tools. I also found Rundeck complicated to set up, but if you want to run some self-service automation, most people do that within their build system. So I figured I could add in some SSH and Q&A interactivity in there to easily get that going. My other idea was to write a VM/container controller, and while I think there still needs to be a simple one as Kubernetes is getting to be a beast, that's too much work to bite off without a lot of help :) Honestly the main thing is I just wanted to work with a lot of the same open source folks again (and a bunch of new ones), so I'm really looking forward to that!
    - sciurus 8y ago
      > My other idea was to write a VM/container controller, and while I think there still needs to be a simple one as Kubernetes is getting to be a beast I'm curious what you think of https://www.nomadproject.io/ https://www.nomadproject.io/
      - mpdehaan2 8y ago
        Maybe it's just me, but I always have trouble figuring out what apps are by reading the documentation. I'd usually understand more if there was an architecture diagram, which also clues me in that I need to do one for Vespene :) One other reason I didn't build something is I don't have a current need to use something like that, so I'm probably not the one that should be developing it. There's a lot of datacenter use cases I'm ultimately not super familiar with.
- tfinch 8y ago
  How does this compare to something like Buildbot (https://buildbot.net/ https://buildbot.net/) - I share a lot of your frustrations with Jenkins and this is where we landed. (BTW thanks for ansible, it's awesome :) )
  - morenoh149 8y ago
    What's frustrating about jenkins?
    - SteveNuts 8y ago
      It almost always turns into a snowflake the second you start using it, at least in my experience.
      - mpdehaan2 8y ago
        I think we owe Jenkins a TON of credit for being a free solution that has a lot of great traction. I'm used to seeing pages of checkboxes and can't stand how obscure some of the configuration is, but mostly I wanted to write something that could handle configuration differences between hundreds of projects, so that is why there are things like Snippets in Vespene. Plus, I wanted to make something that was a little better for ops use cases (so people don't also have to go pick up something like Rundeck), so that's why there are things like the SSH integration and http://docs.vespene.io/launch_questions.html http://docs.vespene.io/launch_questions.html It's more about capabilities and future capabilities than the frustrations IMHO. Though I do have a bunch of friends who are frustrated with plugin compatibility, plugin hunting (we're doing more of a "batteries included" approach like I ran with ansible - just with much less modules), and stuff like that. I was also able to add in some stuff like container build isolation really easily, and that's all included stock.
  - mpdehaan2 8y ago
    I don't know, I'm really not trying to make one of those cliched checkbox X vs Y type grids, but when I started building Vespene I was most concerned about being able to make builds more consistent when you had hundreds of microservices developed by a lot of different teams. This is why I did http://docs.vespene.io/variables.html#snippets http://docs.vespene.io/variables.html#snippets I also saw a lot of people trying to do ops style stuff from tools like Jenkins, which is why Ansible has built-in SSH automation so it can hold on to SSH keys and use them in your behalf, and has some really cool built in RBAC so you can decide who can run what and it can be different than who can edit what. Mostly I just want to build an architecture that can go interesting places - what is here today I think is usuable, but it's just a starting point. My thinking is if you make an architecture that is really pluggable, and the code is easy to read and add to, it can go to some really neat places. I'd just recommend taking a look at the various chapters of http://docs.vespene.io/ http://docs.vespene.io/ to get a feel for features, and if you like it, spin up a copy using the setup instructions and see what you think.
- df24g3t36 8y ago
  Id strongly recommend adding the ability to implement pipelines as code -- ie: you check in the CI/CD configuration into the repo (or another repo, containing CI/CD configurations). Having to configure the pipelines manually is nasty, and often builds are caused by misconfigured of the pipeline. having this checked it allows easy versioning.
  - mpdehaan2 8y ago
    See my earlier reply about http://docs.vespene.io/importing.html http://docs.vespene.io/importing.html - though I'm pretty darn open to making this do a lot more in the .vespene file.
  - PopeDotNinja 8y ago
    I also like the idea of pipelines as code. I'd love to see it go a step further and become pipelines as testable code. I've yet to see a pipeline of non-trivial complexity that stays easy to reason about. I predict the next wave of CI/CD (or pipeline anything) will make pipeline testing & sanity checking simple as heck. If this already exists, please tell me about it, because I don't see people using it ;)
- morenoh149 8y ago
  any plans to provide a docker image?
  - mpdehaan2 8y ago
    I'm pretty sure it's going to happen, or at least having Docker build files where you can run your own (on the workers, you'd want to install what you wanted anyway). I've had a couple of conversations from folks wanting to do that and it seems like it would be pretty easy, just making the entry point supervisor vs systemd.
- deleted 8y ago
  [deleted]
- rlopezcc 8y ago
  I would strongly suggest changing the name. "Ves pene" means something pretty much NSFW in Spanish. Besides that, nice job! Keep it up.
  - koolba 8y ago
    My first thought on seeing the name is the resource from StarCraft: “We require more vespene gas”
    - hug 8y ago
      It's got to be a riff on the StarCraft resource, much like Ansible was a riff on Le Guin's faster-than-light communication device.
      - mpdehaan2 8y ago
        Indeed! I wanted another Science Fiction reference to something I loved. Vespene gas is very important in build order. My probable worst name was Cobbler (the PXE server) for those that were into cockney, but it was originally named "Shoe", and the idea was a Cobbler makes boots. Lo siento!
  - outime 8y ago
    As a native speaker I have no problems with the name. For those who don’t understand it may sound like “(you) see penis”. But to be honest I didn’t even make the connection when I read it the first time.
markovbot 8y ago
> Unlike some other CI tools, pipelines in Vespene are easily and graphically configured, and there is no custom DSL (“Domain Specific Language”) to learn and debug. Does this mean that the CI config cannot be checked into some sort of version control? That was one of my biggest issues with Jenkins. Someone would change something somewhere and suddenly my builds would fail, and even if I track down what setting is causing the problem, I have no idea why it was changed, who changed it, what will happen if I put it back, or even what it's previous value is. Additionally, it made duplicating a given workflow an extremely tedious, manual process.
- attractiveape 8y ago
  Jenkins has plugins so you can version control the job configs in your RCS. Jenkins pipeline or job-dsl2 (older)
  - markovbot 8y ago
    Yes and those are great when the people managing the Jenkins server are willing to install those plugins and get everyone to rewrite their stuff to use that. They did that eventually install that at $WORK, but they have a bunch of configuration for it for various libraries, which they make backwards-incompatible changes to without warning. Fortunately, we're mostly moving to GitLab CI which doesn't have most of this bullshit.
    - marmaduke 8y ago
      It’s worth noting that GitLab CI will turn your Yaml build config into a nice graphical pipeline (Another fan here)
      - achamayou 8y ago
        Jenkins’ Blue Ocean also displays pipelines constructed in Groovy.
      - dsumenkovic 8y ago
ianamartin 8y ago
This looks really great to me, and I can't wait to try it. I've been using buildbot for extremely light-weight CI/CD server that basically just triggers Ansible scripts on git changes. Not at all the way buildbot was designed to work, but very quick and easy to manage. This looks like a tool that could be a lot closer to what I want to do and designed to do it that way. Also, is the repo being moved somewhere? https://github.com/vespene/vespene https://github.com/vespene/vespene is 404.
- cbcoutinho 8y ago
  That's someones personal repo - you're looking for this one: https://github.com/vespene-io/vespene https://github.com/vespene-io/vespene
  - morenoh149 8y ago
    this is what's linked to from the docs so I think it's official
    - mpdehaan2 8y ago
      https://github.com/vespene-io/vespene https://github.com/vespene-io/vespene is the one!
- mpdehaan2 8y ago
  Sorry about that if I still have a typo somewhere - let me know where you found it if I do. Still hard to keep myself from typing that :)
  - ianamartin 8y ago
    Yeah, I ran into it here: http://docs.vespene.io/resources.html http://docs.vespene.io/resources.html If you would like to get involved with Vespene at a code level, “fork” the project at github.com/vespene/vespene.
deleted 8y ago
[deleted]
ensignavenger 8y ago
This looks cool. Unfortunately it is Common Clause licensed and therefore not Open Source. My organization would never consider a Commons Clause licensed tool.
- mpdehaan2 8y ago
  I'd argue the definition of what is open source is up for debate, but I had some reasons for this and thought about it for weeks. I hope people know my history and how I've run projects in the past. But what this does for me is keep me from releasing anything open core. With ansible, ansible was open source but the GUI was not, and we had to hold something back. The traditional alternatives to OSS are to make a support business, which somewhat encourages making buggy or hard to install software, or a consulting business (IMHO, same) and all of those detract from building a product, collectively, with some of your favorite people on the internet. So basically Vespene is going to be like a pseudo-foundation. The structure for this is described here - http://docs.vespene.io/partnership.html http://docs.vespene.io/partnership.html - which keeps consulting and support totally free for small shops, and encouraging larger ones that could potentially make millions form Vespene to give back a very small amount. What normally happens is large companies close six figure contracts supporting open source, and they don't give back a dime, and my goal here is to essentially make a developer's salary off of this project by having those who make more give back a very small amount. This seems pretty fair to me, but I also recognize that some people don't agree with it. That all being said, it's not a "no-commercial" clause in any way, and still free for small consultancies, so I'm not anticipating any major problems. Most of the time, software gets installed in a place of business for that business. For some of my previous thoughts on this, I'd refer folks to https://medium.com/@michaeldehaan/why-open-source-needs-new-licenses-d2d9d819a10 https://medium.com/@michaeldehaan/why-open-source-needs-new-... and later my current thinking https://medium.com/@michaeldehaan/going-with-the-commons-clause-1bdab4c15e5d https://medium.com/@michaeldehaan/going-with-the-commons-cla... It's not something I've considered lightly, but this keeps MORE software open, and for a small one-person shop, I think that's more noble than trying to hold some code back and not release it at all. Some people just want 100% free, gimme gimme, etc - and fine - you're entitled to pick one of those things. That's ok. I personally view this a little more pragmatically, under the view of fairness, and also have taken steps (in the CLA, etc), to guarantee that trust is never going to be abused. For instance, the license reverts to pure Apache if Vespene were ever to change hands. On the downside, I can't get screwed over by IBM or Amazon while I'm trying to crank out free software for everyone. That seems fair too, seeing the time investment running an open source community and working on the code takes.
cjbprime 8y ago
Does every machine running a worker have to be a full node with Django? Workers in a complex CI setup can often be e.g. raspberry pi machines, iOS emulators, Windows machines without admin access deep inside a firewall that make outgoing connections but not incoming, etc. So, you might consider a protocol for builders to report their results (e.g. over SSH) without being a full node, if that doesn't exist. Looks good!
- mpdehaan2 8y ago
  Hey! Any node typically would run the webserver because it's not that heavy, but doesn't technically have to. It could just run one worker process. Right now the workers do need database access and I suspect that will stay that way for the short term. Windows needs to happen, but I'm not sure when it happens. (Also a good list discussion). I'm a bit curious about your raspberry pi use case - maybe a good topic for the forum? http://talk.vespene.io/ http://talk.vespene.io/
peterwwillis 8y ago
> When organizations start to have too many projects, they need an easy way to share values and code snippets between those projects. To support this we have Jinja2 templating of build scripts I would implore everyone implementing CI/CD to do the opposite. If you have a small project, maybe use a build script to get it off the ground quickly. If you have a large organization with complex, disparate tech projects, do not use build scripts. Use CM or orchestration tools. Use pre-rolled containers/images. Use configuration options for existing solutions/plugins/deployment tools. Write only open source, composable, extensions/modules/plugins, with yaml/json/xml configs to control them. The open source bit is actually crucial. By making all your build/test/deploy plugins open source, you force yourself not to include proprietary information, which is always a dark pattern. Hard-coding IPs/hostnames, credentials, product names, regions, etc just makes your work less composable/reusable. Open source all your pipeline code except for your configs. Then everyone in your business can find everyone else's work simply by going to GitHub and looking up a tool. Because they won't be scripts, they can be picked up and used immediately by any team, and not have to be forked by every team that needs a modified build script. To do all this properly requires discipline and training and research and time, which big companies have. The point is not to be fast, but to be reliable, to prevent muda from sapping your productivity.
- mpdehaan2 8y ago
  Compiling C code with puppet or whatever seems a little weird to me, to be honest :) If you want to keep your data externally, there are some good options for this. I'd read the plugin docs about variable plugins, that can easily source variables for things like etcd or Consul. A script for a given repo could be nothing more than "make go", and that actually is a pretty good practice, to keep that in source control. Still, you need something to build your code, and to have a good place to see the status. Ultimately, code still needs to be built, and orgs do want to avoid images they cannot easily recreate. That's the role of a build system and making sure the process to create those artifacts is in source control. I should also mention that your build script CAN be sourced from source control. In your .vespene file, just say "script: <path>. However, the variables in that file can be evaluated by Jinja2 variables, so for instance your security team could set up a snippet everybody could use or your feature flags could be defined from there. Another cool feature of Vespene is in each project build root all those variables appear as vespene.json files, so it can be a good tool to use to launch all kinds of automation from a web console where you want to record results.
michaelmcmillan 8y ago
Cool project! It strikes me that Django might not be the ideal for this project, have you thought about that? This logic [1] looks shoehorned into the framework. [1] https://github.com/vespene-io/vespene/blob/master/vespene/models/pipeline.py#L82 https://github.com/vespene-io/vespene/blob/master/vespene/mo...
- mpdehaan2 8y ago
  While you're welcome to that opinion, Django has been freaking awesome. This whole thing got written in 3 months, not even working full time. I wrote pipelines in like a day. It seems like you're saying I should have used a many-to-many there rather than 7 FKs. Probably, but shrug... it can be fixed later if it ever becomes a problem.
  - michaelmcmillan 8y ago
    I would expect it has been easy, I’m talking about the way forward. I wish you the best of luck anyway!
cagenut 8y ago
I like how ansible was really func 2 and now vespene is the real ansible 2
- mpdehaan2 8y ago
  ha, totally not going to write another CM layer. That is one of the fatal blunders, up there with fighting a land war in Asia and all that.
p1necone 8y ago
Good thing you didn't write this on top of Etherium instead. I suspect you would quickly run into a situation where you... needed more Vespene gas.
bribri 8y ago
needs declarative syntax
- mpdehaan2 8y ago
  There's a fair amount of that available in http://docs.vespene.io/importing.html http://docs.vespene.io/importing.html ... if you have more ideas can you stop by https://talk.vespene.io/c/ideas https://talk.vespene.io/c/ideas ? ... I don't want to make the syntax more complex than the YAML that is there now, but I think we can add other fields if there is something you want or a capability that might be missing.
caleblloyd 8y ago
How do you manage normalizing webhook events from each source control provider, i.e. GitHub, GitLab, BitBucket, etc? I have been interested in building a general purpose CI/CD tool but this part seems like the most "grunt work". Is there an open source library that standardizes webhooks? Or would you be interested in publishing that portion in a library?
- dsumenkovic 8y ago
  Hey Caleb, if you are interested in using GitLab webhooks, this documentation may help you https://docs.gitlab.com/ee/user/project/integrations/webhooks.html https://docs.gitlab.com/ee/user/project/integrations/webhook....

180 points by mpdehaan2 8y ago

Hey folks, this is my new project. You might possibly remember me from creating Cobbler and Ansible. I thought I'd share and if you have any questions, let me know!

jweir 8y ago
What was the inspiration? What problem(s) did you have with other tools that Vespene solves?
- mpdehaan2 8y ago
  So back around 2003 I was working at Adaptec (an IBM spinoff) and we were trying to put together a build system, I think I ended up front-ending it around DamageControl, which was a Ruby port of CruiseControl, except I had to hand-code a fake "SCM" module to run timed (versus commit-based builds). Our builds took like 8 hours! At that time, I really wanted to help people with build environments but never really did it. Then, what sort of happened is I kept running into overly complex Jenkins installs. The typical miles of config checkboxes was overwhelming for me, but a larger problem was that organizations would have like 200 microservices projects, and all the build scripts would be slightly different. The idea was then to take the templating system from Cobbler - which allows lots of variables merged in at various levels, and Jinja2, and allow reuse of build scripts through variables and snippets. There was some other frustration with Jenkins - the idea that it was a ginormous codebase and still not database based, and you had to hunt for good plugins. Ansible was pretty successful for getting everybody together to contribute to common modules (it maybe took on too many though) and make sure you didn't have to go on too much of a plugin hunt. So could possibly we start a "batteries included" build system? Finally, I wanted to try out some opportunity to merge the functionality of a build system with something like Rundeck, so you didn't need two tools. I also found Rundeck complicated to set up, but if you want to run some self-service automation, most people do that within their build system. So I figured I could add in some SSH and Q&A interactivity in there to easily get that going. My other idea was to write a VM/container controller, and while I think there still needs to be a simple one as Kubernetes is getting to be a beast, that's too much work to bite off without a lot of help :) Honestly the main thing is I just wanted to work with a lot of the same open source folks again (and a bunch of new ones), so I'm really looking forward to that!
  - sciurus 8y ago
    > My other idea was to write a VM/container controller, and while I think there still needs to be a simple one as Kubernetes is getting to be a beast I'm curious what you think of https://www.nomadproject.io/ https://www.nomadproject.io/
    - mpdehaan2 8y ago
      Maybe it's just me, but I always have trouble figuring out what apps are by reading the documentation. I'd usually understand more if there was an architecture diagram, which also clues me in that I need to do one for Vespene :) One other reason I didn't build something is I don't have a current need to use something like that, so I'm probably not the one that should be developing it. There's a lot of datacenter use cases I'm ultimately not super familiar with.
tfinch 8y ago
How does this compare to something like Buildbot (https://buildbot.net/ https://buildbot.net/) - I share a lot of your frustrations with Jenkins and this is where we landed. (BTW thanks for ansible, it's awesome :) )
- morenoh149 8y ago
  What's frustrating about jenkins?
  - SteveNuts 8y ago
    It almost always turns into a snowflake the second you start using it, at least in my experience.
    - mpdehaan2 8y ago
      I think we owe Jenkins a TON of credit for being a free solution that has a lot of great traction. I'm used to seeing pages of checkboxes and can't stand how obscure some of the configuration is, but mostly I wanted to write something that could handle configuration differences between hundreds of projects, so that is why there are things like Snippets in Vespene. Plus, I wanted to make something that was a little better for ops use cases (so people don't also have to go pick up something like Rundeck), so that's why there are things like the SSH integration and http://docs.vespene.io/launch_questions.html http://docs.vespene.io/launch_questions.html It's more about capabilities and future capabilities than the frustrations IMHO. Though I do have a bunch of friends who are frustrated with plugin compatibility, plugin hunting (we're doing more of a "batteries included" approach like I ran with ansible - just with much less modules), and stuff like that. I was also able to add in some stuff like container build isolation really easily, and that's all included stock.
- mpdehaan2 8y ago
  I don't know, I'm really not trying to make one of those cliched checkbox X vs Y type grids, but when I started building Vespene I was most concerned about being able to make builds more consistent when you had hundreds of microservices developed by a lot of different teams. This is why I did http://docs.vespene.io/variables.html#snippets http://docs.vespene.io/variables.html#snippets I also saw a lot of people trying to do ops style stuff from tools like Jenkins, which is why Ansible has built-in SSH automation so it can hold on to SSH keys and use them in your behalf, and has some really cool built in RBAC so you can decide who can run what and it can be different than who can edit what. Mostly I just want to build an architecture that can go interesting places - what is here today I think is usuable, but it's just a starting point. My thinking is if you make an architecture that is really pluggable, and the code is easy to read and add to, it can go to some really neat places. I'd just recommend taking a look at the various chapters of http://docs.vespene.io/ http://docs.vespene.io/ to get a feel for features, and if you like it, spin up a copy using the setup instructions and see what you think.
df24g3t36 8y ago
Id strongly recommend adding the ability to implement pipelines as code -- ie: you check in the CI/CD configuration into the repo (or another repo, containing CI/CD configurations). Having to configure the pipelines manually is nasty, and often builds are caused by misconfigured of the pipeline. having this checked it allows easy versioning.
- mpdehaan2 8y ago
  See my earlier reply about http://docs.vespene.io/importing.html http://docs.vespene.io/importing.html - though I'm pretty darn open to making this do a lot more in the .vespene file.
- PopeDotNinja 8y ago
  I also like the idea of pipelines as code. I'd love to see it go a step further and become pipelines as testable code. I've yet to see a pipeline of non-trivial complexity that stays easy to reason about. I predict the next wave of CI/CD (or pipeline anything) will make pipeline testing & sanity checking simple as heck. If this already exists, please tell me about it, because I don't see people using it ;)
morenoh149 8y ago
any plans to provide a docker image?
- mpdehaan2 8y ago
  I'm pretty sure it's going to happen, or at least having Docker build files where you can run your own (on the workers, you'd want to install what you wanted anyway). I've had a couple of conversations from folks wanting to do that and it seems like it would be pretty easy, just making the entry point supervisor vs systemd.
deleted 8y ago
[deleted]
rlopezcc 8y ago
I would strongly suggest changing the name. "Ves pene" means something pretty much NSFW in Spanish. Besides that, nice job! Keep it up.
- koolba 8y ago
  My first thought on seeing the name is the resource from StarCraft: “We require more vespene gas”
  - hug 8y ago
    It's got to be a riff on the StarCraft resource, much like Ansible was a riff on Le Guin's faster-than-light communication device.
    - mpdehaan2 8y ago
      Indeed! I wanted another Science Fiction reference to something I loved. Vespene gas is very important in build order. My probable worst name was Cobbler (the PXE server) for those that were into cockney, but it was originally named "Shoe", and the idea was a Cobbler makes boots. Lo siento!
- outime 8y ago
  As a native speaker I have no problems with the name. For those who don’t understand it may sound like “(you) see penis”. But to be honest I didn’t even make the connection when I read it the first time.

markovbot 8y ago

> Unlike some other CI tools, pipelines in Vespene are easily and graphically configured, and there is no custom DSL (“Domain Specific Language”) to learn and debug. Does this mean that the CI config cannot be checked into some sort of version control? That was one of my biggest issues with Jenkins. Someone would change something somewhere and suddenly my builds would fail, and even if I track down what setting is causing the problem, I have no idea why it was changed, who changed it, what will happen if I put it back, or even what it's previous value is. Additionally, it made duplicating a given workflow an extremely tedious, manual process.

attractiveape 8y ago
Jenkins has plugins so you can version control the job configs in your RCS. Jenkins pipeline or job-dsl2 (older)
- markovbot 8y ago
  Yes and those are great when the people managing the Jenkins server are willing to install those plugins and get everyone to rewrite their stuff to use that. They did that eventually install that at $WORK, but they have a bunch of configuration for it for various libraries, which they make backwards-incompatible changes to without warning. Fortunately, we're mostly moving to GitLab CI which doesn't have most of this bullshit.
  - marmaduke 8y ago
    It’s worth noting that GitLab CI will turn your Yaml build config into a nice graphical pipeline (Another fan here)
    - achamayou 8y ago
      Jenkins’ Blue Ocean also displays pipelines constructed in Groovy.
    - dsumenkovic 8y ago

ianamartin 8y ago

This looks really great to me, and I can't wait to try it. I've been using buildbot for extremely light-weight CI/CD server that basically just triggers Ansible scripts on git changes. Not at all the way buildbot was designed to work, but very quick and easy to manage. This looks like a tool that could be a lot closer to what I want to do and designed to do it that way. Also, is the repo being moved somewhere? https://github.com/vespene/vespene https://github.com/vespene/vespene is 404.

cbcoutinho 8y ago
That's someones personal repo - you're looking for this one: https://github.com/vespene-io/vespene https://github.com/vespene-io/vespene
- morenoh149 8y ago
  this is what's linked to from the docs so I think it's official
  - mpdehaan2 8y ago
    https://github.com/vespene-io/vespene https://github.com/vespene-io/vespene is the one!
mpdehaan2 8y ago
Sorry about that if I still have a typo somewhere - let me know where you found it if I do. Still hard to keep myself from typing that :)
- ianamartin 8y ago
  Yeah, I ran into it here: http://docs.vespene.io/resources.html http://docs.vespene.io/resources.html If you would like to get involved with Vespene at a code level, “fork” the project at github.com/vespene/vespene.

deleted 8y ago

[deleted]

ensignavenger 8y ago

This looks cool. Unfortunately it is Common Clause licensed and therefore not Open Source. My organization would never consider a Commons Clause licensed tool.

mpdehaan2 8y ago
I'd argue the definition of what is open source is up for debate, but I had some reasons for this and thought about it for weeks. I hope people know my history and how I've run projects in the past. But what this does for me is keep me from releasing anything open core. With ansible, ansible was open source but the GUI was not, and we had to hold something back. The traditional alternatives to OSS are to make a support business, which somewhat encourages making buggy or hard to install software, or a consulting business (IMHO, same) and all of those detract from building a product, collectively, with some of your favorite people on the internet. So basically Vespene is going to be like a pseudo-foundation. The structure for this is described here - http://docs.vespene.io/partnership.html http://docs.vespene.io/partnership.html - which keeps consulting and support totally free for small shops, and encouraging larger ones that could potentially make millions form Vespene to give back a very small amount. What normally happens is large companies close six figure contracts supporting open source, and they don't give back a dime, and my goal here is to essentially make a developer's salary off of this project by having those who make more give back a very small amount. This seems pretty fair to me, but I also recognize that some people don't agree with it. That all being said, it's not a "no-commercial" clause in any way, and still free for small consultancies, so I'm not anticipating any major problems. Most of the time, software gets installed in a place of business for that business. For some of my previous thoughts on this, I'd refer folks to https://medium.com/@michaeldehaan/why-open-source-needs-new-licenses-d2d9d819a10 https://medium.com/@michaeldehaan/why-open-source-needs-new-... and later my current thinking https://medium.com/@michaeldehaan/going-with-the-commons-clause-1bdab4c15e5d https://medium.com/@michaeldehaan/going-with-the-commons-cla... It's not something I've considered lightly, but this keeps MORE software open, and for a small one-person shop, I think that's more noble than trying to hold some code back and not release it at all. Some people just want 100% free, gimme gimme, etc - and fine - you're entitled to pick one of those things. That's ok. I personally view this a little more pragmatically, under the view of fairness, and also have taken steps (in the CLA, etc), to guarantee that trust is never going to be abused. For instance, the license reverts to pure Apache if Vespene were ever to change hands. On the downside, I can't get screwed over by IBM or Amazon while I'm trying to crank out free software for everyone. That seems fair too, seeing the time investment running an open source community and working on the code takes.

cjbprime 8y ago

Does every machine running a worker have to be a full node with Django? Workers in a complex CI setup can often be e.g. raspberry pi machines, iOS emulators, Windows machines without admin access deep inside a firewall that make outgoing connections but not incoming, etc. So, you might consider a protocol for builders to report their results (e.g. over SSH) without being a full node, if that doesn't exist. Looks good!

mpdehaan2 8y ago
Hey! Any node typically would run the webserver because it's not that heavy, but doesn't technically have to. It could just run one worker process. Right now the workers do need database access and I suspect that will stay that way for the short term. Windows needs to happen, but I'm not sure when it happens. (Also a good list discussion). I'm a bit curious about your raspberry pi use case - maybe a good topic for the forum? http://talk.vespene.io/ http://talk.vespene.io/

peterwwillis 8y ago

> When organizations start to have too many projects, they need an easy way to share values and code snippets between those projects. To support this we have Jinja2 templating of build scripts I would implore everyone implementing CI/CD to do the opposite. If you have a small project, maybe use a build script to get it off the ground quickly. If you have a large organization with complex, disparate tech projects, do not use build scripts. Use CM or orchestration tools. Use pre-rolled containers/images. Use configuration options for existing solutions/plugins/deployment tools. Write only open source, composable, extensions/modules/plugins, with yaml/json/xml configs to control them. The open source bit is actually crucial. By making all your build/test/deploy plugins open source, you force yourself not to include proprietary information, which is always a dark pattern. Hard-coding IPs/hostnames, credentials, product names, regions, etc just makes your work less composable/reusable. Open source all your pipeline code except for your configs. Then everyone in your business can find everyone else's work simply by going to GitHub and looking up a tool. Because they won't be scripts, they can be picked up and used immediately by any team, and not have to be forked by every team that needs a modified build script. To do all this properly requires discipline and training and research and time, which big companies have. The point is not to be fast, but to be reliable, to prevent muda from sapping your productivity.

mpdehaan2 8y ago
Compiling C code with puppet or whatever seems a little weird to me, to be honest :) If you want to keep your data externally, there are some good options for this. I'd read the plugin docs about variable plugins, that can easily source variables for things like etcd or Consul. A script for a given repo could be nothing more than "make go", and that actually is a pretty good practice, to keep that in source control. Still, you need something to build your code, and to have a good place to see the status. Ultimately, code still needs to be built, and orgs do want to avoid images they cannot easily recreate. That's the role of a build system and making sure the process to create those artifacts is in source control. I should also mention that your build script CAN be sourced from source control. In your .vespene file, just say "script: <path>. However, the variables in that file can be evaluated by Jinja2 variables, so for instance your security team could set up a snippet everybody could use or your feature flags could be defined from there. Another cool feature of Vespene is in each project build root all those variables appear as vespene.json files, so it can be a good tool to use to launch all kinds of automation from a web console where you want to record results.

michaelmcmillan 8y ago

Cool project! It strikes me that Django might not be the ideal for this project, have you thought about that? This logic [1] looks shoehorned into the framework. [1] https://github.com/vespene-io/vespene/blob/master/vespene/models/pipeline.py#L82 https://github.com/vespene-io/vespene/blob/master/vespene/mo...

mpdehaan2 8y ago
While you're welcome to that opinion, Django has been freaking awesome. This whole thing got written in 3 months, not even working full time. I wrote pipelines in like a day. It seems like you're saying I should have used a many-to-many there rather than 7 FKs. Probably, but shrug... it can be fixed later if it ever becomes a problem.
- michaelmcmillan 8y ago
  I would expect it has been easy, I’m talking about the way forward. I wish you the best of luck anyway!

cagenut 8y ago

I like how ansible was really func 2 and now vespene is the real ansible 2

mpdehaan2 8y ago
ha, totally not going to write another CM layer. That is one of the fatal blunders, up there with fighting a land war in Asia and all that.

p1necone 8y ago

Good thing you didn't write this on top of Etherium instead. I suspect you would quickly run into a situation where you... needed more Vespene gas.

bribri 8y ago

needs declarative syntax

mpdehaan2 8y ago
There's a fair amount of that available in http://docs.vespene.io/importing.html http://docs.vespene.io/importing.html ... if you have more ideas can you stop by https://talk.vespene.io/c/ideas https://talk.vespene.io/c/ideas ? ... I don't want to make the syntax more complex than the YAML that is there now, but I think we can add other fields if there is something you want or a capability that might be missing.

caleblloyd 8y ago

How do you manage normalizing webhook events from each source control provider, i.e. GitHub, GitLab, BitBucket, etc? I have been interested in building a general purpose CI/CD tool but this part seems like the most "grunt work". Is there an open source library that standardizes webhooks? Or would you be interested in publishing that portion in a library?

dsumenkovic 8y ago
Hey Caleb, if you are interested in using GitLab webhooks, this documentation may help you https://docs.gitlab.com/ee/user/project/integrations/webhooks.html https://docs.gitlab.com/ee/user/project/integrations/webhook....