15 ms·

Concourse CI

• bdcravens 9y ago
  Kind of odd to see the homepage show Vagrant as the install mechanism, even though it supports Docker as well. In 2017, I'd think more developers are likely to run Docker than Vagrant workloads on their machine.
  • adam_seeliger 9y ago
    Have you noticed that all your generation does is whine, and complain about everything? It is very annoying.
  • fenollp 9y ago
    Probably easier to run Docker inside Vagrant than Docker inside Docker.
    • Samantha34 9y ago
      I'm making $86 an hour working from home. I was shocked when my neighbour told me she was averaging $95 but I see how it works now. I feel so much freedom now that I'm my own boss. This is what I do, ====http://cutt.us/7N2sG http://cutt.us/7N2sG
      • djohnston 9y ago
        how do you report comments like this?
        • iou 9y ago
          Click on the `n minutes ago` link and there'll be a [flag] option. I flagged it for you.
        • adam_seeliger 9y ago
          That comment hurt your feelers?
    • EngineerBetter 9y ago
      Concourse doesn't run Docker, it uses something called Garden to run OCI containers from Docker images. Those containers can run the Docker daemon though, if you like.
      • krakensden 9y ago
        The version of Garden it uses basically just shells out to Docker. Run 'top' on a busy Concourse worker, it's fun.
        • wlamartin 9y ago
          Garden makes use of runc (an Open Container Initiative project with a lot of contributions from Docker), in the same way containerd (component included in Docker) makes use of runc to run images. You won't find the docker engine being used by Garden. Edit: I _think_ what you're likely seeing (if you are seeing docker in the top output) is Concourse using the docker-image-resource to pull images for your tasks to a local docker registry.
  • scaryclam 9y ago
    I still see the majority of developers that I've met using Vagrant. I gave my own team the option of both and as yet, I'm the only one using docker, so I think you'd be surprised. Docker seems to have gone out of favour in a lot of places.
  • arwineap 9y ago
    Concourse makes use of a feature of the kernel that is not compiled in on all distros. Archlinux for example does not compile in user namespaces. Vagrant is the most cross-platform option for quick local concourse testing.
• paloaltokid 9y ago
  Concourse is a really amazing piece of software, but the BOSH requirement I think will keep adoption low for small companies.
  • acidus 9y ago
    I use Concourse every day and I don't know yet what BOSH is and I hope I'll never have to. The installation was pretty standard. I followed a tutorial written by Justin Ellingwood, a great technical writer working for DigitalOcean - https://www.digitalocean.com/community/tutorials/how-to-install-concourse-ci-on-ubuntu-16-04 https://www.digitalocean.com/community/tutorials/how-to-inst....
  • EngineerBetter 9y ago
    We made Concourse-Up so you don't need to learn BOSH. https://github.com/EngineerBetter/concourse-up https://github.com/EngineerBetter/concourse-up It'll give you a working, upgradable, auto-healing Concourse in one command.
  • krakensden 9y ago
    The standalone binaries work great, you don't need BOSH or cloudfoundry.
  • SHyx0rmZ 9y ago
    Shameless plug: I've been maintaining three Concourse clusters and have created https://github.com/SHyx0rmZ/concourse-debian https://github.com/SHyx0rmZ/concourse-debian to package the binary for Debian, so I can get away with just upgrading to a new package whenever a new version releases. It's far from perfect yet, but I'm happy to receive some feedback and planning to move my package repository and pipeline to the public soonish.
  • arwineap 9y ago
    You should try the binaries! They are pre-compiled go binaries, and require nothing other than a modern kernel! If you want an example you can see an example in concourse's upstream docker config
• boyter 9y ago
  My number one complaint with Concourse (which I suspect is due to Go) is that you need to have it hosted with a valid TLS/SSL cert in order to use the fly command. At least this was an issue in the 2.6.0 days, but I couldn't see anything to change this in the recent versions. This is rather annoying if you want to run a copy on your local network say at home. Its very frustrating because the fly command solves the biggest issue with IWOMM (it works on my machine) by allowing you to run code and tests on another machine before committing anything. I think from memory I tried using self signed certs and this also had issues for one reason or another. That said it is still the best CI system I have used to date.
  • captn3m0 9y ago
    I issue valid TLS certs for my internal servers using letsencrypt DNS challenge (there is a nice cloudflare hook for dehydrated that I use). Runs on cron, haven't had to worry about it once I set it up. (Haven't tried with concourse, but don't think that would be a problem)
    • deleted 9y ago
      [deleted]
    • voltagex_ 9y ago
      Do you worry about your internal names being exposed via Certificate Transparency?
      • captn3m0 9y ago
        Not much.
  • rukenshia 9y ago
    you can use the login command with -k (--insecure) to skip the SSL verification.
  • deleted 9y ago
    [deleted]
  • 9y ago
• webo 9y ago
  Mostly off-topic but I've been looking for more than a CI for some quite time, more on the CD side. How are you guys handling some of these cases? Bonus points for hosted options. - Truly a pipeline based stages. No messing around with git branches/tags for each environment release. - Ability to combine multiple builds together. - Build dependencies. Ability to trigger project-B build when project-A build succeeds. (Docker Cloud Builds have something like this.) - Use the same artifact across multiple build stages. - An option to promote a build to the next step either manually or automatically. Amazon's internal Apollo system was amazing. AWS Code Pipelines kind of does some of these things but it's every limited and hard to work with.
  • tomprince 9y ago
    Mozilla's TaskCluster is definitely aimed at CI use cases, but does handle all the requirements you listed. It is entirely open-source, but unfortunately they aren't currently interested in supporting other organizations running it.
    • Stella434 9y ago
      lm måκlng оvêя $7κ å mоnтн шояκlng påят тlmê. l κêpт нêåяlng отнêя pêоplê тêll mê нош mucн mоnêy тнêy cån måκê оnllnê Sо l DêclDêD то lооκ lnто lт. шêll, lт шåS åll тяuê ånD нåS тотålly cнångêD my llfê. This is what I do, ====https://sites.google.com/site/sandralopz78/ https://sites.google.com/site/sandralopz78/
  • sytse 9y ago
    GitLab CI has multi project pipeline, triggers, artifacts, and manually approved next steps. But I'm not sure what your first requirement means, GitLab CI always composes pipelines from individual projects so it probably doesn't meet your criteria. I'm interested to hear what you ran into with previous systems that made this your nr 1 criteria.
  • manojlds 9y ago
    Have you looked at Go.CD - https://www.gocd.org/ https://www.gocd.org/
• witten 9y ago
  I took part in evaluating Concourse CI for the needs of my company (30+ devs). While it has amazing CI pipeline capabilities, we ultimately didn't select Concourse because it felt much more like a CI toolbox, requiring some development to put those tools to use. And what we really wanted was more of a turnkey CI product. Perhaps ironically, we ended up doing some development around the edges of the CI product we ultimately selected (GitLab).
  • sytse 9y ago
    Cool to hear you selected GitLab. We recently added multi project pipeline visualization. That was partially inspired by concourse CI. Is there anything else you would like to see?
    • witten 9y ago
      Interesting you should mention that. Right around the time you implemented multi-project pipeline visualization, we built a bot that listens for GitLab build completion events, hits the (undocumented /unsupported) internal GitLab global code search ElasticSearch index, finds downstream dependencies of the completed build (as declared in setup.py/package.json), and triggers their builds as well.. Taking full advantage of your nifty visualization. So I think the feature we'd want to make that easier is an actual global code search API!
      • sytse 9y ago
        Cool! I think we're building this into GitLab itself. I'll ask the CI CD team to comment.
      • grzesiekb 9y ago
        Thanks for this proposal! It might be a cool feature indeed. I created an issue about it in our issue tracker, see https://gitlab.com/gitlab-org/gitlab-ce/issues/35186 https://gitlab.com/gitlab-org/gitlab-ce/issues/35186. We do not have immediate plans to ship that yet, but I pinged our product team member there. However we do have plans to ship multi-project pipeline with an "inversion of control". You will be able to specify pipeline relation with an upstream project, and when someone pushes a commit to it, a downstream pipeline is going to be trigger automatically. See an issue about cross-project dependencies - https://gitlab.com/gitlab-org/gitlab-ee/issues/1681 https://gitlab.com/gitlab-org/gitlab-ee/issues/1681!
• rjzzleep 9y ago
  I've been eyeing Concourse and Go.CD over Jenkins for a while. The main criticism I saw on Jenkins and Go.CD vs. Concourse was that Jenkins Pipelines aren't first class and that it's easier to export configuration(in that regard Concourse > Go.CD > Jenkins). On the other hand Jenkins and Go.CD supports extensions, which Concourse touts as a feature. I also want the CI builds to create my base boxes with packer in multiple steps. And I somewhat want to be able to hand over the stuff to ops at some point to be able just stay alive for the next 5 years or more. Would anyone know if it makes sense to even consider concourse or go.cd or some other CI/CD solution and if so which? Obviously the boxes need to be used as artifacts and everything has be on premise as well.
  • jackfraser 9y ago
    I've worked with GoCD in production a bit. It's a bit of a beast to keep running, the UI is very strange, but the pipelines are extremely powerful and make for good separation of stages. I found it awkward to configure jobs, though, and the documentation is not excellent. There's also a much smaller plugin ecosystem than with Jenkins. We ended up having a guy spend a good lot of his time, nearing 50% some weeks, just keeping GoCD happy for a couple hundred devs. Overall, I would choose Jenkins first unless you know there's something GoCD can handle significantly better.
  • i386 9y ago
    How are Pipelines not first class in Jenkins? Pipelines are provided in the default installation and they are the first thing we talk about in the docs https://jenkins.io/doc/pipeline/tour/hello-world/ https://jenkins.io/doc/pipeline/tour/hello-world/ Disclosure: I work on Jenkins.
    • arwineap 9y ago
      I think that the features are very different. Admittedly I ditched jenkins before pipelines became a full feature, but my understanding of them in jenkins was that they are mostly for scheduling of jobs. IE, run job A after job B Concourse passes inputs, outputs and resources between jobs as the ONLY STATE, and jobs trigger based off of changes on resources or the availability of new inputs. I think that when you sit down and look at the two products, jenkins is great at running scripts, and concourse is great at managing code versions. In the end we actually run both concouse + a script runner; this allows concourse to manage the tagging, builds, releases, and testing; but still allows us to run ad hoc scripts that concourse doesn't do well. I'd be interested in taking another look at jenkins now that pipelines and the groovy DSL is solidified, but I get the idea that they still fill slightly different needs
• nstart 9y ago
  Not sure when it came about, but when I first evaluated Concourse, not being able to trigger jobs manually was a primary blocker. Glad this showed up on HN again, because they've added it in at some point. My favourite thing about Concourse was really its ideas around "Resources". This always felt so much better than this idea of plugins due to how unified the experience was. Also, the ability to implement resources for yourself is extremely easy. So if I had an internal software running, being able to build something for it meant defining three bash scripts. That said, the docs around implementing resources still needs some improvements. Whatever I learnt was from cloning and modifying existing resources.
  • Alexis54 9y ago
    Af­­­te­r I q­­ui­­tt­ed m­­­­y 8­-t­­o-5­ jo­­­b, I g­­e­t pa­­i­d $9­5/­­h­r...H­­­­o­w I d­i­d i­­­­­­t? I s­­­­­tar­­­te­­d d­­­oi­­­­­ng a­­­­n in­­­­­­­te­r­n­et-b­­­­­as­­­­ed j­­­ob­­­! M­y p­­­­­­­­a­s­t wo­­­­­­rk di­­­­­­­dn't ma­­­­­­ke m­­­­­­e ha­­­­­­pp­­­­­­y s­­­­­­­­­o I w­­­­­an­t­e­­­­­­d t­­­­­o t­­­­­­­­ak­e a b­­r­e­ak a­­­­nd t­­r­y som­­et­­hing ne­­­­w… A­­­­f­t­e­r 6 y­­­r­s I­­­t w­­­as a b­­­­ig st­­e­p f­o­­­­r m­­­e t­­­­o qu­­i­t m­­y d­­­ay jo­­b bu­­­t n­­­­­­ow I a­­­m mo­­­­­­re th­­­an hap­­­­p­y ==== >>http://bit.ly/2coUNgf http://bit.ly/2coUNgf
  • fatterypt 9y ago
    This is also close to my experience. Improving, but not there yet. Unfortunately, it will retrigger the same job with the updated version of the resources. With other CI tools, it was possible to replay the same job, with whatever inputs they took by the time they ran, with concourse it does not seem t be possible. A practical impact of this issue is that we cannot simply reprocess a previous deploy step, with the old artifacts and inputs in a middle of an outage.
    • wlamartin 9y ago
      You can achieve this by clicking on a resource and hitting toggling the "power button" to off for each of the resources you don't want to include in the build.
• rukenshia 9y ago
  We are using concourse at work right now and have 50-ish pipelines for various repositories. IMO, there's definitely some work you have to put into it because you'll sooner or later run into a problem with the existing resources and need a custom one. Writing custom resources is pretty easy however. Concourse also isn't really made to work well with the Git Flow, there is no builtin way to run the CI on multiple branches (there's a git-multibranch community resource which requires redis at some point). we're basically thinking about changing our workflow to trunk-based, but it still feels weird to me that we might change our workflow to fit our CI better. that being said, I personally still really like concourse and it's fun to work with.
  • Amelia55 9y ago
    I'm making $86 an hour working from home. I was shocked when my neighbour told me she was averaging $95 but I see how it works now. I feel so much freedom now that I'm my own boss. This is what I do, ====http://ow.ly/P0wk30aVXfN http://ow.ly/P0wk30aVXfN
• org3432 9y ago
  We looked at Concourse deeply, while we didn't go with it, it's inspired a number of projects I know and I think has pointed out the obvious; representing how you think about a problem is how you should represent it visually, great insight that has been overlooked in CI.
• kieranajp 9y ago
  We're using Concourse extensively at HelloFresh (>130 devs). It's not without its quirks, but I've little to complain about so far, except perhaps the polish of the UI.
  • EngineerBetter 9y ago
    Cool - did you folks know about the Concourse London User Group? The last two have been held just around the corner from you: https://www.meetup.com/Concourse-London-User-Group/ https://www.meetup.com/Concourse-London-User-Group/
    • kieranajp 9y ago
      All our developers are Berlin based, so the London office isn't a great metric. Thanks anyway, though!
• skrebbel 9y ago
  I hadn't heard of this, it looks nice! Would I be far off if I'd say that Concourse is an open source clone of Travis CI and CircleCI? Any strong pluses of the SaaS offerings over Concourse I should know about?
  • sytse 9y ago
    Concourse is developed independently, it is focused more on multi project workflows.
  • rexpop 9y ago
    Distributed systems are first-class citizens of Concourse. It's a larger concept-space than Travis/Jenkins'.
• deleted 9y ago
  [deleted]
• joeseeder 9y ago
  Have been working with it for a few months, and I am not positive about it. UI is pretty but often breaks. Is unusable when you have a pipeline with dozens of concurrent jobs. Concept of teams is fine, but when you have to switch between them, even with oauth authentication, it is a pain. Job concurrency control is binary. We never have been able to have worker pool scale down without a hiccup, always some darn worker hogging containers and atc not removing it, leading to stalled pipelines. That overlay network it comes with, garden thingy, creates so many problems and solves just one... Oh and not having BUILD_ vars available in tasks is rude, thank you very much, but there are cases when it is just mandatory and concourse makes it impossible to do. At least new version has better secrets handling, previously it was a joke.
  • henryaj 9y ago
    > UI is pretty but often breaks. Is unusable when you have a pipeline with dozens of concurrent jobs. Find this hard to believe - we have some pretty spectacular pipelines which render without a hiccup.
    • krakensden 9y ago
      100+ jobs will break your browser. RabbitMQ has been complaining about it for a while.
      • jadeklerk 9y ago
        100+ jobs in one pipeline?
        • joeseeder 9y ago
          Yes, some systems we work with have over 100 items ( jobs and resources) in one pipeline ... because microservices...
    • joeseeder 9y ago
      My experience after having the browser UI open is that intially stuff renders, then starts to take ages to update state ( these pretty pulsing frames are no longer showing or rendering badly ). Quite often switching to a different pipeline renders empty space. Also, when you have around 100+ resources and jobs in one pipeline, in a vertical plane, they are impossible to identify
• manigandham 9y ago
  We're using Buddy - https://buddy.works/ https://buddy.works/ - and it seems to be easier, faster and more reliable than most of these other CI solutions so far.
  • PetrolMan 9y ago
    Interesting. How does it compare to other CI software like Jenkins? I'm always a bit skeptic about tools that look nice but turn out to be limited when put to real work.
    • manigandham 9y ago
      Works well for us, docker/container native so every step in a pipeline is a persistent volume with the environment changing based on what image you need. Also easily builds/pushes your own images and lots of other built-in task runners to make pipelines. Has .yaml or ui config and they have an enterprise version which runs on-prem.
      • PetrolMan 9y ago
        Thanks. Looks like worth giving a try indeed.
• Jedd 9y ago
  Typo on front page. Rather than "Rather than a myriad of checkboxes..." it should be "Rather than myriad checkboxes..."
  • trumbitta2 9y ago
    I thought "Rather than a myriad checkboxes" (no of)
    • Jedd 9y ago
      Certainly no 'of'. Article preceding is possibly optional (but it still sounds wrong to my ear).
      • mst 9y ago
        Speaking from an en_UK POV 'a myriad of' is how I'd normally use it; the poetic adjectival form is nice in poetry, but doesn't feel quite right to me in normal language. Whether I'm correct to feel like this is left as an exercise to the reader.
        • Jedd 9y ago
          When I lived in London I noticed lots of people got it wrong. If you think of it as a synonym for 'many' rather than 'lots' it's easier to slot into sentences correctly.
  • kapep 9y ago
    http://www.thefreedictionary.com/myriad http://www.thefreedictionary.com/myriad > Usage Note: Throughout most of its history in English myriad was used as a noun, as in a myriad of reasons. In the 1800s, it began to be used in poetry as an adjective, as in myriad dreams. Both usages in English are acceptable, as in Samuel Taylor Coleridge's "Myriad myriads of lives." This poetic, adjectival use became so well entrenched generally that many people came to consider it as the only correct use. In fact, however, both uses are acceptable today.
• mugsie 9y ago
  I know everyone hates on Jenkins these days, but (at least for the workloads I am building) Concourse feels like a toy. My issues: - everything has to be in a docker container (not all things can run in garden) - Just trying to find logs is a pain, and getting the UI to show a full log output is basically impossible - each install of concourse may need an entirely different version of fly - There was no way of retriggering a CI run on a single PR (without force pushing to the branch, which removes Github reviews) Jenkins + Jenkinsfiles + Pipelines provides all of the good bits of Concourse, with none of the Pivitol enforced workflow + tools. I get why CloudFoundry uses Concourse - the build process is so arcane that you basically need to run the full CI locally, but I really do not get the hype for other projects.
  • wlamartin 9y ago
    > not all things can run in garden Can you provide an example of what you'd want to do but you aren't able to?
    • mugsie 9y ago
      Booting VMs is kind of the main one.
      • wlamartin 9y ago
        Booting VMs on your worker seems like exactly the kind of debris concourse is encouraging the avoidance of by enforcing the containerisation of your builds. That said, I can't think of a reason why you couldn't boot a VM in a privileged Garden container, have you tried?
        • mugsie 9y ago
          Yeah - I can't remember the actual failure, but it did not work out well. We also need the VMs to survive across build steps. We are working on an P/IaaS - so integration testing requires actually booting a full stack, then running an install, then testing the resultant servers are actually running Kubernetes
• rememberlenny 9y ago
  We use concourse.ci to successfully run cloud.gov at 18F.
• joneholland 9y ago
  We've been happy teamcity users for years.
• Siecje 9y ago
  During my Jenkins build I am creating a file with the number of database queries per test, since I am generating the file it can be in any format. How can I feed this to Jenkins and plot it over time?
• arsenico 9y ago
  Curious to know why no one actually mentions Bamboo - definitely a great alternative to Jenkins, especially for those, who are into Atlassian infrastructure.
  • witten 9y ago
    At least last I looked, Bamboo requires manual configuration of all builds. This doesn't scale to more than a handful of build plans. It also doesn't have much in the way of modern CI pipeline functionality.
• abraham_s 9y ago
  I evaluated concourse CI and I was impressed by the concept. Pros - Setup was really easy.(I didn't use bosh). - Both server and build pipeline configuration (Yaml file) are easy to backup and recreate. Cons - When I evaluated it , it had a few bugs which forced me to restart the server almost daily to keep it working. It should be more stable now. -
  • jadeklerk 9y ago
    Did your evaluation happen a long time ago? Or, another question... did you run the binaries and appropriately screen them? I had the same problem until I realized I sucked at making a binary live a long time, and my eyes were opened to screen :)
    • mugsie 9y ago
      Never run long running services in screen. Use a proper init system - screen is not the solution.
    • abraham_s 9y ago
      It was a known bug in the some go library. It was recently fixed. But at the time it caused concourse to stop working until restarted.
• zimbatm 9y ago
  Avoid! Concourse looks good on the surface but it's really not that great. The UI is clunky. The abstraction layer is too low and leads to a lot of repeated YAML. Which leads to YAML programming. There are simple scenarios like deployment rollbacks who are hard to do. For some reason they decided to develop their own container engine which leads to all sorts of trouble and maintenance issue. It's generally slow and we had 100% CPU usage when the worker was doing almost nothing. I have used for 4 months and it was only problems. Gitlab CI is much better. Or even Jenkins is better.
  • jadeklerk 9y ago
    > Concourse looks good on the surface but it's really not that great. I've used teamcity, jenkins, gocd, circleci, concourse, and travisci. For multi-project systems, concourse is king. (I like travisci for by-itself, non-system projects) > The UI is clunky. What? You just said the UI looks good... It's simple and clean; everything is async javascript (no page loads). > The abstraction layer is too low and leads to a lot of repeated YAML. Which leads to YAML programming. Which is an intentional choice. If you don't like YAML, use one of the MANY yaml abstraction layers of your choice... > There are simple scenarios like deployment rollbacks who are hard to do. First of all, a CI system shouldn't be your answer to rollbacks. Your deployment system should handle that. Secondly, assuming your deployment system can do rollbacks, concourse has on-fail jobs that can trigger rollbacks just fine. > For some reason they decided to develop their own container engine which leads to all sorts of trouble and maintenance issue. It's generally slow and we had 100% CPU usage when the worker was doing almost nothing. garden is used because cloudfoundry builds it. It is not slow... it is a light layer on top of runc (as opposed to docker which is a rather heavy layer on top of runc). You should pretty much never have to care about it, and in 3 years of using concourse I haven't had to - and we have some pretty gnarly large pipelines. Also I call rubbish on your 100% CPU. I have two workers t2.xlarge workers running 22 and 30 containers (like, right NOW) and neither is above 10% CPU (which, actually, I should make those a lot smaller). Don't run workers on a potato and you'll be fine. > Gitlab CI is much better. Or even Jenkins is better. Ohhh you're trolling.
    • jacques_chester 9y ago
• aarondl0 9y ago
  Excellently designed system. Very poorly implemented. Our team's been using it since it's initial releases. It's been nothing short of disastrous for all but the smallest pipelines. The design is great. Keeping configs in yaml instead of little white boxes in a Jenkins database is much better. Pipelines as a first class concept. It feels inspired by a functional programming language. You get great build reproducibility since there are no workers that get dirtier over time if you forget to clean up. The resource model is awesome. Very cool stuff. I'm hoping every CI system learns from what's here. Second to none in design. However it performs like a dud. No scheduling to speak of, just runs everything as soon as it can. We've run into nodes dying under load (-not- underprovisioned, could run all these jobs manually at once on these monsters). We've run into problems with volume reaping, fork bombs, ui freezes, everything under the sun. I really like Concourse and will hopefully one day be able to come back to it when its implementation is as solid as its paradigms are. But I'd avoid use for now.
  • jadeklerk 9y ago
    > However it performs like a dud. No scheduling to speak of, just runs everything as soon as it can. We've run into nodes dying under load (-not- underprovisioned, could run all these jobs manually at once on these monsters). We've run into problems with volume reaping, fork bombs, ui freezes, everything under the sun. I've used concourse as a consumer for 3 years and I've very, very rarely seen any of the problems you're describing, even on the older versions and certainly not in the last year or so. > no scheduling to speak of Concourse has a massive scheduling system built into it.. https://github.com/concourse/atc https://github.com/concourse/atc Furthermore, you can configure jobs to run in serial (default is parallel). > ui freezes Put your `web` binary on a decently-sized VM and your problem should disappear. Also, don't have your workers on the same VM as your `web`.
    • aarondl0 9y ago
      UI freezes are completely client side and related to the elm implementation and your browser's execution of the code. The size of your VM doesn't matter at all. ATC is a more of a dependency scheduler. The code [1] shows that it basically gets all pending jobs, and then runs them. There's no concept of queueing or maximum number of jobs, you just have to hope your limits are high enough (max containers, max tasks in systemd, max fds in the same) and that your machine doesn't fall over in the attempts. The "massive" scheduling system also has no idea what nodes need work and which do not [2] so the idea is to heavily overprovision until it doesn't fall over (on top of already beefy requirements which others have alluded to in this thread). You can not serialize multiple pipelines. Only within pipelines. If I have 10 pipelines, they will all run independently and there's nothing you can do about it other than attempt serialization with the pool resource (which we've recently had problems with - it also appears to be buggy and we're looking at submitting patches). I've got a tremendous amount of experience with this system and I believe it's everything I made it out to be. The rebuttals you've provided to my issues are simply a lack of understanding of our context, not every user will have the same experience with any given product. [1] https://github.com/concourse/atc/blob/master/scheduler/scheduler.go#L46 https://github.com/concourse/atc/blob/master/scheduler/sched... [2] https://github.com/concourse/concourse/issues/675 https://github.com/concourse/concourse/issues/675
• jacques_chester 9y ago
  I'm late to the party. Concourse is difficult to come to from other CI tools. A little more aloof. There have been real, serious implementation difficulties. But I kinda love it, because it's a handful of simple ideas that unlock incredible power. It goes beyond "build and test each commit" to becoming a full project automation tool, a software manufacturing robot. When I talk to people about Concourse, I tell them: your pipeline and your tasks are production code. Keep the discipline, care and engineering practices that you bring to the apps and services you create. The problem is that most of the deep tribal knowledge about how to get started and how to best apply it is locked up inside a handful of organisations, most notably Pivotal. I had been working on a video series which was meant to walk through both the concrete business of building pipelines, as well as the concepts of how best to do so. Unfortunately visa conditions got in the way and I have abandoned that effort. Interested persons are welcome to email me for links to the first 4 episodes that I made, but be aware that it stops even more suddenly than Firefly. Disclosure: I work for Pivotal.
  • boundlessdreamz 9y ago
    Why not put it up on youtube?
    • jacques_chester 9y ago
      It's an hour of work per minute of screen time. For now it ends at 4 episodes.
• Ataraxic 9y ago
  A little late to this discussion, but although I'm a big fan of many of the concepts in Concourse, I think it's lacking a lot of polish. For example, I have actually implemented patterns within Jenkins to force all jobs to run inside of containers. A job is just a container + command with some linking using the jenkins pipeline plugin that reads some json configuration to determine how jobs are linked. The primary issues I have surround the fact that my company uses kubernetes, thus we have no insight into the runc containers. Load balancing in concourse is non-existent. If a worker goes down due to load, if you bring it back up, it's going to go down immediately from all the jobs that have been triggered while the worker was offline. Not only that, the resource requirements seem pretty high. Recently a concourse worker stalled because the amount of volumes/images it was caching was over 100 gigs, and not knowing the internals, I wasn't sure what the best way was to clear this cache. Having to tell the infrastructure team that we just need to spend more money is a hard sell when we've upped the cpu, memory, storage, postgres disk, all more than once. I understand that different images have vastly different sizes, and jobs different amounts of work, but their need to be some clear suggestions for sizing. If there exist them now, I apologize but I haven't seen them. So yeah I've had some fun developing in it, but more help making it reliable would be really nice. Also if kubernetes is absolutely the wrong way to run it, which it seems like, I'd have to be provided a better/easier alternative to really become an advocate. Final note: has anyone actually setup metrics/monitoring for concourse that doesn't know BOSH? The docs describing it seem huge unless you already have the infrastructure pieces. Let's setup riemann (we already have statsd and no experience with riemann), emit to influxDB (we have prometheus, no experience with influxDB), then use Grafana (ok we already have that). I just wanted a better idea of disk, cpu, mem, # of containers, lifecycle without having to setup all these new pieces of infrastructure. Finally, just not that interested in BOSH, which all of the example metrics repo's are.
• ckok 9y ago
  I like how it looks, but first impressions after trying to make it run on windows ended up in failures: Refused to execute script from 'http://127.0.0.1:8080/public/index.js?id=932c553753eec4ef375e1c17f4b28c10' http://127.0.0.1:8080/public/index.js?id=932c553753eec4ef375... because its MIME type ('text/plain') is not executable, and strict MIME type checking is enabled.
• gerhardlazu 9y ago
  Disclaimer: I work for Pivotal, on the RabbitMQ team. We push Concourse to its limits every day. We work closely with Jenkins, GoCD, Travis & Concourse. They all have their limitations. All things will break horribly if the conditions are right. It's unreasonable to assume that the things which work in [insert your current CI] will work in Concourse. It's still a new and relatively immature product, but it works well in most cases. Half the secret to a good Concourse experience is not upgrading it in-place - stand up fresh deployments. The other half is gradually transitioning between Concourse deployments, because bad versions have been and will continue to be released - mistakes are only human. As long as you share the Concourse vision and are willing to keep up with the pace of change - not everyone can or wants to - then it's an amazing CI. Concourse still makes me excited, even after many years of hard lessons, because it is a genuinely innovative approach to building better software. Most miss this, and I understand why, but give it time - the ideas behind it will mature and become the norm. Even though Concourse can work really well, it's not always the best choice. Make it better if you can & want to, use something else if it's easier. There is no right or wrong, just preferences : )