8 ms·

Does anybody _actually_ use the HashiCorp stack, besides Vagrant, for serious work? I tried and honestly found their products sorely, sorely lacking. Very shiny

• code_research 10y ago
  unfortunately even vagrant is buggy and very slow and I would like to see a better alternative. consul and vault are very nice for devops.
• morgante 10y ago
  I use and love Terraform. Even for small projects, I find it a lot better for provisioning AWS resources than any of the alternatives.
  • Mizza 10y ago
    Have you tried Troposphere? I compared the two and found I liked Tropo much much more.
  • nixgeek 10y ago
    Seemingly every couple of days, someone on Twitter whom I follow will exclaim "Oh noes, Terraform did something bad". Seems like a 'principle of least surprise' violation. Is that something others have observed?
    • morgante 10y ago
      I've never run into that myself. I do always inspect the plan before applying, but fortunately Terraform makes that easy. Personally, I prefer it to alternatives like Boto/Troposphere because it's fully declarative and not coupled to a single cloud.
      • Leon 10y ago
        I'm going to throw one thing in here about terraform over boto / etc, it's actually really nice to have a way to put together your own modules. I think they could do a better job with some common libs to make it easier, but overall it's super powerful.
  • deleted 10y ago
    [deleted]
  • jcoby 10y ago
    I really, really want to like Terraform. I tried it on a medium project by starting with and modifying the Segment stack. I ran into constant problems with it not being able to deal with errors and not being able to modify the stack in a way that would work. I ran into several situations where I would have to change names and labels just so it could work around itself while I was trying to iterate on the stack and bring it to a workable state. To give you an idea of how many problems I had I currently have 4 different tfstate files from 4 days of testing. I had to go into AWS and manually delete resources because it couldn't recover from the errors it created. One example: I was using the ECS option and changed the container source for a service. Seems easy enough and something that should work. Terraform wedged itself after applying the change so badly that I had to blow away the entire setup to get it reset to where it could even run `plan` without erroring. Otto looked nice but it had fundamental, basic issues and it seems like nobody was actually working on it. I +1'd a bug with the PHP implementation where it didn't give you the option to change the web root and never got an update. This is something that every single decent PHP framework out there REQUIRES and wasn't supported. Otto PHP seemed like it was designed simply to work with Wordpress.
    • morgante 10y ago
      Have you considered starting your own stack from the ground up? I definitely find it a lot easier to manage and reason about if I mostly avoid third-party Terraform modules. Out of probably a dozen different Terraform projects, I've never run into a situation which I needed to manually resolve. This includes both projects which I started myself and cases where I'm helping to improve/manage client deployments. > I was using the ECS option and changed the container source for a service. What do you mean by this? I've used ECS with Terraform extensively and never had a problem with updating the container/image which a service referenced. That being said, I never used Otto. It definitely seems like they tried to bite off more than they could chew and I wasn't really interested in such high-level solutions.
      • jcoby 10y ago
        > Have you considered starting your own stack from the ground up? Yes. I'm actually in the middle of trying that now. I want to set up a vpc, a few web servers (1-10) with an autoscaling policy behind the vpc along with a bastion server and a cron server, a code deploy setup to work with autoscaling, cloudwatch logging and monitoring, a load balancer, an elasticache instance, and an rds instance. I've been working on this off an on for months. If you or anyone else can point me in a direction to simplify this I'd be grateful. The core of the problem I had with terraform (outside of the ECS issue) is that there is one AWS service that gets soft deleted. I can't remember what it is right now but it really threw tf for a loop. So I'd setup the stack, do some testing, decide to shut everything down for the day with a `terraform destroy` and the next day i couldn't resume because tf thinks the resource exists but aws doesn't think it does.
• omni 10y ago
  We use Serf for deploy triggering and lightweight messaging between our asynchronous processing nodes and it works perfectly day in and day out. Really like it, provides a great alternative to pub-sub for some use cases.
• tapsboy 10y ago
  I work for a mid-sized media company and we use Vagrant & Packer to build all our infra. AWS CF Templates have been a big pain and we are evaluating Terraform as an alternative
• lobster_johnson 10y ago
  We are evaluating Vault, which has a reputation for being a quality product (I believe). Terraform is also popular.
• nullcipher 10y ago
  We tried terraform and gave up. We do use vagrant a lot though. How does HashiCorp make money?
  • troymc 10y ago
    They have enterprise versions of Terraform, Vault, Nomad, and maybe more. You can get them all in "Atlas". They'll also host it all for you, if you want (for a fee). They charge enterprise prices for all that.
• deleted 10y ago
  [deleted]
• wise0wl 10y ago
  I work for a Healthcare start-up, and we use Consul and Vault extensively. We did find Terraform to be troublesome though, and rather difficult to work with (error handling, corrupted state files etc.). Vagrant is used sometimes for development and testing, but the Developers prefer docker (faster start up, lower footprint) and who am I to argue with progress. I will continue using Consul (it's DNS interface for service discovery is awesome) and Vault, but most everything else seems like it's a bit too ambitious and ultimately lacking.
• esigler 10y ago
  We use Serf for infrastructure command propagation ("Chef, go converge on host X", "Docker, start running container Y on host Z") across our entire staging and production sets of hosts, and Consul for holding most of our network topology information across three different geographic regions. Works pretty well. We're bringing in Nomad (edit: and Vault) now, though they still have a few rough spots.
• mitchellh 10y ago
  Ouch! My feelings are hurt. :( Everyone has a different experience and perception I suppose. I agree we have a long way to go, but I disagree that our tooling is "very incomplete" OR "un-battle-tested". Ignoring Vagrant as you did, Consul is used at multi-thousand node (per datacenter) scale by dozens of companies and a couple specific companies are using it at an even larger scale. And that's ignoring the thousands of hobbyist and smaller company usage at dozen to hundred node scale. I only don't mention specific names because I don't have explicit permission, but you'll just have to trust I don't intend to lie here. Vault as another example: if you interacted with a financial institution, your transaction at some point likely hit a Vault cluster. Did you visit some websites today? One of the world's largest CDNs has fully deployed Vault for internal TLS cert management. Those are just a couple examples. Or, ignoring tech usage completely, we just had our first seven-figure quarter after only three quarters of sales (and that was dozens of deals, not just a handful). You just can't get those sorts of numbers without real world usage. On completeness, I think the adoption speaks for itself. Tools don't get adopted at the scale they're being adopted without being complete enough to productively solve a real pain point. I believe we have a long way to go but what we have already in most of our tools is relatively complete by measure of being able to get real, meaningful, and productive work done. Its unfortunate that we can't get to every open source issue and resolve every problem for everyone, but please try to understand that the issue inbound across our projects is massive in addition to trying to build enterprise solutions for customers and run a business. We'd love to hire hire hire to handle all the community inbound but that'd be irresponsible of us financially. Our teams are slowly growing and we're also promoting more and more community members to core committers who help out quite a bit, too. Packer has ups and downs since we don't have a full team around it at the moment but its on our radar of things to work on. Ultimately, we can improve in every area and we'll strive to do so. In the process, we're motivated and encouraged by our community and also by the "serious work" we see our tools doing every day across various industries. I'll follow up with you via email to see where we've fallen short for you. I find these criticisms educational and would like to see where we went wrong. I'm not doing that to hide anything from the public, but only because its hard to have meaningful back-and-forth discourse in a few nesting levels of comments. :)
  • bl4ckcontact 10y ago
• slavik81 10y ago
  I found found packer to be quite useful for building VM images for my team. It's a little hacked together, but far better than the manual process it replaced.
  • meddlepal 10y ago
    Packer was great when I was doing immutable infrastructure with VMs, but for Docker it doesn't offer a compelling story anymore. I still think Terraform is great though.
• bsuh 10y ago
  Heck I found the same to be true of vagrant. Big promises, short on delivery. While mostly usable, it's definitely not as seamless across various guests, providers, and provisioners as their docs might have you think. Stray off the beaten path even the tiniest bit and expect to spend hours looking at github issues for workarounds (submitted by other users not the dev team). My experience. Windows guests: buggy FreeBSD guests: buggy Ansible provisioning: buggy Windows support was incorporated from a community plugin. Now it's in limbo maintained neither by the original creators nor the (seemingly) lone dev from hashicorp assigned to vagrant. They use DRM to protect their closed source proprietary plugins and it makes it a nightmare to build a version from source that works with said plugins. Building the installer is closed source for some reason. After finally being able to run with modified source, I was able to monkey patch their paid VMWare plugin to add a capability so I could use private_network with static IP on Windows guests . I posted the monkey patch to github issues and it took two weeks for their paid plugin to be updated. I wanted to love and contribute to vagrant, but the somewhat dishonest docs, unresponsive dev team, and DRM have turned me into a grudging user only because there are no alternatives. Vagrant is definitely a wolf in sheepskin in terms of appearing open source and open to community collaboration.
  • gameofdrones 10y ago
    Abstractions-on-top-of-abstractions which promise to reduce complexity by adding more levels of indirection, when the UX of the underlying tools need to be better. (Having wrote a multihypervisor disk-cloner in Ruby.)
    • gaius 10y ago
      Ah well, that's because people didn't pay enough attention to Wheeler.
  • aleksi 10y ago
    Never used Windows or FreeBSD guests, but what's wrong with Ansible provisioning?
    • 10y ago
• brobinson 10y ago
  I'd love to hear if anyone is using Nomad in production and what they think of it.
• kanwisher 10y ago
  Consul is probably one of the best pieces of software I've ever used and is the category leader for service discovery
  • nixgeek 10y ago
    I'd be interested in any data which backs this up. Things like ZooKeeper have been around a lot longer and are definitely also in widespread usage.
    • Leon 10y ago
      We've been using consul as well, it's just fantastic. Comparatively, zookeeper was a pain and didn't do all the things we wanted.
• pwelch 10y ago
  Yes! TerraForm has been huge for us. Packer has been great. We plan on rolling out Vault fully and consul to our entire infrastructure. The tools I listed above are essential along with Chef, ansible, and git are essential for our tools stack.
  • rdtsc 10y ago
    You mentioned Chef and Ansible. Using both? What are you thoughts on one vs the other? Chef seems more powerful / flexible, ansible maybe a bit simpler.
    • larrywright 10y ago
      I'm not the parent, but I've used both (Chef more heavily - going on 6 years, and I'm certified as a Chef trainer). Your assessment is roughly correct, Ansible is definitely easier to get started with, and if you just want to manage a small number of nodes that you have SSH access to, it's a good choice. Chef has a larger ecosystem, and a different communication model (VMs under management communicate out to a Chef server over HTTPS, no need for inbound ports to be opened). The tradeoff is a slightly longer learning cycle. I'm biased towards Chef because it's what I know, but having spent some time with Ansible it's a good choice for some use cases.
      • rdtsc 10y ago
        Thanks for explaining. We are using Chef heavily and we like it, but some people have been experimenting with Ansible so we started to wonder how they would compare in general. The concensus seems to be to stay with Chef.
• erichmond 10y ago
  Packer, Vagrant, Terraform, and we are working internally with Vault and Consul. Best devops tools (along with ansible) I've ever worked with.
  • gameofdrones 10y ago
    What other deveops tools have you used/want to use?
    • erichmond 10y ago
      I've been doing this professionally for 20 years, so that list is far too long to mention here.
• apparentlymart 10y ago
  At work we use Packer, Terraform and Consul across all of our apps, and little smatterings of other stuff in some places. A little on each: - Packer: Not my favorite, honestly. I can't argue that it's doing the job, but it feels inflexible and hard to integrate into a coherent workflow. It seems that Hashicorp Atlas can smooth this over in principle; we don't use it, because it didn't seem to fit with our use of Terraform at the time we got started, and we have a semi-home-grown alternative now. - Terraform: We're using Terraform not only for low-level infrastructure stuff (VPCs, subnets, etc) but also for application deployment. I'd say our success with Terraform was due to a couple things. First: we picked up Terraform at a time when we were in the process of a total infrastructure rework in our org anyway, so we were effectively starting from scratch. Second: I spent a few months using Terraform for toy things and learning what it was good at, what it was less good at, and building a "pattern library" of techniques that had worked out. Once we started applying it to real problems, we just cherry-picked suitable patterns from that library and used them. I expect that Terraform is tougher for someone who already has significant infrastructure deployed and is trying to manage it with Terraform with few changes, since there are definitely approaches that are harder to model in Terraform than others. - Consul: I really enjoy the simplicity of Consul. Getting a cluster up and running is pretty straightforward. Once you have it running, you get a highly-available, datacenter-aware key/value store and a service registry. We honestly don't use the service registry very much, but we have made extensive use of the consul-template utility in conjunction with Terraform's consul_key_prefix resource to have applications/services announce where their endpoints are for consumption by their clients. We actually decided against using Vagrant because it was "more bulky" than our app developers were willing to tolerate. Instead we continued with our previous solution (running the apps direction on the users' laptops with a README in each app describing how to get it running) being optimistic that the new Docker for Mac and Docker for Windows would be awesome enough to get the good parts of Vagrant in a lighter package. Vault showed up a bit late for our "architecture remix" so we solved our Vault-ish problems in other ways. I like its design in theory, and would probably give it a try if the opportunity arose. Similar story with Nomad: too late for us, and we'd gone down an alternative path before it showed up. Can't really speak to it, since I only dabbled with it very briefly. I'm sad but honestly not surprised to see Otto phased out. I was initially excited when it was announced last year but I could never really figure out how to get it to behave in the way I expected... I always felt like I was fighting it, and doing things in a way it didn't expect. I think there's room for the Hashicorp family of tools to "tessellate better", but Otto seemed like a very coarse, heavy solution -- essentially wrapping and templating the complex tools underneath -- where I was more hoping for the tools themselves to grow features to close the gaps. This turned in to a bit of a rant, so I'll stop. :D
• doh 10y ago
  In my experience all their tools have a pretty high barrier of entry as they require to learn in most cased a new language or at least alter your thinking. But once you master it, most of them are kicking ass. For example we use Packer daily and it's just fantastic. Yes, once you need to debug it, it's a pain in the ass. Also it makes some basic assumptions for you (for instance it turns off project wide ssh keys while provisioning [in GCE], so you have to manually add them anytime you want do something). We also use Terraform daily and it works well. Couple of bugs now and then, but it does things so effectively that once it works, you save a ton of time.
• daurnimator 10y ago
  We use terraform and I've gotten a few other people on to it.
• toddnni 10y ago
  We/I use - packer, that is a really nice and simple image building tool. It saved lot's of headache compared to tools like oz, because packer fails fast and saves debugging time. - terraform, that is comparable to OpenStack Heat or AWS CloudFormations. Each have their own issues. But terraform being platform agnostic is an advantage for me. - vagrant is not modern anymore, but it's ubiquitous.
• Dobbs 10y ago
  We use Conusl, Terraform, Packer and (sometimes) Vagrant. Terraform is probably my favorite you have to be a bit careful running it against live infrastructure but it is so nice to be able to figure out who did what to your machines rather than clicking having people click around in the AWS dashboard.
• killbrad 10y ago
  Yep, many people do.
• colemickens 10y ago
  Packer (qemu builder) is a huge step up from manually orchestrating image builds with qemu by hand. Terraform seems to be a fully unique product. No CloudFormation isn't really the same thing, nor are Azure ARM Templates. Terraform has solved multiple problems for me that were not solvable any other way except for lots of manual orchestration done with bash scripts. I haven't used Nomad yet, but I've heard good things and am happy Kubernetes/Mesos has some competition (no I don't consider Swarm Mode a real competitor yet).
• bashtoni 10y ago
  We do pretty heavily - Packer for production workloads for > 2 years - Consul for production workloads for > 1.5 years - Terraform for building production (and all other) infrastructure for > 1 year
• csomar 10y ago
  I'm having the same miserable experience with Terraform. Lacking documentation, and really un-batted-tested product. The concept is nice, and the concept of execution is nice too. The execution itself is a failure. But, hey, they know how to make shiny websites.
• eropple 10y ago
  Speaking for myself, I've put enough tooling around Packer to make it tolerable in my build pipeline and I haven't found a better tool that does what it does (suggestions welcome!). I would be using Vault, except it's just on the far edge of usable for me (particularly around the aws-ec2 provider) and I am repelled sufficiently by Go that I'm not driven to hack it to suit my needs until I exhaust my other alternatives, and I haven't yet. Consul is...fine, I guess? It's not particularly useful for me, but that's because I have a Zookeeper background and am comfortable with its primitives. I'd use it if I had a need for it. OTOH, I've had extremely bad experiences with Terraform, both in terms of trying to build tooling around it and in having it hose states when resources fail to create correctly (I've mentioned previously around here that multiple clients independently invented the term "terrafucked" for the results of their Terraform state) and I don't feel like I can trust the tool with something as critical as my cloud infrastructural resources. I might bite the bullet and use it if I were working on Google Cloud or Azure, but in AWS, CloudFormation (with tooling on top) is sufficient.
  • axelfontaine 10y ago
    If what you are after is generating AMIs for JVM, Node.js or Go apps, we at Boxfuse offer a great alternative to Packer with very fast creation of very small and secure AMIs https://boxfuse.com/blog/amis-in-30-seconds https://boxfuse.com/blog/amis-in-30-seconds Disclaimer: I am the founder and CEO of Boxfuse
    • superswordfish 10y ago
      Your limited, proprietary, paid tool? Yea...
    • eropple 10y ago
      So I spent some time studying your product, and it looks like you'd like to charge me $100/month besides for a worse experience with more constrained features than what Packer gives me. At a very fundamental level it doesn't even look like it considers the notion that not everything is an app, that high-availability, fault-tolerant services exist and need to be run, that even "immutable" servers need a consistently updated method of service discovery and credential retrieval to be effective. If I'm wrong, please let me know how I can write, as I would using Chef and Packer, a set of directives to install and prep Zookeeper, then discover other Zookeeper nodes in my cluster at runtime, while remaining aware of nodes that are replaced due to system failure. (If the explanation exists and involves "our proprietary, closed-source agent," or "our open-source agent that talks to a proprietary backend", you don't have an answer.)
• 0xmohit 10y ago
  It is unfortunate that all people can do is to complain. One should probably think that about sending a PR for a feature request or a bug report if it really impacts them. An attempt, even if in the wrong direction, would probably push the priority of the underlying issue higher for the core devs working on the project. If you can't do that, at least attempt to be constructive.
  • woodhull 10y ago
    I've found that PRs against terraform tend to languish for months at a time with no feedback from hashicorp.
  • deleted 10y ago
    [deleted]
• stephenr 10y ago
  Based on regular use of Vagrant and irregular use of Packer (to build base boxes for vagrant) I would be reluctant to rely on their other tools. They mostly work, but it's never as seamless as they make it out to be. I do appreciate the work that's put into these tools but I also believe anything can be criticised. That doesn't mean the author has to fix what everyone says is wrong but an attitude of "well it's free, either PR or stfu" doesn't help either.
• theaustinseven 10y ago
  If you use vagrant heavily, you are almost forced to use packer at one point. There are certain things you simply have to put into a base image and packer is the best way to do that. However, I will say the support for vagrant seems fairly responsive, while packer seems basically unsupported.
• dblooman 10y ago
  Used Vagrant a few years ago, but almost completely moved to Docker, it doesn't make sense to have a massive VM image anymore. Use Packer for all AMI building, probably one of the easiest tools to get started with. Use Consul for service discovery with consul-template and envconsul, all work nicely together. Terraform. I've been using for the past few months, but it has its problems. Modules use can be very tricky to manage, with no real guidelines on how to use them or TF itself. The isolation can be more complex than say, Cloudformation, which has its faults too. With CFN you have a concept of a infra stack, which will share common resources, with supplementary stacks providing additional resources. Unless you are happy to have a load of main.tf files in sub directories, which seems to be the pattern, you end up with a giant load of TF as one "stack". The benefit is never hard coding things, which you are probably going to do with CFN, plus you have a much better scope of changing resources in TF, but if you don't need that, TF can offer little over CFN with a language to generate it, like CFNDSL or troposphere. We also regularly break though our AWS API limits for TF because it touches so many things, if you have TF in a single git repo with multiple hooks, your TF can take a long time to actually plan. Recently, I did a plan on 0.7 to see what the impact was for upgrading, without realising that it would overwrite the state file to comply with 0.7, making it impossible to switch back to 0.6. I did roll back to a previous version of the state, but even that caused some issues that I had to manually resolve. Finally, Atlas. I would say that it is far from complete. There are a lot of things that UI is missing. The API for atlas is also non existent, so you have to use the UI constantly unless you want to auto deploy, which is probably fine for a dev environment, but not production. If you have a plan apply that needs manual intervention, it will sit there while commits pile up above it, so you either let apply and let atlas hit your account with massive amounts of API calls, or you cancel all the plans currently being blocked. It obviously works very different with say S3 and using TF from a CI, but i'm not sure how much focus there is on Atlas vs other products. Atlas also doesn't let you set up things like organisational access tokens, for Atlas itself and for Github. Right now, if we removed someone from our Github or Atlas org, stuff will simply break.
• nevir 10y ago
  Yes, we use the hell out of Terraform, Consul, and Vault - they're wonderful to work with