7 ms·

What do I do when one of Otto's magic commands breaks my website in production? Will I ever have to give Otto my root password? I am deeply afraid of this too

by gcr 11y ago

What do I do when one of Otto's magic commands breaks my website in production? Will I ever have to give Otto my root password? I am deeply afraid of this tool. The marketing heavily pushes the angle of "Don't worry, it's magic! It knows everything! You won't have to break a sweat!" But statements like this, while appealing to newcomers, sicken the people who understand the stack well enough to also understand why this tool could help them. If Otto wants to gain mindshare of the devops people who will be forced to support it, I strongly suggest they reconsider their branding. I really, really hope I never have to use this tool.

mitchellh 11y ago
Good questions, and I'm happy to answer. I'm going to answer with a philosophical point of view. I'm not trying to dodge the question, but I'm hoping it provides you some idea of how we're approaching the problem. If not, just let me know I can try to clarify. The bigger philosophical idea is the centralization of knowledge. How does Otto know how to deploy a PHP application? Because people who can be considered experts in PHP encoded their knowledge of how to do it. You no longer have to be that expert. (Note that 0.1 deploys PHP in an awful way, 0.2 will do a lot better, thanks to the "experts" coming in). Our goal is to centralize dev/deploy knowledge so that we can focus on higher level problems and teach Otto how to do the details. The next question is how do I know its safe? Under the covers, Otto uses a lot of production proven tooling. On top of that, we have a really extensive acceptance test suite to verify behavior against real things prior to release. We're doing the best we can do, but Otto will need to earn a lot of trust. We're going to build that trust over time. Note that you can always inspect the ".otto" directory after compiling your Appfile to see the configurations it generates. You can choose to not use Otto's deployment features. That is completely fine. Our goal is that in less than a year, it does what you'd want it do (help us by teaching it!), and you'll gain trust for it. We have extensive acceptance tests to verify certain behaviors. Of course, we can't cover every edge case but we are going to do our best. This will be a trust building process. And, I don't blame you being afraid. It is a very new tool that is trying something that is threatening to a lot of people. No worries. Just wait a while; maybe it earns your trust, maybe it doesn't. Take a look at our other tools if you want absolute control, Otto might not be for you. :) Finally, and this is not a negative thing: the HN crowd is generally not the crowd Otto is aimed at. HN folks are tinkerers, they want to know how things work. That is what makes HN great. They question things and want to know the full details. Otto scares people like that because it _removes_ power from you (much in the same way compilers removed power from assembly programmers). For these folks, I ask you to take a look at the tools that are under Otto: Consul, Nomad, Packer, Terraform, even Vagrant. They provide absolute control. But, Otto is already making great great adoption in the group of people who think "I don't care how, just deploy this application." Think folks that work with well known technologies like the LAMP stack, Wordpress, Rails, etc. all day. Otto for them is fairly revolutionary, because this is a tool that actually makes them more productive and simplifies thing, vs. a lot of the other tools even we've built which appear to add complexity. And for the folks that want simplicity, what Otto is trying to do is give them industry best practices in addition to simplicity. As an extreme example: we want `otto deploy` to be easier than FTP-ing PHP files to a server, because it isn't a best practice for many reasons. We want you to get an AWS infrastructure designed by experts deployed onto a server that is configured by experts with supporting tools for monitoring, logging, security, and more that only experts would really properly use and configure. These are Otto's aspirations. I believe we can get there, but Otto obviously is new so we haven't proven it yet. But I'm going to try. I hope that clarifies things!
- twa927 11y ago
  It's OK to use heuristics and "magic" in many cases - when it's acceptable to have correctness rate lower than 100%. When an IDE generates some code it's OK that it's correct in 90% cases only, because I can review the code and correct it. But for a deployment tool even 99.99% correctness is not acceptable - it must be 100%. That's why I wouldn't use a deployment tool built on heuristics. You can't make it correct 100% even when having huge number of tests. (unless the tool works like a magic config generator only that I can review)
  - mentat 11y ago
    I'm not sure what real world deployment tools you're talking about, but 99.99% would be pretty good. Actually distributed deployment is "a hard problem".
  - brianmurphy 11y ago
    For the same reason that "90%" okay IDE code is acceptable for most people, 90% correct deployment will also be okay for most. Primarily because less experienced users could be doing a 40 - 60% optimal deployment today.
  - coldtea 11y ago
    I think you're overstating your case. Have you ever seen a 100% correct deployment? Even if someone thinks they have, I doubt it wont be revealed to have numerous issues.
  - jacques_chester 11y ago
    I've worked on the Cloud Foundry buildpacks team. What mitchellh describes for PHP (taking his example) is something that Cloud Foundry already does: take a codebase, apply heuristics to build and package it, then pass it off to a runtime. Heroku and OpenShift do it too. The thing is that you're worried about snowflake deployment. You're still in the mindset that your app deserves special treatment; if it doesn't stage and launch using community-accepted best practices, then the platform is to blame. I disagree. If your app needs zany configurations just to launch, I think that's a smell. From time to time on buildpacks we would receive bug reports of the form "my app doesn't work". Usually, upon closer inspection, there was an easier fix: follow best practices. The thing to remember about PaaSes like Cloud Foundry, Heroku, OpenShift and now Otto is that they are opinionated by design. The goal is not to provide an infinite supply of knobs and dials. We tried that with mod_perl, with PHP, with Java. It's a freaking nightmare made of sentient, glutinous mud that wants to eat your soul. Our job as engineers is not to tinker. It's to create value for users and businesses. Clinging to the illusion of control is wasteful. When I fly I don't care about how the plane works; I just care that it's fast enough, safe enough and cheap enough.
meddlepal 11y ago
I think this is the beginning of the end of DevOps for small organizations. Release and infrastructure engineers are expensive.
- lambdasquirrel 11y ago
  I wouldn't mind if my job went away. I don't see the point in forcing engineering on startups that shouldn't need it. I'd rather be messing around with my image processing code, not fscking with the damn deploy.
  - anentropic 11y ago
    amen to that
lambdasquirrel 11y ago
I don't know man, isn't that a little knee-jerk? If Otto can break your website in production, then any deploy script could break your website in production. I mean yea, I have a script right now that can bring my company's website down. Since I happen to be in ancient python land, it's just `fab conf:web,prod stop`. Devops has always had this capability. :P I think they're working on some security thing called Vault if you want to take care of password management. In the meantime, literally every other shop in SF is storing half their passwords in plaintext, in code (optimistically. things are probably worse). It's really hard to understate how bad things are. I'm not buying that these tools are going to make devops go away. I hope that these tools will mean we spend less time taking care of seriously dumb stuff in devops.
- IanCal 11y ago
  > If Otto can break your website in production, then any deploy script could break your website in production. The concern isn't "could it break my website?" but "what do I do if it does?". The more black-box something is, the harder it is for me to quickly fix something / override whatever weird bug my deployment script has done.
biomene 11y ago
> If Otto wants to gain mindshare of the devops people who will be forced to support it, I strongly suggest they reconsider their branding. It sounds like Otto isn't aimed at devops people at all. It's aimed at the poor (in the material sense) developers who have to churn out 10 Wordpress sites a year to make a living wage and don't want to spend half their time figuring out how the permission system works.
- gcr 11y ago
  That's....a great point. As the saying goes, "If you don't like the advertisement, you're not in the target market."
  - jacques_chester 11y ago
    On Cloud Foundry we've found that we get buy-in from both operators and developers. Operators heave a sigh of relief: "Finally, I don't have to clean up after devs! They can't break the platform for everyone else! I don't get 3AM phonecalls to fix someone else's mess!" Developers heave a sigh of relief: "Finally, I can just deploy without dealing with suspicious operators! If it breaks I don't get berated by an operator who got a 3AM phonecall!" I imagine OpenShift and Heroku guys can tell the same stories. I work for Pivotal Labs, so I get to see projects with PaaSes and projects without PaaSes in a variety of companies, on a variety of platforms, with a variety of goals. But one thing remains constant: PaaS-deployed projects are easier. In a PaaS-deployment project, you set up your credentials, set up CI/CD, then you just get on with doing the thing that matters: building a product. In non-PaaS projects, you spend eye-blistering amounts of time and money trying to just deploy the damn software. The whole feedback loop turns into a tangled pile of spaghetti behind somebody's bespoke deployment process. Backpressure causes enormous shear stresses throughout the project because development is iterating quickly but there's no way to, you know, see if the users like it or to know if it actually works in the real environment. It always boggles my mind that folk are happier to squander amazing amounts of engineering time and salary dollars instead of just renting or installing a PaaS.
seivan 11y ago
Would you be equally afraid of Capistrano?
dunkelheit 11y ago
That were my initial thoughts as well, but after reading mitchellh's and others' responses it made more sense. Developers are rightly weary and suspicious of bad abstractions. And what separates good abstraction from a bad one is how fast you are going to trip over its scope of applicability and how surprising it will be. Good general purpose abstractions for distributed services are not there yet but I think this tool will be successful if it will be a good enough abstraction for a large enough cohort of users. And those for whom it is not enough will be able to learn a thing or two by peeking under the hood. But I agree with you wholeheartedly that reading through the marketing copy to calibrate what my expectations regarding this tool should be is very tiring. I really wish there were a little link titled "explanation for technical-minded people" leading to a clear description of what it really does behind the curtain without any bullshytt words like "boost to productivity", "scalable" or "best-in-class".

by gcr 11y ago

mitchellh 11y ago

Good questions, and I'm happy to answer. I'm going to answer with a philosophical point of view. I'm not trying to dodge the question, but I'm hoping it provides you some idea of how we're approaching the problem. If not, just let me know I can try to clarify. The bigger philosophical idea is the centralization of knowledge. How does Otto know how to deploy a PHP application? Because people who can be considered experts in PHP encoded their knowledge of how to do it. You no longer have to be that expert. (Note that 0.1 deploys PHP in an awful way, 0.2 will do a lot better, thanks to the "experts" coming in). Our goal is to centralize dev/deploy knowledge so that we can focus on higher level problems and teach Otto how to do the details. The next question is how do I know its safe? Under the covers, Otto uses a lot of production proven tooling. On top of that, we have a really extensive acceptance test suite to verify behavior against real things prior to release. We're doing the best we can do, but Otto will need to earn a lot of trust. We're going to build that trust over time. Note that you can always inspect the ".otto" directory after compiling your Appfile to see the configurations it generates. You can choose to not use Otto's deployment features. That is completely fine. Our goal is that in less than a year, it does what you'd want it do (help us by teaching it!), and you'll gain trust for it. We have extensive acceptance tests to verify certain behaviors. Of course, we can't cover every edge case but we are going to do our best. This will be a trust building process. And, I don't blame you being afraid. It is a very new tool that is trying something that is threatening to a lot of people. No worries. Just wait a while; maybe it earns your trust, maybe it doesn't. Take a look at our other tools if you want absolute control, Otto might not be for you. :) Finally, and this is not a negative thing: the HN crowd is generally not the crowd Otto is aimed at. HN folks are tinkerers, they want to know how things work. That is what makes HN great. They question things and want to know the full details. Otto scares people like that because it _removes_ power from you (much in the same way compilers removed power from assembly programmers). For these folks, I ask you to take a look at the tools that are under Otto: Consul, Nomad, Packer, Terraform, even Vagrant. They provide absolute control. But, Otto is already making great great adoption in the group of people who think "I don't care how, just deploy this application." Think folks that work with well known technologies like the LAMP stack, Wordpress, Rails, etc. all day. Otto for them is fairly revolutionary, because this is a tool that actually makes them more productive and simplifies thing, vs. a lot of the other tools even we've built which appear to add complexity. And for the folks that want simplicity, what Otto is trying to do is give them industry best practices in addition to simplicity. As an extreme example: we want `otto deploy` to be easier than FTP-ing PHP files to a server, because it isn't a best practice for many reasons. We want you to get an AWS infrastructure designed by experts deployed onto a server that is configured by experts with supporting tools for monitoring, logging, security, and more that only experts would really properly use and configure. These are Otto's aspirations. I believe we can get there, but Otto obviously is new so we haven't proven it yet. But I'm going to try. I hope that clarifies things!

twa927 11y ago
It's OK to use heuristics and "magic" in many cases - when it's acceptable to have correctness rate lower than 100%. When an IDE generates some code it's OK that it's correct in 90% cases only, because I can review the code and correct it. But for a deployment tool even 99.99% correctness is not acceptable - it must be 100%. That's why I wouldn't use a deployment tool built on heuristics. You can't make it correct 100% even when having huge number of tests. (unless the tool works like a magic config generator only that I can review)
- mentat 11y ago
  I'm not sure what real world deployment tools you're talking about, but 99.99% would be pretty good. Actually distributed deployment is "a hard problem".
- brianmurphy 11y ago
  For the same reason that "90%" okay IDE code is acceptable for most people, 90% correct deployment will also be okay for most. Primarily because less experienced users could be doing a 40 - 60% optimal deployment today.
- coldtea 11y ago
  I think you're overstating your case. Have you ever seen a 100% correct deployment? Even if someone thinks they have, I doubt it wont be revealed to have numerous issues.
- jacques_chester 11y ago
  I've worked on the Cloud Foundry buildpacks team. What mitchellh describes for PHP (taking his example) is something that Cloud Foundry already does: take a codebase, apply heuristics to build and package it, then pass it off to a runtime. Heroku and OpenShift do it too. The thing is that you're worried about snowflake deployment. You're still in the mindset that your app deserves special treatment; if it doesn't stage and launch using community-accepted best practices, then the platform is to blame. I disagree. If your app needs zany configurations just to launch, I think that's a smell. From time to time on buildpacks we would receive bug reports of the form "my app doesn't work". Usually, upon closer inspection, there was an easier fix: follow best practices. The thing to remember about PaaSes like Cloud Foundry, Heroku, OpenShift and now Otto is that they are opinionated by design. The goal is not to provide an infinite supply of knobs and dials. We tried that with mod_perl, with PHP, with Java. It's a freaking nightmare made of sentient, glutinous mud that wants to eat your soul. Our job as engineers is not to tinker. It's to create value for users and businesses. Clinging to the illusion of control is wasteful. When I fly I don't care about how the plane works; I just care that it's fast enough, safe enough and cheap enough.

meddlepal 11y ago

I think this is the beginning of the end of DevOps for small organizations. Release and infrastructure engineers are expensive.

lambdasquirrel 11y ago
I wouldn't mind if my job went away. I don't see the point in forcing engineering on startups that shouldn't need it. I'd rather be messing around with my image processing code, not fscking with the damn deploy.
- anentropic 11y ago
  amen to that

lambdasquirrel 11y ago

I don't know man, isn't that a little knee-jerk? If Otto can break your website in production, then any deploy script could break your website in production. I mean yea, I have a script right now that can bring my company's website down. Since I happen to be in ancient python land, it's just `fab conf:web,prod stop`. Devops has always had this capability. :P I think they're working on some security thing called Vault if you want to take care of password management. In the meantime, literally every other shop in SF is storing half their passwords in plaintext, in code (optimistically. things are probably worse). It's really hard to understate how bad things are. I'm not buying that these tools are going to make devops go away. I hope that these tools will mean we spend less time taking care of seriously dumb stuff in devops.

IanCal 11y ago
> If Otto can break your website in production, then any deploy script could break your website in production. The concern isn't "could it break my website?" but "what do I do if it does?". The more black-box something is, the harder it is for me to quickly fix something / override whatever weird bug my deployment script has done.

biomene 11y ago

> If Otto wants to gain mindshare of the devops people who will be forced to support it, I strongly suggest they reconsider their branding. It sounds like Otto isn't aimed at devops people at all. It's aimed at the poor (in the material sense) developers who have to churn out 10 Wordpress sites a year to make a living wage and don't want to spend half their time figuring out how the permission system works.

gcr 11y ago
That's....a great point. As the saying goes, "If you don't like the advertisement, you're not in the target market."
- jacques_chester 11y ago
  On Cloud Foundry we've found that we get buy-in from both operators and developers. Operators heave a sigh of relief: "Finally, I don't have to clean up after devs! They can't break the platform for everyone else! I don't get 3AM phonecalls to fix someone else's mess!" Developers heave a sigh of relief: "Finally, I can just deploy without dealing with suspicious operators! If it breaks I don't get berated by an operator who got a 3AM phonecall!" I imagine OpenShift and Heroku guys can tell the same stories. I work for Pivotal Labs, so I get to see projects with PaaSes and projects without PaaSes in a variety of companies, on a variety of platforms, with a variety of goals. But one thing remains constant: PaaS-deployed projects are easier. In a PaaS-deployment project, you set up your credentials, set up CI/CD, then you just get on with doing the thing that matters: building a product. In non-PaaS projects, you spend eye-blistering amounts of time and money trying to just deploy the damn software. The whole feedback loop turns into a tangled pile of spaghetti behind somebody's bespoke deployment process. Backpressure causes enormous shear stresses throughout the project because development is iterating quickly but there's no way to, you know, see if the users like it or to know if it actually works in the real environment. It always boggles my mind that folk are happier to squander amazing amounts of engineering time and salary dollars instead of just renting or installing a PaaS.

seivan 11y ago

Would you be equally afraid of Capistrano?

dunkelheit 11y ago

That were my initial thoughts as well, but after reading mitchellh's and others' responses it made more sense. Developers are rightly weary and suspicious of bad abstractions. And what separates good abstraction from a bad one is how fast you are going to trip over its scope of applicability and how surprising it will be. Good general purpose abstractions for distributed services are not there yet but I think this tool will be successful if it will be a good enough abstraction for a large enough cohort of users. And those for whom it is not enough will be able to learn a thing or two by peeking under the hood. But I agree with you wholeheartedly that reading through the marketing copy to calibrate what my expectations regarding this tool should be is very tiring. I really wish there were a little link titled "explanation for technical-minded people" leading to a clear description of what it really does behind the curtain without any bullshytt words like "boost to productivity", "scalable" or "best-in-class".