4 ms·

> ‘Engineers get sidetracked by things that make engineers excited, not that solve real problems for users’ — we’ve heard it all before. There’s no shocking rev

by cwyers 8y ago

> ‘Engineers get sidetracked by things that make engineers excited, not that solve real problems for users’ — we’ve heard it all before. There’s no shocking revelation here (you are on Medium.com after all …). But there seems to be a pervasive sense that you can’t launch a product without a K8s cluster or two, load balanced across a couple of regions, oh and if you have to deploy anything manually how can you possibly expect to turn a profit? Not everyone needs K8s, not everyone needs multi-region. But as far as manual deployment goes... it's all fun and games until someone loses an eye. Often what you find if you have a process that can't be automated as-is is that you have a process that's got problems. Maybe those problems are bugs. Maybe those problems are "only one person knows how to run this, and if he wants to take a vacation or more onto something else, we're hosed." Automation is a good thing at any scale.

yters 8y ago
Where I work they used to have an ad hoc and manual deployment strategy that could take months and sometimes resulted in an errant dev taking down production. Switching to cloud infrastructure with CI/CD pipeline seems to have made things much better, at least for the products that have switched over.
- jerguismi 8y ago
  Or you can have manual deployment strategy and gradually automatize it.
  - yters 8y ago
    Yes, that's what they sort of had before. It was pretty hacky, brittle, and slow.
  - spookthesunset 8y ago
    Why? That seems like re-inventing the wheel. What makes your deployment process so much different than average that you can’t just write a check to heroku/aws and call it good? What value exists in inventing yet another bespoke deployment system?
    - jboles 8y ago
      The deployment process might be vanilla but depending on the complexity of the software, automated validation gates before/after deployment stages might need to be quite custom to achieve required test coverage.
      - spookthesunset 8y ago
        Sure but any good off the shelf pipeline will have hooks for that...
        deleted 8y ago
        [deleted]
jerguismi 8y ago
> Automation is a good thing at any scale. Well, financially if the project is generating revenue worth of one persons salary, then it doesn't make sense to make things that would require more personnel to improve mainteinability, since you can't afford it. You just have to live with the risk.
- cwyers 8y ago
  I rarely see automation as increasing the amount of labor required so much as reducing the overall amount of labot but timeshifting it from "later" to "now." And sometimes you can't make that tradeoff, admittedly.
hjk05 8y ago
> Automation is a good thing at any scale. Once dealt with a company that spend close to 2 years to develop 2 different generations of a robotics system to glue together two plastic pieces. One of the pieces changed slightly close to the end making both robots useless. The pieces ended up being glued together manually for next to no labor cost because a person could actually do the gluing quite fast. Premature automation wrecks budgets and production lines and can kill entire companies.
- ASalazarMX 8y ago
  Those robots were heroes. Not only did they not take our jobs, they made more of them.
- dtech 8y ago
  There's a lot of difference between a gluing robot and the tasks that Devops advocates to automate. Spending a few hours on setting up an automated build on a SaaS CI server will start bringing benefits immediately and pay off very quickly. Not only quantitative (next deployments will take less human time) but also qualitative (less chance of errors/bugs/mistakes, easier to do them more frequently etc). Obligatory XKCD: https://xkcd.com/1205/ https://xkcd.com/1205/
  - jdmichal 8y ago
    The other obligatory XKCD: https://xkcd.com/1319/ https://xkcd.com/1319/
mntmoss 8y ago
Automation is just leverage. Sometimes you need the leverage of a trowel, other times you need a six-ton digger. And if you use the trowel, you can employ fewer people and skip approvals and certifications. It does make sense to automate - after you have the design. But the type of automation also changes with scale. Ideally, at a small scale, your automation can be something like: A script to pull the database for its regular backup. A script to automate uploading new PHP scripts and site assets. A script to reinstall the dependencies on a fresh image. Some version control and another backup layer over all of this. And then a huge test suite running the gamut: user workflows, failure conditions, site attacks, data integrity, backups. It's the followthrough into automating the QA processes that makes the product great, not the underlying stack(which in most cases is going to have to be treated like a placeholder in the event of real scale). And it mostly isn't substantial engineering challenges at any level: they're "tick off the boxes" exercises, they just punch above their weight in terms of delivered value. It's the tendency to add more things to configure and more systemic non-linearity("in this mode it does X, in that mode it does Y, but when Z is enabled both modes do Q") that creates a serious IT headache, and that in turn kills your progress as an independent. Oftentimes you have to give up on using the new stuff, the theoretically good stuff, because the additional layers of automation mean that you end up doing original R&D for an apparently ordinary problem, and you can't actually get a build off the ground. Or you fight through numerous issues and get it working but with no established "best practice" to follow, you only half-understand how to configure it properly, leading to technical debt and future unknowns in the risk profile.
- jboles 8y ago
  Your last paragraph sums it up well, in my opinion. Too often the answer chosen is to "add a new configuration setting", which means one more thing for ops to keep track of and one more potential cause of failure for devs to investigate. Cleaning up dead/deprecated code and working to consolidate code paths provides opportunity to clean up obsolete configuration settings, remove special cases, and improve the reasonability and maintainability of the software into the future.

by cwyers 8y ago

yters 8y ago

Where I work they used to have an ad hoc and manual deployment strategy that could take months and sometimes resulted in an errant dev taking down production. Switching to cloud infrastructure with CI/CD pipeline seems to have made things much better, at least for the products that have switched over.

jerguismi 8y ago
Or you can have manual deployment strategy and gradually automatize it.
- yters 8y ago
  Yes, that's what they sort of had before. It was pretty hacky, brittle, and slow.
- spookthesunset 8y ago
  Why? That seems like re-inventing the wheel. What makes your deployment process so much different than average that you can’t just write a check to heroku/aws and call it good? What value exists in inventing yet another bespoke deployment system?
  - jboles 8y ago
    The deployment process might be vanilla but depending on the complexity of the software, automated validation gates before/after deployment stages might need to be quite custom to achieve required test coverage.
    - spookthesunset 8y ago
      Sure but any good off the shelf pipeline will have hooks for that...
      - deleted 8y ago
        [deleted]

jerguismi 8y ago

> Automation is a good thing at any scale. Well, financially if the project is generating revenue worth of one persons salary, then it doesn't make sense to make things that would require more personnel to improve mainteinability, since you can't afford it. You just have to live with the risk.

cwyers 8y ago
I rarely see automation as increasing the amount of labor required so much as reducing the overall amount of labot but timeshifting it from "later" to "now." And sometimes you can't make that tradeoff, admittedly.

hjk05 8y ago

> Automation is a good thing at any scale. Once dealt with a company that spend close to 2 years to develop 2 different generations of a robotics system to glue together two plastic pieces. One of the pieces changed slightly close to the end making both robots useless. The pieces ended up being glued together manually for next to no labor cost because a person could actually do the gluing quite fast. Premature automation wrecks budgets and production lines and can kill entire companies.

ASalazarMX 8y ago
Those robots were heroes. Not only did they not take our jobs, they made more of them.
dtech 8y ago
There's a lot of difference between a gluing robot and the tasks that Devops advocates to automate. Spending a few hours on setting up an automated build on a SaaS CI server will start bringing benefits immediately and pay off very quickly. Not only quantitative (next deployments will take less human time) but also qualitative (less chance of errors/bugs/mistakes, easier to do them more frequently etc). Obligatory XKCD: https://xkcd.com/1205/ https://xkcd.com/1205/
- jdmichal 8y ago
  The other obligatory XKCD: https://xkcd.com/1319/ https://xkcd.com/1319/

mntmoss 8y ago

Automation is just leverage. Sometimes you need the leverage of a trowel, other times you need a six-ton digger. And if you use the trowel, you can employ fewer people and skip approvals and certifications. It does make sense to automate - after you have the design. But the type of automation also changes with scale. Ideally, at a small scale, your automation can be something like: A script to pull the database for its regular backup. A script to automate uploading new PHP scripts and site assets. A script to reinstall the dependencies on a fresh image. Some version control and another backup layer over all of this. And then a huge test suite running the gamut: user workflows, failure conditions, site attacks, data integrity, backups. It's the followthrough into automating the QA processes that makes the product great, not the underlying stack(which in most cases is going to have to be treated like a placeholder in the event of real scale). And it mostly isn't substantial engineering challenges at any level: they're "tick off the boxes" exercises, they just punch above their weight in terms of delivered value. It's the tendency to add more things to configure and more systemic non-linearity("in this mode it does X, in that mode it does Y, but when Z is enabled both modes do Q") that creates a serious IT headache, and that in turn kills your progress as an independent. Oftentimes you have to give up on using the new stuff, the theoretically good stuff, because the additional layers of automation mean that you end up doing original R&D for an apparently ordinary problem, and you can't actually get a build off the ground. Or you fight through numerous issues and get it working but with no established "best practice" to follow, you only half-understand how to configure it properly, leading to technical debt and future unknowns in the risk profile.

jboles 8y ago
Your last paragraph sums it up well, in my opinion. Too often the answer chosen is to "add a new configuration setting", which means one more thing for ops to keep track of and one more potential cause of failure for devs to investigate. Cleaning up dead/deprecated code and working to consolidate code paths provides opportunity to clean up obsolete configuration settings, remove special cases, and improve the reasonability and maintainability of the software into the future.