6 ms·

I'm frankly astounded at the complexity around modern development due to dev ops and how much worse developer experience is because of it. I used to develop in

I'm frankly astounded at the complexity around modern development due to dev ops and how much worse developer experience is because of it. I used to develop in a great IDE with debugging, right click re-run failed tests, I could follow the console right there in a nice, clean window integrated with my IDE, click on stack traces, etc. Now I'm running my app via multiple docker images, trying to get a buggy remote debugger protocol to work (no local variables, wtff?) where I have to coordinate the container and the IDE, command line wrangling / using print debugging in the container to chase down stuff, digging around through multiple log files, copying-and-pasting stack traces into my IDE to analyze, which kinda works in some cases since the file paths are different, and on and on. DevOps came and took my nice dev environment away, and I am mad.

stepbeek 5y ago
Given the second principle of devops is amplifying feedback loops, I think a more charitable view could be "the common misunderstanding of DevOps has lead to a terrible dev environment". Fwiw, I'm fully onboard with you, but really love the core message of books like the phoenix project and don't really want it to be lost because of overzealous use of docker.
- hnthrowaway0315 5y ago
  OMG, I read the sample chapters from Amazon about the Phoenix project and that was a really really good read, thank you!
  - stepbeek 5y ago
    It’s an amazingly good read. The follow up - The Unicorn Project - wasn’t as good, but definitely worth reading.
    - hnthrowaway0315 5y ago
      Thanks, really LOL a few times during the read.
excuses_ 5y ago
I second that. I experienced this drive towards overly complicated pipelines that require a bunch of external systems to work in order to test my application and claims like “it’s impossible to run it locally”. The answer should be more modular code with clear interfaces and contracts so that I can test as much as possible locally.
- Spivak 5y ago
  This really isn't a DevOps thing. If you hard depend on lots of cloud bullshit and 3rd party services but don't also put in the work to emulate it locally in your dev environment then congrats you can't really run it locally anymore. I'm in that situation now and the best solution is to just give every dev their own scaled down but complete copy of production with a little magic so the load balancer proxies requests from their dev URL to their laptop. Like it's pants-on-head stupid but I'll be damned if it didn't make onboarding new devs instant.
- 2ion 5y ago
  > require a bunch of external systems to work - unit tests should always complete locally - integration could also complete locally but are allowed more complex scenarios - in any case, we require developers to pull all their deps through a caching proxy (external system) to keep a lid on what they use, keep a lid on licenses used in our software, and keep open a door to centralized vuln tracking & management. Though if you can't fully test a software locally by getting all units and integration components to spin up, this is mostly a consequence of bad system design, for example, if the one application depends on another one which can't be run locally, reasons for which are (a) no bootstrappable dev environment (b) no test data management/generation (c) no license (d) too complex for the originating team to provide or (e) proprietary applience no one but the operating team understands. (a) through (e) all encountered at $currentjob and not happy about it, but what can you do? beg the other team to fix their shit and wait or just bite the apple and deploy to "dev stage" to get the testing done.
marvinblum 5y ago
Did it? I'm using GoLand and have exactly the experience you described. I can debug, rerun tests, terminal integrated, ... It's just a decision you took or someone else took for you. I prefer to be able to fully run my application locally, with all databases required, and test everything on my machine. DBs and other tools like a simple SMTP server run in Docker, yes, but that's just a decision I took and I could easily run these tools without containerization. Maybe I'm just in a lucky position to be able to decide this by myself, but I think we can get back there.
hermanradtke 5y ago
What you just described is not really related to devops. I agree that a complicated dev environment is bad. However, that isn’t a devops thing. You can have devops with a monolith that is statically compiled into a single binary. I’ve built it and done it. It is quite nice.
- kazen44 5y ago
  we currently do this at our company. compiled code gets (autodeployed) with SFTP in some cases even. deployment has become massively complex because of the need to keep production and test environments roughly the same. Deployment to production should not be complex and basically boils down to the following: - think about method of getting the binaries installed on the system in a secure way. - think about sending some state back that the binary has been installed succesfully. - add node/member/app/whathaveyou to the production membership pool/loadbalancer/BGP Route reflector/whathaveyou.
mindwok 5y ago
Sounds like what you're really angry at here is a case of micro-services taken to the extreme.
Noumenon72 5y ago
PyCharm Professional lets you use a remote docker-compose interpreter[1] which brings back that live debugging experience. I did have to work pretty hard to figure out the correct strings to launch Gunicorn and connect to nginx. But it's all worth it to be able to evaluate code at a breakpoint and refresh my code changes without relaunching all my containers. [1]: https://www.jetbrains.com/help/pycharm/using-docker-compose-as-a-remote-interpreter.html https://www.jetbrains.com/help/pycharm/using-docker-compose-...
waynesonfire 5y ago
exactly right. the rug was pull out from underneath us and replaced with technology lacking parity. devops was able to convince the industry that "software engineers own their own infra" is a viable thing to do so they could do engineering work, expect, they're out of their element. That's not their job. devops should be pulling in open-source technology that's developed by real engineers and write tiny packets of glue code to make it work to add value. Never mind devops teams, I remember when my team had a _BUILD_ team. Serious companies still do this, engineers don't touch production. devops and sysadmins make sure that the business software stays operational, they could care less what it is. their mandate is to make sure it's operational and they hold engineers accountable. k8s is a terrible experience and a terrible abstraction. I'm not smart enough to do better but I'm smart enough to know it terrible. It's snake-oil. My team is now migrating to the cloud, along with of course k8s, and guess what? the k8s env. in the cloud are to be managed and owned by the software engineering teams, because devops doesn't want to do their job. What a disaster. at my company, a bunch of devops folks that deployed k8s 80% of the way left soon later to make 2x the pay as contractors at other companies to help them push through their last 80%. This is another reason that there is a lack of parity on the new infrastructure. Unfortunately, everyone is still stuck at 80%. Strange.
- deanCommie 5y ago
  So much to unpack here. Let's start with: > devops was able to convince the industry that "software engineers own their own infra" is a viable thing to do so they could do engineering work, expect, they're out of their element. Devops wasn't about getting ops people to start doing development. It was about getting developers to be start doing ops because infrastructure definition became a tye of coding. What's incredible about your inversion of understanding is it's exactly the point of OP's posted article - that people misunderstand devops in exactly the way you just did. > That's not their job. devops should be pulling in open-source technology that's developed by real engineers and write tiny packets of glue code to make it work to add value. "Real engineers"? This is a Googleism and their caste system of "real engineers" and "software reliability engineers". The rest of the industry doesn't coddle their precious "real engineers". Designing a resilient, scalable, frugal, operateable architecture of load balancers, servers, caches, queues, dead letter queues, and all their associated metrics and alarms is absolutely "real engineering" and worth the time and attention of the same engineers that are building the core business value of their company. > k8s is a terrible experience and a terrible abstraction. Completely agreed. But note that that K8S came from Google and their 2-tier system of engineering. But the quote in OP's article is from Werner Vogels - AWS's CTO, and Amazon is legendary (nay infamous) for asking their engineers to work on ops. So, I agree with the last half of your post - K8s IS a scam, and should not be operated by an ops-focused team (probably not operated by anyone at all) But if you had follow the intention of the DevOps movement, not the perverted definition of DevOps, then the k8s scam was obvious from day one.
2ion 5y ago
A huge part is companies rejecting simpler solutions. For example, you can use latest systemd to basically replace all of docker and then some. It provides advanced sandboxing, resource control and job control for units and even manages "containers". Which is why my private servers mostly use systemd properly configured to provide what containers are thought to provide. The deployment is switching out a binary or add a directory tree, override a .service line and then reload and restart. Easy. Though over a 10 year period, companies, developers and operators went all-in on "container runtimes" and here we are. That being said, things like kubernetes are NOT NEW and NOT DIFFERENT from solutions like corosync+pacemaker (who remembers them?). They do the same thing, with the same levels of flexibility and extensibility and pluggability, just with containers. From my POV people who are surprised by k8s and its complexity never lived or forgot the cluster tech of the 00s. It's just about to come together as a circle, and the next iteration seems to be micro VMs instead of containers.
jeffisabelle 5y ago
I started my career as a backend developer and did a fair bit of frontend development as well. However, I'm solely doing infrastructure engineering for the last 6-7 years. I think you are kind of mistaken here, as a _devops_ I never enforced our development teams to use docker, or another specific tooling. The thing is, software engineering become a whole lot difficult and complicated over the years. When I was first learning web-development 15 years ago, notepad was the only thing you needed to do web-development, now look what you need to have to have a simple CRUD app running locally. Setting up local environments become so much harder, so people found the answer within the containers as you set it up once and you can share it within the team easily. Please do not blame _devops_ for that. You don't have to run your app via multiple docker images locally, you can still configure everything to work natively. It's just way more difficult because of the dependencies of modern technologies.
jiveturkey 5y ago
This problem you describe isn't due to devops. It's due to lack of care or understanding of the dev and deploy environments. If your company is smaller you can be part of the solution. If you company is too large for that, just move on. What you are describing is not universal.
james_s_tayler 5y ago
I use containers in production and for local development dependencies but for exactly these reasons I run the application locally via the IDE like I always have, with the option to run it locally via Docker for the times I need to dip into making sure specific bits run there fine. So, I still have all those nice things.

stepbeek 5y ago

Given the second principle of devops is amplifying feedback loops, I think a more charitable view could be "the common misunderstanding of DevOps has lead to a terrible dev environment". Fwiw, I'm fully onboard with you, but really love the core message of books like the phoenix project and don't really want it to be lost because of overzealous use of docker.

hnthrowaway0315 5y ago
OMG, I read the sample chapters from Amazon about the Phoenix project and that was a really really good read, thank you!
- stepbeek 5y ago
  It’s an amazingly good read. The follow up - The Unicorn Project - wasn’t as good, but definitely worth reading.
  - hnthrowaway0315 5y ago
    Thanks, really LOL a few times during the read.

excuses_ 5y ago

I second that. I experienced this drive towards overly complicated pipelines that require a bunch of external systems to work in order to test my application and claims like “it’s impossible to run it locally”. The answer should be more modular code with clear interfaces and contracts so that I can test as much as possible locally.

Spivak 5y ago
This really isn't a DevOps thing. If you hard depend on lots of cloud bullshit and 3rd party services but don't also put in the work to emulate it locally in your dev environment then congrats you can't really run it locally anymore. I'm in that situation now and the best solution is to just give every dev their own scaled down but complete copy of production with a little magic so the load balancer proxies requests from their dev URL to their laptop. Like it's pants-on-head stupid but I'll be damned if it didn't make onboarding new devs instant.
2ion 5y ago
> require a bunch of external systems to work - unit tests should always complete locally - integration could also complete locally but are allowed more complex scenarios - in any case, we require developers to pull all their deps through a caching proxy (external system) to keep a lid on what they use, keep a lid on licenses used in our software, and keep open a door to centralized vuln tracking & management. Though if you can't fully test a software locally by getting all units and integration components to spin up, this is mostly a consequence of bad system design, for example, if the one application depends on another one which can't be run locally, reasons for which are (a) no bootstrappable dev environment (b) no test data management/generation (c) no license (d) too complex for the originating team to provide or (e) proprietary applience no one but the operating team understands. (a) through (e) all encountered at $currentjob and not happy about it, but what can you do? beg the other team to fix their shit and wait or just bite the apple and deploy to "dev stage" to get the testing done.

marvinblum 5y ago

Did it? I'm using GoLand and have exactly the experience you described. I can debug, rerun tests, terminal integrated, ... It's just a decision you took or someone else took for you. I prefer to be able to fully run my application locally, with all databases required, and test everything on my machine. DBs and other tools like a simple SMTP server run in Docker, yes, but that's just a decision I took and I could easily run these tools without containerization. Maybe I'm just in a lucky position to be able to decide this by myself, but I think we can get back there.

hermanradtke 5y ago

What you just described is not really related to devops. I agree that a complicated dev environment is bad. However, that isn’t a devops thing. You can have devops with a monolith that is statically compiled into a single binary. I’ve built it and done it. It is quite nice.

kazen44 5y ago
we currently do this at our company. compiled code gets (autodeployed) with SFTP in some cases even. deployment has become massively complex because of the need to keep production and test environments roughly the same. Deployment to production should not be complex and basically boils down to the following: - think about method of getting the binaries installed on the system in a secure way. - think about sending some state back that the binary has been installed succesfully. - add node/member/app/whathaveyou to the production membership pool/loadbalancer/BGP Route reflector/whathaveyou.

mindwok 5y ago

Sounds like what you're really angry at here is a case of micro-services taken to the extreme.

Noumenon72 5y ago

PyCharm Professional lets you use a remote docker-compose interpreter[1] which brings back that live debugging experience. I did have to work pretty hard to figure out the correct strings to launch Gunicorn and connect to nginx. But it's all worth it to be able to evaluate code at a breakpoint and refresh my code changes without relaunching all my containers. [1]: https://www.jetbrains.com/help/pycharm/using-docker-compose-as-a-remote-interpreter.html https://www.jetbrains.com/help/pycharm/using-docker-compose-...

waynesonfire 5y ago

exactly right. the rug was pull out from underneath us and replaced with technology lacking parity. devops was able to convince the industry that "software engineers own their own infra" is a viable thing to do so they could do engineering work, expect, they're out of their element. That's not their job. devops should be pulling in open-source technology that's developed by real engineers and write tiny packets of glue code to make it work to add value. Never mind devops teams, I remember when my team had a _BUILD_ team. Serious companies still do this, engineers don't touch production. devops and sysadmins make sure that the business software stays operational, they could care less what it is. their mandate is to make sure it's operational and they hold engineers accountable. k8s is a terrible experience and a terrible abstraction. I'm not smart enough to do better but I'm smart enough to know it terrible. It's snake-oil. My team is now migrating to the cloud, along with of course k8s, and guess what? the k8s env. in the cloud are to be managed and owned by the software engineering teams, because devops doesn't want to do their job. What a disaster. at my company, a bunch of devops folks that deployed k8s 80% of the way left soon later to make 2x the pay as contractors at other companies to help them push through their last 80%. This is another reason that there is a lack of parity on the new infrastructure. Unfortunately, everyone is still stuck at 80%. Strange.

deanCommie 5y ago
So much to unpack here. Let's start with: > devops was able to convince the industry that "software engineers own their own infra" is a viable thing to do so they could do engineering work, expect, they're out of their element. Devops wasn't about getting ops people to start doing development. It was about getting developers to be start doing ops because infrastructure definition became a tye of coding. What's incredible about your inversion of understanding is it's exactly the point of OP's posted article - that people misunderstand devops in exactly the way you just did. > That's not their job. devops should be pulling in open-source technology that's developed by real engineers and write tiny packets of glue code to make it work to add value. "Real engineers"? This is a Googleism and their caste system of "real engineers" and "software reliability engineers". The rest of the industry doesn't coddle their precious "real engineers". Designing a resilient, scalable, frugal, operateable architecture of load balancers, servers, caches, queues, dead letter queues, and all their associated metrics and alarms is absolutely "real engineering" and worth the time and attention of the same engineers that are building the core business value of their company. > k8s is a terrible experience and a terrible abstraction. Completely agreed. But note that that K8S came from Google and their 2-tier system of engineering. But the quote in OP's article is from Werner Vogels - AWS's CTO, and Amazon is legendary (nay infamous) for asking their engineers to work on ops. So, I agree with the last half of your post - K8s IS a scam, and should not be operated by an ops-focused team (probably not operated by anyone at all) But if you had follow the intention of the DevOps movement, not the perverted definition of DevOps, then the k8s scam was obvious from day one.

2ion 5y ago

A huge part is companies rejecting simpler solutions. For example, you can use latest systemd to basically replace all of docker and then some. It provides advanced sandboxing, resource control and job control for units and even manages "containers". Which is why my private servers mostly use systemd properly configured to provide what containers are thought to provide. The deployment is switching out a binary or add a directory tree, override a .service line and then reload and restart. Easy. Though over a 10 year period, companies, developers and operators went all-in on "container runtimes" and here we are. That being said, things like kubernetes are NOT NEW and NOT DIFFERENT from solutions like corosync+pacemaker (who remembers them?). They do the same thing, with the same levels of flexibility and extensibility and pluggability, just with containers. From my POV people who are surprised by k8s and its complexity never lived or forgot the cluster tech of the 00s. It's just about to come together as a circle, and the next iteration seems to be micro VMs instead of containers.

jeffisabelle 5y ago

I started my career as a backend developer and did a fair bit of frontend development as well. However, I'm solely doing infrastructure engineering for the last 6-7 years. I think you are kind of mistaken here, as a _devops_ I never enforced our development teams to use docker, or another specific tooling. The thing is, software engineering become a whole lot difficult and complicated over the years. When I was first learning web-development 15 years ago, notepad was the only thing you needed to do web-development, now look what you need to have to have a simple CRUD app running locally. Setting up local environments become so much harder, so people found the answer within the containers as you set it up once and you can share it within the team easily. Please do not blame _devops_ for that. You don't have to run your app via multiple docker images locally, you can still configure everything to work natively. It's just way more difficult because of the dependencies of modern technologies.

jiveturkey 5y ago

This problem you describe isn't due to devops. It's due to lack of care or understanding of the dev and deploy environments. If your company is smaller you can be part of the solution. If you company is too large for that, just move on. What you are describing is not universal.

james_s_tayler 5y ago

I use containers in production and for local development dependencies but for exactly these reasons I run the application locally via the IDE like I always have, with the option to run it locally via Docker for the times I need to dip into making sure specific bits run there fine. So, I still have all those nice things.