4 ms·

One thing I really hate about a dedicated ops team (DevOps that is independent of devs) is that they have to use just one hammer for all kinds of nails. It's si

One thing I really hate about a dedicated ops team (DevOps that is independent of devs) is that they have to use just one hammer for all kinds of nails. It's simply impossible for a small team to maintain multiple hammers. So somehow the tools and workflow are exactly same for an app that has millions of external users and an internal app that only 10 people use. You still have to go through a whole stacks of K8S/docker who-knows-wtf-they-are configurations just to publish a dashboard from an internal IP. It took more than 8 months for my previous team to try to pull it off but it never went through.

mrweasel 5y ago
As an ops guy: You’re absolutely right. For better or worse I’m constantly annoyed by developers trying be “special”. Most projects involve stuffing things into a database, calling a webservice and reading stuff back. Don’t try to be clever about it, just stick to one type of database and Springboot so we can all go home and sleep with no incidents. But there are projects which really do need to be different, and we don’t deal with them well, and they don’t scale.
- hnthrowaway0315 5y ago
  I don't blame the devops guys though, I think this is a corporate thing as teams have to follow procedures and devops are not an exception (to waive some rules for us). I just hope it's a lot easier if we can just use a VM instead, but that would need to go through security certification. I wonder how other companies deal with this kind of issues, eventually my previous team decided to host the app on one of the laptops and tell clients to visit a "192.168.0.X:port".
- digianarchist 5y ago
  Nothing better than a team that doesn't use the internal platform. They go and write their own scripts, tools and terraform only for the developers responsible to leave the company. They found another way to skin the cat purely because they didn't like the existing approach and didn't want the hassle of getting changes integrated upstream.
  - throw868788 5y ago
    Integrating the two apps hosted on different platforms is also quite difficult normally. Things you take for granted that need to be shared, reachable, etc between apps all now need to be done manually for that snowflake platform.
nucleardog 5y ago
From the other extreme: One thing I hate about most developers is that they have to use 15 different kinds of nails to join two boards together. We have a full platform implemented, base images available, standards on how applications are developed and deployed, extensive documentation, etc. This has evolved over time along with the projects and business. This represents a _huge_ amount of work being abstracted away. But every week we have to have another discussion about pulling in the database-of-the-week or language-of-the-week because apparently we were completely unable to develop basic CRUD apps until the new hottest crypto-document-as-a-service-database came along to let us store and retrieve data. And every time they think _their_ application deserves special snowflake status. We're just making things difficult, they could fire up a VM and have this set up in 15 minutes. This completely fails to account for _most_ of the actual work involved in setting most production systems up properly. How is this being kept updated and patched? Where are the backups? Where's the monitoring? What's the disaster recovery plan? Where's the documentation? Never mind trying to integrate it into all of our existing systems (making sure it's on the appropriate subnet, has appropriately configured security groups, etc, etc). This also completely fails to account for the long tail: Even if you reimplement all of this, now it _all_ needs to be maintained for the life of your application. Any time we need to make major infrastructure-level changes, we need to account for n+1 different systems that need to be updated. If we need to do any migrations, we need to migrate n+1 things. If you need functionality X, we can't re-use the existing battle-tested modular templates to provide it to you it has to be implemented from scratch (and also maintained indefinitely). Most devs I've worked with aren't well-versed enough in ops to understand the full immediate scope nor the long tail. They only see the project as far as, and only plan to see it through as far as, get it working and shitfix as required. To your dashboard example... it's for 10 people, but that doesn't change the underlying equation for most of these things. The fact that only 10 people use it will not stop it being turned into a jumpbox for some hacker group or getting our external IPs blackholed after it sends a bunch of spam because nobody bothered to patch the OS for five years. The fact that it's only for 10 people doesn't mean it's not business critical and could be down indefinitely. The fact that it's only for 10 people doesn't mean we shouldn't have a local/staging environment and should be working in prod. The fact that it's only for 10 people doesn't mean it doesn't need to integrate into any other systems (where is it sourcing data? how is it authenticating users? how is it authenticating itself to other systems?). The fact that it's only for 10 people doesn't mean someone doesn't have to maintain it after you're gone. Following the standard _should_ be the happy path so that you don't have to worry about all this stuff. If it's not, I think your ops department was just dropping the ball.

mrweasel 5y ago

As an ops guy: You’re absolutely right. For better or worse I’m constantly annoyed by developers trying be “special”. Most projects involve stuffing things into a database, calling a webservice and reading stuff back. Don’t try to be clever about it, just stick to one type of database and Springboot so we can all go home and sleep with no incidents. But there are projects which really do need to be different, and we don’t deal with them well, and they don’t scale.

hnthrowaway0315 5y ago
I don't blame the devops guys though, I think this is a corporate thing as teams have to follow procedures and devops are not an exception (to waive some rules for us). I just hope it's a lot easier if we can just use a VM instead, but that would need to go through security certification. I wonder how other companies deal with this kind of issues, eventually my previous team decided to host the app on one of the laptops and tell clients to visit a "192.168.0.X:port".
digianarchist 5y ago
Nothing better than a team that doesn't use the internal platform. They go and write their own scripts, tools and terraform only for the developers responsible to leave the company. They found another way to skin the cat purely because they didn't like the existing approach and didn't want the hassle of getting changes integrated upstream.
- throw868788 5y ago
  Integrating the two apps hosted on different platforms is also quite difficult normally. Things you take for granted that need to be shared, reachable, etc between apps all now need to be done manually for that snowflake platform.

nucleardog 5y ago

From the other extreme: One thing I hate about most developers is that they have to use 15 different kinds of nails to join two boards together. We have a full platform implemented, base images available, standards on how applications are developed and deployed, extensive documentation, etc. This has evolved over time along with the projects and business. This represents a _huge_ amount of work being abstracted away. But every week we have to have another discussion about pulling in the database-of-the-week or language-of-the-week because apparently we were completely unable to develop basic CRUD apps until the new hottest crypto-document-as-a-service-database came along to let us store and retrieve data. And every time they think _their_ application deserves special snowflake status. We're just making things difficult, they could fire up a VM and have this set up in 15 minutes. This completely fails to account for _most_ of the actual work involved in setting most production systems up properly. How is this being kept updated and patched? Where are the backups? Where's the monitoring? What's the disaster recovery plan? Where's the documentation? Never mind trying to integrate it into all of our existing systems (making sure it's on the appropriate subnet, has appropriately configured security groups, etc, etc). This also completely fails to account for the long tail: Even if you reimplement all of this, now it _all_ needs to be maintained for the life of your application. Any time we need to make major infrastructure-level changes, we need to account for n+1 different systems that need to be updated. If we need to do any migrations, we need to migrate n+1 things. If you need functionality X, we can't re-use the existing battle-tested modular templates to provide it to you it has to be implemented from scratch (and also maintained indefinitely). Most devs I've worked with aren't well-versed enough in ops to understand the full immediate scope nor the long tail. They only see the project as far as, and only plan to see it through as far as, get it working and shitfix as required. To your dashboard example... it's for 10 people, but that doesn't change the underlying equation for most of these things. The fact that only 10 people use it will not stop it being turned into a jumpbox for some hacker group or getting our external IPs blackholed after it sends a bunch of spam because nobody bothered to patch the OS for five years. The fact that it's only for 10 people doesn't mean it's not business critical and could be down indefinitely. The fact that it's only for 10 people doesn't mean we shouldn't have a local/staging environment and should be working in prod. The fact that it's only for 10 people doesn't mean it doesn't need to integrate into any other systems (where is it sourcing data? how is it authenticating users? how is it authenticating itself to other systems?). The fact that it's only for 10 people doesn't mean someone doesn't have to maintain it after you're gone. Following the standard _should_ be the happy path so that you don't have to worry about all this stuff. If it's not, I think your ops department was just dropping the ball.