5 ms·

> There is an overwhelming amount of good advice, practices, tools, and processes that you should use to improve. I disagree. Advice is contradictory, practice

by dvt 6y ago

> There is an overwhelming amount of good advice, practices, tools, and processes that you should use to improve. I disagree. Advice is contradictory, practices and processes are often orthogonal, and tools quite literally don't exist. I say this as someone that's worked in large companies and saw how lengthy not only process feedback loops were (especially developer ↔ product team), but also engineering feedback loops (testing, deployment, dev environment setup, etc.) And I also say this as someone that works on side-projects by myself. The tooling simply isn't there. Setting up fast engineering feedback loops (multi-tiered deployment, test, dev, local, prod, etc. environments, heck even hot reloading or debugging) is needlessly difficult, or janky, or simply impossible. This goes for just about any language/framework out there: from Java, to Javascript, to Go. It's not surprising that FAANG often designs their own internal systems that handle this sort of thing (from DevOps, to automated testing, to A/B testing, to deployment). I'm a believer that this is also a space that's ripe for disruption. I want environments to just work. I want code to just compile. I want containers to just run. Without me having to start digging through documentation, looking at thirteen Stack Overflow threads, and cobbling a solution that will inevitably break 3 months from now.

peterkos 6y ago
> It's not surprising that FAANG often designs their own internal systems that handle this sort of thing This is one reason why I've loved using Notion. I maintain a few open source repos, and our team's process has grown organically over the past few months. Being able to use a Kanban board for one set of information, then decide that it's just better as a bulleted list; storing meeting notes from quick bullet points up to detailed step-by-step guides on how to fix some issue people had that week... The low friction to high flexibility combo is great!
jiggawatts 6y ago
I'm going through this right now. I'm helping a customer deploy Azure App Service (similar to AWS Elastic Beanstalk). Jesus wept. It's sooo fiddly to set up everything. The prod and non-prod environments need dozens of parameters that are all slightly different. Some of these are exposed as resource properties, making them obvious, some are environment variables that are nearly undocumented. Everything is off by default and has to be turned on. Logging. Monitoring. High availability. Secure parameter storage. Staging. On and on.... Oh, you want high availability? Sure, if you pay an extra $3K per month and switch to internal only networking, build Azure DevOps "agent" VMs in a zone-redundant set, and register a bunch of private DNS zones. Good luck automating this. There are template samples available, but they've been broken for years. Regional disaster recovery? Just duplicate everything! Okay, but not like that! Slightly differently because replication is asymmetric and you may not want active-active. Then you have to layer stuff on top to actually fail over. But not one thing at a time! That's a performance disaster. All or nothing, but that's not an Azure feature, so good luck with that... This is what I thought the public cloud provides: Some place to upload a ZIP file of code that "just runs" without me having to baby-site the infrastructure. What's actually provided is a bunch of infrastructure parts that the public cloud vendor will patch for you. Everything else is your problem.
- data_ders 6y ago
  Azure product teams could do with a bit more customer obsession IMHO. And not just the existing customers you’ve had for 20 years with stale IT departments
  - jiggawatts 6y ago
    I recently opened a bunch of support tickets with them because core product functionality is either missing or broken. For every ticket they've come back with: It is like that by design, go complain on the feedback forums. Like the other 10 people that also had the same issue and did complain? Their "feedback" was marked as rejected...
- dagss 6y ago
  Things are not perfect anywhere but Azure is a bit of a special case. It seems optimized for unskilled people being able to stumble in the right direction by using IDEs and clicking in portal; not for efficiently streamlining development processes. ARM system in particular is deeply broken in so many ways. All cloud providers could have been a lot better but Azure is worse than average. (Yes I have to usw Azure a lot)
  - jiggawatts 6y ago
    Conceptually the resource manager part of ARM is actually fairly elegant. The implementation on the server-side is a bit messy. Things like read-only properties being mixed in with read-write properties, and internal code-names used for final production systems. Then the actual client-side tooling is a disaster. The ARM template language is just raw JSON, but isn't case sensitive and includes comments and a bizarre expression language. It's very weakly typed and then even ignores the weak typing that it does have. I've lost count of the times I've been forced to put a number into a string because... reasons. They're working on a replacement language called BICEP, but IMHO they should have gone down the same path that Pulumi did.
addicted 6y ago
In my experience for any particular aspect of software development, if you ask for advice, you will find 100 people saying about 10 different things. It sounds confusing and contradictory. And it's likely that 1 of those ten things is much better than the others. However, the gap between what you are doing right now and the worst of those ten things is almost certainly much greater than the gap between the worst of the ten things and the best. IOW, just go ahead and implement one of the potentially contradictory improvements that people on HN, for example, have suggested. You will already be way better off than before. And once you start doing this often enough you will get to the point where you're able to identify the best 2 or 3 of the 10 options presented almost immediately. And the gap then between #3 and #1 is small enough to possibly not even matter.
- naniwaduni 6y ago
  > However, the gap between what you are doing right now and the worst of those ten things is almost certainly much greater than the gap between the worst of the ten things and the best. I can't really get behind this sentiment, since in my experience the worst of those ten things is frequently not only ineffective (which would be bad enough since switching costs are very real), but actively harmful compared to whatever you're doing. Chances are you didn't arrive at your current processes entirely in a vacuum, and it's already a mishmash of ideas from the zeitgeist, filtered to not being the worst on all dimensions by the fact that you're still around to observe it. You really need to be able to pick out the good ideas from the bad ideas, and that doesn't come readily to everyone.
  - addicted 6y ago
    Well, my assumption is that if you're seeking help (a) what you have isnt currently working, and that (b) you're seeking help at a place that usually provides decent info such as HN or Stack Overflow. Or maybe I've become experienced enough to distinguish between the good and the bad ideas without realizing it and don't really remember the earlier days when I had mentors who would help me navigate those decisions.
- yen223 6y ago
ehnto 6y ago
> I want environments to just work. I want code to just compile. I want containers to just run. Without me having to start digging through documentation, looking at thirteen Stack Overflow threads, and cobbling a solution that will inevitably break 3 months from now. Because we are always shifting frameworks and approaches, there are few consistent targets for tools like that. What you might call the "Developer Experience" tooling ecosystem, is woefully immature for this fact. I would way rather use old approaches with mature developer experience tooling than new approaches with constantly breaking environments and pipelines. But we constantly start all over again with new approaches, and never quite make it to the mature tooling phase...
- dvt 6y ago
  I'd agree with you, but people were saying this exact thing about automating deployments and then Heroku came along and changed the whole game. Terraform is trying to solve this but it's still a far cry from "Heroku for infra."
torbital 6y ago
> I want environments to just work. I want code to just compile. I want containers to just run. Without me having to start digging through documentation, looking at thirteen Stack Overflow threads, and cobbling a solution that will inevitably break 3 months from now. Sometimes I conjure up the inspiration to actually implement an idea that's been on my mind. Then I spend half a day trying to setup an environment that just fucking compiles. I'll spend several days kicking at an open door before I actually write any code. In most cases I just give up before this point.
- pantelisk 6y ago
  If you don't mind me asking. What stack are you working with? And how do you approach things? My experience has been very different (was able to validate ideas and mvps extremely fast, sometimes even in a day/night - but I am very product driven which I think makes the whole difference (meaning that I care to see if the idea works from an end-user perspective first, the rest comes after and only after the project has grown for a while))
- dgellow 6y ago
  Do you work with C++? Because that’s how it feels to me every time I decide to go with the language
  - MaxBarraclough 6y ago
    Are you thinking of the build system specifically?
cjfd 6y ago
I think the biggest factor is in this is that so many different tools are being used. Often tools are brought in to make something easier or solve some problem but nobody seems to be thinking about the huge cost that this entails in terms of keeping all of these moving parts working. I think staying with older and more solid stuff that is not in a rapid flux of so-called improvement is often a much more pleasant experience. I think the barrier for introducing some new kind of tooling should be much higher than it often is in practice.
karl42 6y ago
> I want environments to just work. I want code to just compile. Only mature tools can be complex and still work. But many projects and ecosystems increase complexity much faster than their maturity can set it. I think this loss of balance between maturity and complexity is the root cause of most of the brokenness we experience as developers.
bpicolo 6y ago
Heroku has been the dream all along. I'm very spike on fast-compiling, statically typed monoliths for orgs of most sizes - especially for product-shaped companies versus infrastructure. Once you go distributed, so much becomes somuch harder, and the amount of dev effectiveness focused engineers required to keep the effectiveness high becomes prohibitive until you hit a much larger scale. Logging, debugging, scaling, keeping dependencies up to date, A/B testing and feature flags, and perhaps the biggest one: developer environments.
aiisjustanif 6y ago
Have you heard of BackStage [1]? A few companies have been adopting it to help make teams and departments to help devs. Target is a notable example of going from a really unproductive environment to much more supportive of devs [2]. Including handling building out infrastructure stacks you mentioned. * 1. https://backstage.io/ https://backstage.io/ * 2. https://tech.target.com/ https://tech.target.com/
anthony_barker 6y ago
I find it depressing. People talk of an AI revolution but it is super hard just to setup some baseline infrastructure on these "AI" providers infra.

by dvt 6y ago

peterkos 6y ago

> It's not surprising that FAANG often designs their own internal systems that handle this sort of thing This is one reason why I've loved using Notion. I maintain a few open source repos, and our team's process has grown organically over the past few months. Being able to use a Kanban board for one set of information, then decide that it's just better as a bulleted list; storing meeting notes from quick bullet points up to detailed step-by-step guides on how to fix some issue people had that week... The low friction to high flexibility combo is great!

jiggawatts 6y ago

I'm going through this right now. I'm helping a customer deploy Azure App Service (similar to AWS Elastic Beanstalk). Jesus wept. It's sooo fiddly to set up everything. The prod and non-prod environments need dozens of parameters that are all slightly different. Some of these are exposed as resource properties, making them obvious, some are environment variables that are nearly undocumented. Everything is off by default and has to be turned on. Logging. Monitoring. High availability. Secure parameter storage. Staging. On and on.... Oh, you want high availability? Sure, if you pay an extra $3K per month and switch to internal only networking, build Azure DevOps "agent" VMs in a zone-redundant set, and register a bunch of private DNS zones. Good luck automating this. There are template samples available, but they've been broken for years. Regional disaster recovery? Just duplicate everything! Okay, but not like that! Slightly differently because replication is asymmetric and you may not want active-active. Then you have to layer stuff on top to actually fail over. But not one thing at a time! That's a performance disaster. All or nothing, but that's not an Azure feature, so good luck with that... This is what I thought the public cloud provides: Some place to upload a ZIP file of code that "just runs" without me having to baby-site the infrastructure. What's actually provided is a bunch of infrastructure parts that the public cloud vendor will patch for you. Everything else is your problem.

data_ders 6y ago
Azure product teams could do with a bit more customer obsession IMHO. And not just the existing customers you’ve had for 20 years with stale IT departments
- jiggawatts 6y ago
  I recently opened a bunch of support tickets with them because core product functionality is either missing or broken. For every ticket they've come back with: It is like that by design, go complain on the feedback forums. Like the other 10 people that also had the same issue and did complain? Their "feedback" was marked as rejected...
dagss 6y ago
Things are not perfect anywhere but Azure is a bit of a special case. It seems optimized for unskilled people being able to stumble in the right direction by using IDEs and clicking in portal; not for efficiently streamlining development processes. ARM system in particular is deeply broken in so many ways. All cloud providers could have been a lot better but Azure is worse than average. (Yes I have to usw Azure a lot)
- jiggawatts 6y ago
  Conceptually the resource manager part of ARM is actually fairly elegant. The implementation on the server-side is a bit messy. Things like read-only properties being mixed in with read-write properties, and internal code-names used for final production systems. Then the actual client-side tooling is a disaster. The ARM template language is just raw JSON, but isn't case sensitive and includes comments and a bizarre expression language. It's very weakly typed and then even ignores the weak typing that it does have. I've lost count of the times I've been forced to put a number into a string because... reasons. They're working on a replacement language called BICEP, but IMHO they should have gone down the same path that Pulumi did.

addicted 6y ago

In my experience for any particular aspect of software development, if you ask for advice, you will find 100 people saying about 10 different things. It sounds confusing and contradictory. And it's likely that 1 of those ten things is much better than the others. However, the gap between what you are doing right now and the worst of those ten things is almost certainly much greater than the gap between the worst of the ten things and the best. IOW, just go ahead and implement one of the potentially contradictory improvements that people on HN, for example, have suggested. You will already be way better off than before. And once you start doing this often enough you will get to the point where you're able to identify the best 2 or 3 of the 10 options presented almost immediately. And the gap then between #3 and #1 is small enough to possibly not even matter.

naniwaduni 6y ago
> However, the gap between what you are doing right now and the worst of those ten things is almost certainly much greater than the gap between the worst of the ten things and the best. I can't really get behind this sentiment, since in my experience the worst of those ten things is frequently not only ineffective (which would be bad enough since switching costs are very real), but actively harmful compared to whatever you're doing. Chances are you didn't arrive at your current processes entirely in a vacuum, and it's already a mishmash of ideas from the zeitgeist, filtered to not being the worst on all dimensions by the fact that you're still around to observe it. You really need to be able to pick out the good ideas from the bad ideas, and that doesn't come readily to everyone.
- addicted 6y ago
  Well, my assumption is that if you're seeking help (a) what you have isnt currently working, and that (b) you're seeking help at a place that usually provides decent info such as HN or Stack Overflow. Or maybe I've become experienced enough to distinguish between the good and the bad ideas without realizing it and don't really remember the earlier days when I had mentors who would help me navigate those decisions.
yen223 6y ago

ehnto 6y ago

> I want environments to just work. I want code to just compile. I want containers to just run. Without me having to start digging through documentation, looking at thirteen Stack Overflow threads, and cobbling a solution that will inevitably break 3 months from now. Because we are always shifting frameworks and approaches, there are few consistent targets for tools like that. What you might call the "Developer Experience" tooling ecosystem, is woefully immature for this fact. I would way rather use old approaches with mature developer experience tooling than new approaches with constantly breaking environments and pipelines. But we constantly start all over again with new approaches, and never quite make it to the mature tooling phase...

dvt 6y ago
I'd agree with you, but people were saying this exact thing about automating deployments and then Heroku came along and changed the whole game. Terraform is trying to solve this but it's still a far cry from "Heroku for infra."

torbital 6y ago

> I want environments to just work. I want code to just compile. I want containers to just run. Without me having to start digging through documentation, looking at thirteen Stack Overflow threads, and cobbling a solution that will inevitably break 3 months from now. Sometimes I conjure up the inspiration to actually implement an idea that's been on my mind. Then I spend half a day trying to setup an environment that just fucking compiles. I'll spend several days kicking at an open door before I actually write any code. In most cases I just give up before this point.

pantelisk 6y ago
If you don't mind me asking. What stack are you working with? And how do you approach things? My experience has been very different (was able to validate ideas and mvps extremely fast, sometimes even in a day/night - but I am very product driven which I think makes the whole difference (meaning that I care to see if the idea works from an end-user perspective first, the rest comes after and only after the project has grown for a while))
dgellow 6y ago
Do you work with C++? Because that’s how it feels to me every time I decide to go with the language
- MaxBarraclough 6y ago
  Are you thinking of the build system specifically?

cjfd 6y ago

I think the biggest factor is in this is that so many different tools are being used. Often tools are brought in to make something easier or solve some problem but nobody seems to be thinking about the huge cost that this entails in terms of keeping all of these moving parts working. I think staying with older and more solid stuff that is not in a rapid flux of so-called improvement is often a much more pleasant experience. I think the barrier for introducing some new kind of tooling should be much higher than it often is in practice.

karl42 6y ago

> I want environments to just work. I want code to just compile. Only mature tools can be complex and still work. But many projects and ecosystems increase complexity much faster than their maturity can set it. I think this loss of balance between maturity and complexity is the root cause of most of the brokenness we experience as developers.

bpicolo 6y ago

Heroku has been the dream all along. I'm very spike on fast-compiling, statically typed monoliths for orgs of most sizes - especially for product-shaped companies versus infrastructure. Once you go distributed, so much becomes somuch harder, and the amount of dev effectiveness focused engineers required to keep the effectiveness high becomes prohibitive until you hit a much larger scale. Logging, debugging, scaling, keeping dependencies up to date, A/B testing and feature flags, and perhaps the biggest one: developer environments.

aiisjustanif 6y ago

Have you heard of BackStage [1]? A few companies have been adopting it to help make teams and departments to help devs. Target is a notable example of going from a really unproductive environment to much more supportive of devs [2]. Including handling building out infrastructure stacks you mentioned. * 1. https://backstage.io/ https://backstage.io/ * 2. https://tech.target.com/ https://tech.target.com/

anthony_barker 6y ago

I find it depressing. People talk of an AI revolution but it is super hard just to setup some baseline infrastructure on these "AI" providers infra.