3 ms·

I’ve spent hundreds and even thousands of man-hours to undo the lack of planning that developers have inflicted upon their early-stage start-up environments bec

I’ve spent hundreds and even thousands of man-hours to undo the lack of planning that developers have inflicted upon their early-stage start-up environments because they simply haven’t ever managed networks or systems beyond the scale of their laptops. They have no context for setting up VLANs or subnetting beyond their cute 192.168.0.1/24 home networks. Putting everything into huge /8 subnets, extremely loose IAM permissions, overlapping VPC CIDRs, etc. have caused a great, great deal of issues that make growth slowed or even impossible. For example, we’ve had to do a number of DB migrations resulting in some maintenance periods (when our product is sold as a zero-downtime solution as a competitive differentiator) because we had to re-IP our AWS VPCs from the ground up to accommodate multiple regions and accounts. If you read through the best practices of AWS networking, all of this would have been not a big deal. You wouldn’t hire an operations engineer to architect your software, why would you have a software architect setup your network? Software may be tough to change but I can assure everyone that the millions of miles of cabling out there in legacy data centers out there would be much better if someone with sufficient networking backgrounds had gotten involved earlier. Operations is a cost center, but it can be a force multiplier for your revenue centers of less than 1.0 If you don’t have the knowledge and experience to lay out a network and expect to grow the footprint beyond more than a handful of instances, you should probably at least get a 1 hour session over coffee with a moderately experienced engineer that has actually thought about these things before. Heck, I’d do it practically for just beer money because I’d consider it an act of goodwill for any future infrastructure engineer that has to work on the environment.

lstamour 8y ago
I totally agree with the last paragraph but would point out that as a front-end software engineer exposed to all aspects of business-critical systems front-to-back and troubleshooting them alongside folks whose job it was to fix things (but maybe couldn’t...) I’ve been trying since mid-2014 to follow a certain amount of network infrastructure best practices through the PacketPushers podcasts—-what started with trying to learn more about the buzzword of Software Defined Networking led to independent study by doing my own research into things like FD.io, virtual machine networking, and the various networking solutions and benefits that Kubernetes and cloud providers can offer and the important role of usually-centralized control plane abstractions in distributed systems like networking — be it serverless or self-serve, efficient routing, or troubleshooting/monitoring. I’ve found that if you don’t have an experienced cloud-native software engineering resource to learn from, you’ll have to follow many different sources (and source code!) in order to pick up how some of this works. In the end, I’m left both with a greater appreciation for the sysadmin skills required, but also for the disruption new technologies can bring. And I’ll leave looking up the details on my next implementation to, as noted above, looking up best practices in a just-in-time sort of way. (One of my favourite methods to pick up SRE tidbits is to search recent HN comments for advice from the trenches, I really appreciate these mentions and resources...) That said, every time I use terms that aren’t directly related to code or software architecture, I feel imposter syndrome creeping up on me—as in writing much of this post. (I’d welcome comments or additional learning resources!) Also, I’m not sure AWS is the best example, their network terms and best practices seem far more oddly named and unique to AWS than what I’ve heard from Google’s recent conference presentations for enterprise adoption of Google Cloud.
- devonkim 8y ago
  For your specific case, you're already far, far ahead of the curve on my experiences with developers familiar with infrastructure concerns and spreading yourself too thin does nobody any good job or career-wise. I'm in the camp myself - I'm building my depth back and dropping whatever curiosities and novelties show up to get my head on straight. I might argue even that you've gone too far - your value as someone that can code beyond a rudimentary level is much more generalized by default than any network engineer, and context-switching between coding and infrastructure work is almost as bad as switching from coding to meetings. Computers are really fast these days and unless you're typically working with massive scale infrastructures I have trouble seeing an advantage in a developer understanding much more than how to troubleshoot things effectively so that a network engineer can determine what's wrong (being able to identify when packet fragmentation and inappropriate MTU is causing app slow-downs on top of using MTR rather than traceroute will make you a great friend to many network engineers already). Your infrastructure / SRE peers will appreciate you more for writing software that is easier to deploy and maintain when you're clueless about networking than if you understand networks and designed a system that is extremely stateful when it offers no technical advantage to be that way (databases / caches get a pass, everyone else writing software in 2018 has no excuse to keep state on a machine for longer than a business transaction window). High performing software teams deploy often and have the culture to encourage it in a healthy manner - there is no excuse to write software that is deployed weeks or months as huge chunks of changes unless you fall into very niche enterprisey domains and even still you should at least be making production-releasable software daily. And like it or not, AWS is the enterprise cloud by fiat now, so it'll determine the bar and etymology for other vendors to meet and (hopefully) exceed.
fogetti 8y ago
Here is what I see happening today: we have a bunch of hype inducted by cloud companies and their marketing departments around their solutions. Which makes their solutions look desirable and inevitable. Also makes them seem cutting-edge. We also have another bunch of actors in the industry: devops and sysadmins who because their old cushy position was kind of unnecessitated by the emergence of these new services (after all ANYONE can create an AWS account) realized that they have to come up with new smokes and mirrors to rationalize why are their role is so important. I am very very much against this stance and proposition and I will fight this thinking in every possible forum. The possibility is real and the risk/cost is very low to empower dev teams in today's cloud landscape so there is really no need to prevent this happening. And this discussion also reminds me of an old Uncle Bob article [0] where Uncle Bob summarizes the (back then situation) as follows: > I witnessed the rise of a new job function. The DBA! Mere programmers could not be entrusted with the data – so the marketing hype told us. The data is too precious, too fragile, too easily corrupted by those undisciplined louts. We need special people to manage the data. People trained by the database companies. People who would safeguard and promulgate the giant database companies’ marketing message: that the database belongs in the center. The center of the system, the enterprise, the world, the very universe. MUAHAHAHAHAHAHA! Now replace data in the above excerpt with infrastructure and you will arrive to the same conclusion as I did. Q.E.D. [0] https://blog.cleancoder.com/uncle-bob/2012/05/15/NODB.html https://blog.cleancoder.com/uncle-bob/2012/05/15/NODB.html
- devonkim 8y ago
  I view the democratization of infrastructure similar to democracy- the best part of it is that anyone can do it, and the worst part of it is that anyone can do it. On the flipside of specialists getting involved, I also see an awful lot of bad / inappropriate networks and security layouts in cloud environments created by traditional infrastructure engineers because they carried too many principles from managing physical networks. I'm just happy that I shouldn't need to be hassled by anyone to create a random VM for them to test something quick out with such flexible infrastructure. The devops / Agile philosophy of everyone being empowered to do most things works pretty well when people want to do all these things, are invested in the outcome together, and are at least vaguely competent in their tasking. However, the approach has limitations when it comes to tasks that nobody wants or can do but is still important to the business. It's even worse when something is important but nobody even knows it because of groupthink blindness. I don't think I'm being hypocritical in recommending specialists for topics I know something about while advocating for empowerment in other functions because if my previous employers / clients knew what they were doing with infrastructure and healthy software development practices, they could have grown much more before needing to hire someone to do it full-time for them. I really don't want to have to re-IP another awful network again and have to tell leadership that you have to incur downtime to do it because their software can't handle database hiccups like when failing over to a hot replica. It is boring, unfulfilling, stressful work to me that - even worse - offers no tangible business value when done well but when done inappropriately is an albatross. Compute infrastructure across different industries is in an overall state of health where everyone loves junk food but is starting to recognize its harm, some vaccines have been developed for the flu but nobody gets it or the vaccine costs $20k per shot for some people, doctors for Hollywood actors and pro athletes debate publicly over which lifting program is more optimal, and the two fitness trends are competitive decathlons and walking from their car to their desk instead of taking a Bird. In comparison, software is much further along with at least a vague sense of a board of medicine in different states (that is determined through a pageant and feats of strength, not experience in Mississippi), people are taught about the dangers of junk food, there is a debate on GMOs (Uncle Bob is strictly against it, I see some positives although DBs are much more controversial now than the well-researched topic of GMOs), and only the literally crazy people don't believe in use of vaccines. You and I both agree that tons of people needlessly hire a personal trainer when the information to start exercising is out there and basically free now. What I think you're suggesting is to "just start jogging and it'll work out - everyone can run without a trainer helping you" but I think it's mistaken not because I think trainers are required. Right now most cloud providers don't give you shoes for free because they want to sell you Air Jordans or hiking shoes to recuperate their substantial investments, the roads are totally unpaved except for paths through lemonade stands charged by how fast you run, and I've seen a lot of people hit by cars while running because they kept stopping to pick glass out of their feet all because the common theme is they started running with socks and they were "forced" to keep running. I don't think I'm being unreasonable in saying that by default people start walking with socks on because they think personal trainers are too much when flip flops can work really well until you need to start running. By your view every other former sysadmin is now a personal trainer trying to get people into some shoes when we can do fine without one, and while I can see that I'm personally not the typical sysadmin type because I started off as a developer only caring about running fast and have learned starting off on the wrong foot can cause serious problems that can be very cheaply and easily corrected. Perhaps we are disagreeing over how much those flip flops cost or how difficult bad footwear is to discard?

lstamour 8y ago

I totally agree with the last paragraph but would point out that as a front-end software engineer exposed to all aspects of business-critical systems front-to-back and troubleshooting them alongside folks whose job it was to fix things (but maybe couldn’t...) I’ve been trying since mid-2014 to follow a certain amount of network infrastructure best practices through the PacketPushers podcasts—-what started with trying to learn more about the buzzword of Software Defined Networking led to independent study by doing my own research into things like FD.io, virtual machine networking, and the various networking solutions and benefits that Kubernetes and cloud providers can offer and the important role of usually-centralized control plane abstractions in distributed systems like networking — be it serverless or self-serve, efficient routing, or troubleshooting/monitoring. I’ve found that if you don’t have an experienced cloud-native software engineering resource to learn from, you’ll have to follow many different sources (and source code!) in order to pick up how some of this works. In the end, I’m left both with a greater appreciation for the sysadmin skills required, but also for the disruption new technologies can bring. And I’ll leave looking up the details on my next implementation to, as noted above, looking up best practices in a just-in-time sort of way. (One of my favourite methods to pick up SRE tidbits is to search recent HN comments for advice from the trenches, I really appreciate these mentions and resources...) That said, every time I use terms that aren’t directly related to code or software architecture, I feel imposter syndrome creeping up on me—as in writing much of this post. (I’d welcome comments or additional learning resources!) Also, I’m not sure AWS is the best example, their network terms and best practices seem far more oddly named and unique to AWS than what I’ve heard from Google’s recent conference presentations for enterprise adoption of Google Cloud.

devonkim 8y ago
For your specific case, you're already far, far ahead of the curve on my experiences with developers familiar with infrastructure concerns and spreading yourself too thin does nobody any good job or career-wise. I'm in the camp myself - I'm building my depth back and dropping whatever curiosities and novelties show up to get my head on straight. I might argue even that you've gone too far - your value as someone that can code beyond a rudimentary level is much more generalized by default than any network engineer, and context-switching between coding and infrastructure work is almost as bad as switching from coding to meetings. Computers are really fast these days and unless you're typically working with massive scale infrastructures I have trouble seeing an advantage in a developer understanding much more than how to troubleshoot things effectively so that a network engineer can determine what's wrong (being able to identify when packet fragmentation and inappropriate MTU is causing app slow-downs on top of using MTR rather than traceroute will make you a great friend to many network engineers already). Your infrastructure / SRE peers will appreciate you more for writing software that is easier to deploy and maintain when you're clueless about networking than if you understand networks and designed a system that is extremely stateful when it offers no technical advantage to be that way (databases / caches get a pass, everyone else writing software in 2018 has no excuse to keep state on a machine for longer than a business transaction window). High performing software teams deploy often and have the culture to encourage it in a healthy manner - there is no excuse to write software that is deployed weeks or months as huge chunks of changes unless you fall into very niche enterprisey domains and even still you should at least be making production-releasable software daily. And like it or not, AWS is the enterprise cloud by fiat now, so it'll determine the bar and etymology for other vendors to meet and (hopefully) exceed.

fogetti 8y ago

Here is what I see happening today: we have a bunch of hype inducted by cloud companies and their marketing departments around their solutions. Which makes their solutions look desirable and inevitable. Also makes them seem cutting-edge. We also have another bunch of actors in the industry: devops and sysadmins who because their old cushy position was kind of unnecessitated by the emergence of these new services (after all ANYONE can create an AWS account) realized that they have to come up with new smokes and mirrors to rationalize why are their role is so important. I am very very much against this stance and proposition and I will fight this thinking in every possible forum. The possibility is real and the risk/cost is very low to empower dev teams in today's cloud landscape so there is really no need to prevent this happening. And this discussion also reminds me of an old Uncle Bob article [0] where Uncle Bob summarizes the (back then situation) as follows: > I witnessed the rise of a new job function. The DBA! Mere programmers could not be entrusted with the data – so the marketing hype told us. The data is too precious, too fragile, too easily corrupted by those undisciplined louts. We need special people to manage the data. People trained by the database companies. People who would safeguard and promulgate the giant database companies’ marketing message: that the database belongs in the center. The center of the system, the enterprise, the world, the very universe. MUAHAHAHAHAHAHA! Now replace data in the above excerpt with infrastructure and you will arrive to the same conclusion as I did. Q.E.D. [0] https://blog.cleancoder.com/uncle-bob/2012/05/15/NODB.html https://blog.cleancoder.com/uncle-bob/2012/05/15/NODB.html

devonkim 8y ago
I view the democratization of infrastructure similar to democracy- the best part of it is that anyone can do it, and the worst part of it is that anyone can do it. On the flipside of specialists getting involved, I also see an awful lot of bad / inappropriate networks and security layouts in cloud environments created by traditional infrastructure engineers because they carried too many principles from managing physical networks. I'm just happy that I shouldn't need to be hassled by anyone to create a random VM for them to test something quick out with such flexible infrastructure. The devops / Agile philosophy of everyone being empowered to do most things works pretty well when people want to do all these things, are invested in the outcome together, and are at least vaguely competent in their tasking. However, the approach has limitations when it comes to tasks that nobody wants or can do but is still important to the business. It's even worse when something is important but nobody even knows it because of groupthink blindness. I don't think I'm being hypocritical in recommending specialists for topics I know something about while advocating for empowerment in other functions because if my previous employers / clients knew what they were doing with infrastructure and healthy software development practices, they could have grown much more before needing to hire someone to do it full-time for them. I really don't want to have to re-IP another awful network again and have to tell leadership that you have to incur downtime to do it because their software can't handle database hiccups like when failing over to a hot replica. It is boring, unfulfilling, stressful work to me that - even worse - offers no tangible business value when done well but when done inappropriately is an albatross. Compute infrastructure across different industries is in an overall state of health where everyone loves junk food but is starting to recognize its harm, some vaccines have been developed for the flu but nobody gets it or the vaccine costs $20k per shot for some people, doctors for Hollywood actors and pro athletes debate publicly over which lifting program is more optimal, and the two fitness trends are competitive decathlons and walking from their car to their desk instead of taking a Bird. In comparison, software is much further along with at least a vague sense of a board of medicine in different states (that is determined through a pageant and feats of strength, not experience in Mississippi), people are taught about the dangers of junk food, there is a debate on GMOs (Uncle Bob is strictly against it, I see some positives although DBs are much more controversial now than the well-researched topic of GMOs), and only the literally crazy people don't believe in use of vaccines. You and I both agree that tons of people needlessly hire a personal trainer when the information to start exercising is out there and basically free now. What I think you're suggesting is to "just start jogging and it'll work out - everyone can run without a trainer helping you" but I think it's mistaken not because I think trainers are required. Right now most cloud providers don't give you shoes for free because they want to sell you Air Jordans or hiking shoes to recuperate their substantial investments, the roads are totally unpaved except for paths through lemonade stands charged by how fast you run, and I've seen a lot of people hit by cars while running because they kept stopping to pick glass out of their feet all because the common theme is they started running with socks and they were "forced" to keep running. I don't think I'm being unreasonable in saying that by default people start walking with socks on because they think personal trainers are too much when flip flops can work really well until you need to start running. By your view every other former sysadmin is now a personal trainer trying to get people into some shoes when we can do fine without one, and while I can see that I'm personally not the typical sysadmin type because I started off as a developer only caring about running fast and have learned starting off on the wrong foot can cause serious problems that can be very cheaply and easily corrected. Perhaps we are disagreeing over how much those flip flops cost or how difficult bad footwear is to discard?