5 ms·

I look forward to the day when every software developer has half a clue about monitoring, logging, high availability, configuration management, orchestration/sc

by _qc3o 10y ago

I look forward to the day when every software developer has half a clue about monitoring, logging, high availability, configuration management, orchestration/scheduling, performance tuning, build/deployment pipelines, data management/archiving, security and exploit mitigation, etc. The full-stack developer like the serverless/ops-less future is a pipe dream. Most technology organizations barely even know how to build the software to begin with let alone figure out the right way to operate it.

toomuchtodo 10y ago
The startup scene is rife with overconfidence that you have enough waking hours to know everything you listed in one role. Regret I have but one upvote for you.
- hibikir 10y ago
  The issue is that some people can spend just enough hours to make it LOOK like they really understand enough, when in practice, they have wide breadth, but not anywhere enough depth to do the job well. When the startup becomes mid size, suddenly you have a spaghetti codebase with unfortunate tradeoffs, a wide infrastructure full of hard coding that badly attempts to mimic the state of the art of 5 years ago, and one nine of reliability. All of this is of course undocumented, so when you bring in new people, they take forever to learn the existing system, as nothing resembles good practices. As those new people try to clean everything up, they look far less productive than those people that have made this gigantic mess in the first place, and then everyone wonders how they got so lucky with their early, uberproductive employees, which seem to be so much better than anyone else. If the startup has a good economic situation, they might survive while compounding the problem by overhiring (I am sure you all have heard the stories). If the finances aren't quite as good, then the startup folds. I see roads out of this, but they all pass by different early employee compensation: A good early employee is not a good employee in a midsized company, but stock options don't work all that well when early engineers should be leaving the company 3 years in, before exercising the options makes any sense.
  - toomuchtodo 10y ago
    > A good early employee is not a good employee in a midsized company, but stock options don't work all that well when early engineers should be leaving the company 3 years in, before exercising the options makes any sense. Some early startup roles should vest out in 2 years instead of 4, or (what I'd do if I ever worked for a startup again) have a single ratchet provision where, if you decide you don't need my role anymore, the role has changed dramatically, etc, the remainder of my options vest immediately. Life lessons are learned the hard way.
photonwins 10y ago
And blatant disregard for performance analysis & capacity planning. Application is slowing down at 5000 QPS? Let's upgrade to 64 core 128GB Server and while we are at it, let's throw in a bunch of SSDs too. </s>
- JimmyAustin 10y ago
  It's worth it to analyse how much that server costs vs how it costs to rearchitect the application. Those 64 cores are probably cheaper then 2 months of a senior engineer's salary.
  - phamilton 10y ago
    I think the bigger point is that by adding SSDs and increasing CPU/memory all at once, we don't figure out what is actually the bottleneck.
    - aaronbrethorst 10y ago
      In the short term, it is far cheaper to fix everything that might be causing the business to collapse than it is to perform root cause analysis.
      - snovv_crash 10y ago
        The problem is that it means you need to fix the same problem again soon, and finding something faster than SSDs which you can store your 1TB of data on isn't really feasible. On the other hand, if you'd just added an index to the column you are querying on...
        deleted 10y ago
        [deleted]
      - davidgerard 10y ago
        IME this puts the problem off six months (which might be enough!) but the technical debt interest bill comes calling. If your algorithm is fundamentally shitty, you can scale it up by brute force for a certain time, then it outstrips your ability to do so, and you may need to apply actual competence to the problem. If you have any on hand that knows your systems. (I'm a sysadmin. I have full confidence in my job existing for many decades to come. Because even in the future, nothing works.)
sroussey 10y ago
There are plenty of us. But there are more of them. Best of luck differentiating between the two. And I am being sincere.
jsudhams 10y ago
In small setup yes the developer knowing this will help,but in large companies I would assume this is what solution architect and CTO's do. If we do 80/20 heck even 20/80 with Infra experts and app architect and decide on day one the characteristics and limitations of the software to be developed then you don't need every dev to know the infra and every infra guy to know code. In my experience seeign other super support professional it is that knowledge of how app and tech works make it easy to support. But yeah i training people with this kind of knowledge. I would say 1 in 5 is capable but only 1 in 10 or 20 only make it due to time it requires to know all and able to do ad-hoc work. May people hate ad-hoc challenges everyday . But there some some who take it as day to day.
bbcbasic 10y ago
When do they learn this stuff? More weekend reading and side projects? Aka free overtime.
- Rapzid 10y ago
  There is pretty much no Uni track for this stuff so, yeah, on the job and after work. I have no degree so everything after high school(which I was very fortunate to have had C++, CCNA, etc classes at).. FREE though? Man, my current salary doesn't make me feel like I gave my time learning this stuff away for free..
  - Jtsummers 10y ago
    Learning isn't about your current salary. It's to qualify you for your next salary.
    - jonaf 10y ago
      I find this to be just a little bit oversimplified. Indeed, I find that learning on a daily basis keeps me employed, whether I am physically in the office or not, or what time of day it is. So, learning may be about your current salary -- keeping it, that is.
- sigil 10y ago
  Not sure exactly. Tinkering, curiosity, free reign to learn new things at a few lucky jobs, guilt at throwing problems "over the wall" to ops people, and the fact that we're all operators sometimes (we all probably run personal servers and local networks) -- these were all a factor for me. I mostly build, but running things is also fun, and building things with a knowledge of what's downstream is the most fun, imo.
lathiat 10y ago
As someone in the hosting industry, you should try pair the average Wordpress "developer" with the customer who wants 1,000 people to hit the site and place an order at the same time. Or just run a site at all. I'm not sure how these guys survive on your average web host. Our team can (and) do a lot of low level and language level debugging to figure out issues. Most cPanel resellers would send you packing. I can't imagine the frustration the end customers often experience bouncing these issues around for weeks. In other news, I'm clearly highly over qualified!
snovv_crash 10y ago
Not everybody develops applications that are hosted. Who do you think writes your hardware drivers, IDEs, display managers, browsers, command line tools? The list you presented is about as one-sided as saying everybody needs to know how to write shaders or bluetooth drivers - it is niche to the work you do, and just because somebody isn't well versed in it doesn't mean that they don't have a large body of industry-specific knowledge that doesn't even feature on your radar.
- dkarapetyan 10y ago
  You're making my point even better. So how exactly is someone that is making a game going to get by with a serverless/op-less/shader-less/driver-less future? Is there a game developer crisis or a kernel driver developer identity crisis we are not aware of? When put that way the claim sounds even more nonsensical. So before we/anyone claim there is an identity crisis maybe the terms should be defined better since a lot of the terms the author uses are marketing gimmicks. There is a trend toward employing tools and patterns for managing distributed systems that reduces operational burden and consequently requires fewer people because the tools are handling more of what used to be handled by humans. That's great but I wouldn't call that a crisis, identity or otherwise, in fact I'd call that progress.
  - randoTroll 10y ago
    It's often good to look at the background of the authors of these posts.
    - dkarapetyan 10y ago
      I know who she is that's why I expected better.
- dsjoerg 10y ago
  You missed the sarcasm of the comment you are responding to. You two are in violent agreement.
formula1 10y ago
The world isnt guided by people who learn every facet and mechanism of every subject. They are lead by those that see an opportunity and have the humility, audacity and will power to learn just enough or make just enough to make a successful product. They dont care about standards, quality nor best practices. The day when every developer has a hood understanding of distributed systems and devops is when there is a simple way to use it, we start learning programming concepts from s young age or when every developer graduates from MIT and only big companies are allowed to hire people. I personally think its absurd to be on a high horse in the fastest pace ecosystem out there. We constantly need to look for te newest technology and are competing with highly specialized minds (and eventually AI) that can generally do our work 10x better than we can. I mean these last couples months have been a PR disaster for docker which has been seen as a "standard" for years now. Though technology specific aspects is different than abstract concepts, very few flowcharts actual ACT as production code
FLUX-YOU 10y ago
>I look forward to the day when every software developer has half a clue about monitoring, logging, high availability, configuration management, orchestration/scheduling, performance tuning, build/deployment pipelines, data management/archiving, security and exploit mitigation, etc. Just keep piling on job requirements until all developers need 20 years of college to get a junior position. Everyone seems to have written an "X things every developer must know" article.
ownagefool 10y ago
As someones whos worked within both disciplines, I don't actually think it's terribly difficult to learn all those things. The problem is, we're all too busy learning the new shiney without ever really delving deeper down the stack. I find it a bit sad that such smart people jump on the microservices bandwagon without learning how to prove their application is working or deal with the fact that the network will fail.
emmelaich 10y ago
The only way to make that happen is to make them responsible for service. It sharpens the mind when they know that. Unfortunately this takes a culture change that management need to drive. At the moment the typical developer contract might be only 6 months or a year so they are not invested in the proper running of their code.

by _qc3o 10y ago

toomuchtodo 10y ago

The startup scene is rife with overconfidence that you have enough waking hours to know everything you listed in one role. Regret I have but one upvote for you.

hibikir 10y ago
The issue is that some people can spend just enough hours to make it LOOK like they really understand enough, when in practice, they have wide breadth, but not anywhere enough depth to do the job well. When the startup becomes mid size, suddenly you have a spaghetti codebase with unfortunate tradeoffs, a wide infrastructure full of hard coding that badly attempts to mimic the state of the art of 5 years ago, and one nine of reliability. All of this is of course undocumented, so when you bring in new people, they take forever to learn the existing system, as nothing resembles good practices. As those new people try to clean everything up, they look far less productive than those people that have made this gigantic mess in the first place, and then everyone wonders how they got so lucky with their early, uberproductive employees, which seem to be so much better than anyone else. If the startup has a good economic situation, they might survive while compounding the problem by overhiring (I am sure you all have heard the stories). If the finances aren't quite as good, then the startup folds. I see roads out of this, but they all pass by different early employee compensation: A good early employee is not a good employee in a midsized company, but stock options don't work all that well when early engineers should be leaving the company 3 years in, before exercising the options makes any sense.
- toomuchtodo 10y ago
  > A good early employee is not a good employee in a midsized company, but stock options don't work all that well when early engineers should be leaving the company 3 years in, before exercising the options makes any sense. Some early startup roles should vest out in 2 years instead of 4, or (what I'd do if I ever worked for a startup again) have a single ratchet provision where, if you decide you don't need my role anymore, the role has changed dramatically, etc, the remainder of my options vest immediately. Life lessons are learned the hard way.

photonwins 10y ago

And blatant disregard for performance analysis & capacity planning. Application is slowing down at 5000 QPS? Let's upgrade to 64 core 128GB Server and while we are at it, let's throw in a bunch of SSDs too. </s>

JimmyAustin 10y ago
It's worth it to analyse how much that server costs vs how it costs to rearchitect the application. Those 64 cores are probably cheaper then 2 months of a senior engineer's salary.
- phamilton 10y ago
  I think the bigger point is that by adding SSDs and increasing CPU/memory all at once, we don't figure out what is actually the bottleneck.
  - aaronbrethorst 10y ago
    In the short term, it is far cheaper to fix everything that might be causing the business to collapse than it is to perform root cause analysis.
    - snovv_crash 10y ago
      The problem is that it means you need to fix the same problem again soon, and finding something faster than SSDs which you can store your 1TB of data on isn't really feasible. On the other hand, if you'd just added an index to the column you are querying on...
      - deleted 10y ago
        [deleted]
    - davidgerard 10y ago
      IME this puts the problem off six months (which might be enough!) but the technical debt interest bill comes calling. If your algorithm is fundamentally shitty, you can scale it up by brute force for a certain time, then it outstrips your ability to do so, and you may need to apply actual competence to the problem. If you have any on hand that knows your systems. (I'm a sysadmin. I have full confidence in my job existing for many decades to come. Because even in the future, nothing works.)

sroussey 10y ago

There are plenty of us. But there are more of them. Best of luck differentiating between the two. And I am being sincere.

jsudhams 10y ago

In small setup yes the developer knowing this will help,but in large companies I would assume this is what solution architect and CTO's do. If we do 80/20 heck even 20/80 with Infra experts and app architect and decide on day one the characteristics and limitations of the software to be developed then you don't need every dev to know the infra and every infra guy to know code. In my experience seeign other super support professional it is that knowledge of how app and tech works make it easy to support. But yeah i training people with this kind of knowledge. I would say 1 in 5 is capable but only 1 in 10 or 20 only make it due to time it requires to know all and able to do ad-hoc work. May people hate ad-hoc challenges everyday . But there some some who take it as day to day.

bbcbasic 10y ago

When do they learn this stuff? More weekend reading and side projects? Aka free overtime.

Rapzid 10y ago
There is pretty much no Uni track for this stuff so, yeah, on the job and after work. I have no degree so everything after high school(which I was very fortunate to have had C++, CCNA, etc classes at).. FREE though? Man, my current salary doesn't make me feel like I gave my time learning this stuff away for free..
- Jtsummers 10y ago
  Learning isn't about your current salary. It's to qualify you for your next salary.
  - jonaf 10y ago
    I find this to be just a little bit oversimplified. Indeed, I find that learning on a daily basis keeps me employed, whether I am physically in the office or not, or what time of day it is. So, learning may be about your current salary -- keeping it, that is.
sigil 10y ago
Not sure exactly. Tinkering, curiosity, free reign to learn new things at a few lucky jobs, guilt at throwing problems "over the wall" to ops people, and the fact that we're all operators sometimes (we all probably run personal servers and local networks) -- these were all a factor for me. I mostly build, but running things is also fun, and building things with a knowledge of what's downstream is the most fun, imo.

lathiat 10y ago

As someone in the hosting industry, you should try pair the average Wordpress "developer" with the customer who wants 1,000 people to hit the site and place an order at the same time. Or just run a site at all. I'm not sure how these guys survive on your average web host. Our team can (and) do a lot of low level and language level debugging to figure out issues. Most cPanel resellers would send you packing. I can't imagine the frustration the end customers often experience bouncing these issues around for weeks. In other news, I'm clearly highly over qualified!

snovv_crash 10y ago

Not everybody develops applications that are hosted. Who do you think writes your hardware drivers, IDEs, display managers, browsers, command line tools? The list you presented is about as one-sided as saying everybody needs to know how to write shaders or bluetooth drivers - it is niche to the work you do, and just because somebody isn't well versed in it doesn't mean that they don't have a large body of industry-specific knowledge that doesn't even feature on your radar.

dkarapetyan 10y ago
You're making my point even better. So how exactly is someone that is making a game going to get by with a serverless/op-less/shader-less/driver-less future? Is there a game developer crisis or a kernel driver developer identity crisis we are not aware of? When put that way the claim sounds even more nonsensical. So before we/anyone claim there is an identity crisis maybe the terms should be defined better since a lot of the terms the author uses are marketing gimmicks. There is a trend toward employing tools and patterns for managing distributed systems that reduces operational burden and consequently requires fewer people because the tools are handling more of what used to be handled by humans. That's great but I wouldn't call that a crisis, identity or otherwise, in fact I'd call that progress.
- randoTroll 10y ago
  It's often good to look at the background of the authors of these posts.
  - dkarapetyan 10y ago
    I know who she is that's why I expected better.
dsjoerg 10y ago
You missed the sarcasm of the comment you are responding to. You two are in violent agreement.

formula1 10y ago

The world isnt guided by people who learn every facet and mechanism of every subject. They are lead by those that see an opportunity and have the humility, audacity and will power to learn just enough or make just enough to make a successful product. They dont care about standards, quality nor best practices. The day when every developer has a hood understanding of distributed systems and devops is when there is a simple way to use it, we start learning programming concepts from s young age or when every developer graduates from MIT and only big companies are allowed to hire people. I personally think its absurd to be on a high horse in the fastest pace ecosystem out there. We constantly need to look for te newest technology and are competing with highly specialized minds (and eventually AI) that can generally do our work 10x better than we can. I mean these last couples months have been a PR disaster for docker which has been seen as a "standard" for years now. Though technology specific aspects is different than abstract concepts, very few flowcharts actual ACT as production code

FLUX-YOU 10y ago

>I look forward to the day when every software developer has half a clue about monitoring, logging, high availability, configuration management, orchestration/scheduling, performance tuning, build/deployment pipelines, data management/archiving, security and exploit mitigation, etc. Just keep piling on job requirements until all developers need 20 years of college to get a junior position. Everyone seems to have written an "X things every developer must know" article.

ownagefool 10y ago

As someones whos worked within both disciplines, I don't actually think it's terribly difficult to learn all those things. The problem is, we're all too busy learning the new shiney without ever really delving deeper down the stack. I find it a bit sad that such smart people jump on the microservices bandwagon without learning how to prove their application is working or deal with the fact that the network will fail.

emmelaich 10y ago

The only way to make that happen is to make them responsible for service. It sharpens the mind when they know that. Unfortunately this takes a culture change that management need to drive. At the moment the typical developer contract might be only 6 months or a year so they are not invested in the proper running of their code.