9 ms·

As a former linux expert and sysadmin turned developer, I think this attitude is what pissed me off most about current developers. I thought I'd have a huge up

As a former linux expert and sysadmin turned developer, I think this attitude is what pissed me off most about current developers. I thought I'd have a huge upper hand, understanding dark magic. Come to find out most developers don't know and don't care about any of it, and have infected the whole community with the idea that none of it matters and servers and processes are just cattle. /rant

anonymoushn 5y ago
Devs don't generally have the time to take good care of these things while meeting the goals of the business, especially because they probably work at a company with 0 sysadmins.
cuddlecake 5y ago
As a current developer, what pisses me off about anyone is sweeping statements like this :(
- silisili 5y ago
  I'm also a current developer. This isn't a slight against developers, but the prevailing mindset. And that mindset is little more than 'move fast and keep rebooting things'
  - zerocount 5y ago
    Yep, the quickest solution with the least amount of effort, unfortunately.
    - jeltz 5y ago
      But the amount of technical debt created by always using the leadt effort solution can be huge.
- jiveturkey 5y ago
  Please, where do you work such that you are afforded adequate time to debug deep OS level issues? That it's acceptable to code to handle the 1-10^12 type error? I want to work there too!
lillecarl 5y ago
I mean, while we shouldn't leak but the systems should be able to deal with it right?
- silisili 5y ago
  Different mindsets. I was cutting my teeth on a set of 12 static servers/processes, with no autoscaling or auto reboots. Memory leaks were rare but reported. Usually FD leaks were culprits. So learned a ton about how to write really correct applications, and today it just feels like none of that matters. And in complete fairness, with today's ops tools, it probably doesn't.
  - Arainach 5y ago
    At scale, server failures are going to happen. Maybe it's a power outage. Maybe it's a failed drive. Maybe it's a kernel crash. Only one of those can be remedied (not even prevented) with careful debugger investigation, so you're going to have to do the work to be able to handle such failures gracefully. At that point, unless the leaks are causing outages so rapidly that the machines can't handle, then rebooting becomes not that big of a deal, while wasting senior eng days trying to track them down is a real cost.
    - hvgk 5y ago
      There are some nasty escalation scenarios I’ve seen in that space. Cost increases as does workload backlogs and latency. Eventually something breaks and causes an outage. All leaks should be resolved lest you end up with floods.
- numpad0 5y ago
  So long as it isn’t running anything realtime and process restart cost isn’t a problem (realtime == consistent run time, predictable behavior, != “fast”)
dmitriid 5y ago
> As a former linux expert and sysadmin turned developer, I think this attitude is what pissed me off most about current developers Where by "current" you mean "always has been" (I remember this being a solution to early Rails problems, for example)
- samhw 5y ago
  He's getting it from the line of code in everyone's head that goes `let imageOfWorld = exaggeratedAntithesis(self.ideals)`
onion2k 5y ago
"My app has a memory leak. Please can I have the engineering time to debug an operating system." wouldn't fly at any company. It's not that developers don't care; it's that developers don't have any choice.
- ajuc 5y ago
  I'm pretty sure I did this several times and I worked in a company of 15 people where our whole IT department was me and the other guy. Depends on the domain. For example we had our devices for impaired vision people stop responding to touch events on our cheap OEM touchscreens when German locale was enabled. Turns out linux driver used locale-specific code to read calibration configuration so each event position was scaled by orders of magnitude :) The funny thing is that then they fixed that in the driver and if you had commas in the config file as a workaround - you get a kernel panic :)
  - onion2k 5y ago
    That's a problem where you had no real choice to do anything else by the sounds of it though. You couldn't just "turn it off and on again" to fix it. In the case of a server where you can just have health check that reboots the server if memory gets a bit too low, it's far simpler just to automate a power cycle. It's not laziness or incompetence; it's understanding where to spend your effort best.
    - ajuc 5y ago
      If we had memory leaks caused by drivers we would have to debug them all the way as well, restarting the device every few hours wasn't an option.
      - onion2k 5y ago
        Exactly. We all choose the most cost effective solution - it's a balance between engineering effort and how effective the fix is. In the case of a touch screen, rebootingit every few hours isn't as effective as debugging the underlying cause. In the case of a web server having a memory leak, killing a pod and bringing up a new one automatically when it gets unhealthy definitely is. It's not surprising that each dev team settles on different solutions to each problem.
deleted 5y ago
[deleted]
hvgk 5y ago
That’s about right. I’ve got a dev who’s container just exits 0 ten times a day. He told me kubernetes takes care of it so it’s not a priority.
_def 5y ago
Of course I can't speak about your experiences, but I think the reason for this are mostly business goals. At least I have the impression that lots of developers care for technical excellence, but often that is not desirable from a business perspective. That said, I really wish this wasn't the case. I miss the days when efficient software really mattered (just look at modern tv firmwares, or even mobile stuff).
- silisili 5y ago
  You're probably right. I still remember coming in to 'take over' a new product one (crazy) guy had written. It had no safe guards at all - no queueing, kept everything in memory, no retries or really error handling other than printlines. I remember pointing out 'what if x happens', where x is a really realistic scenario, and the author(and VP) kept saying 'oh thats a caviar problem.' When I asked what that meant, they said "by the time we have to worry about that scenario, we'll be so rich we'll all be eating caviar." Apparently retrying network requests or getting say, 50 requests at a time would make us all millionaires or something. Predictably, the thing fell over multiple times per day until we trashed it and started from scratch...and I still have no idea what caviar tastes like.
marcan_42 5y ago
They don't care, until a kernel panic causes disk corruption in just the right way to cause a cross-security-boundary information leak that also ends up breaking an entire global pipeline, or a rare buffer overread in a ubiquitous piece of code causes once in a petabyte mismatches in cross-checked processes with compliance implications. Source: xoogler, developed a reputation for black magic debugging, debugged both of those things. Correctness still matters, no matter how big you are or how much you think auto-restarts help.
- jasmas 5y ago
  Nah they still don't care
  - nisegami 5y ago
    Never stay at one place long enough for these things to come up /s
  - bluGill 5y ago
    That depends on how noticed it gets. My company makes embedded systems that in some failure cases can kill - we have a history of losing large lawsuits over the base 70 years (before computers of course). As such we have a powerful safety review system in place because million dollar lawsuits make CEOs care - it isn't perfect but there are a lot of things that we don't do that our customers would love.
- discardable_dan 5y ago
  Yes, they really don't care unless it results in a site going down. An industrial giant is happy to throw infinite machines at a problem until it is resolved. The issue isn't in their behavior, it's in Linux's implementation being so poor that they have to. But what are the alternatives? Should Google/Facebook/Amazon throw infinite developer time resolving kernel issues? Should they walk away from Linux?
  - mhoad 5y ago
    I mean one of them at least went ahead and wrote (technically is in the process of writing) an entire operating system from scratch in response. I say this as someone who uses and has used linux as a daily driver for years now but honestly I don’t feel good about the future of the platform in the medium term at all. If anyone has some time to throw at understanding why this is probably the single most sober and rational set of arguments I’ve found of the topic yet https://m.youtube.com/watch?v=WtJ9T_IJOPE https://m.youtube.com/watch?v=WtJ9T_IJOPE
einpoklum 5y ago
> Come to find out most developers don't know and don't care about any of it Can you better define "it"? > and have infected the whole community with the idea that none of it matters and servers and processes are just cattle. Perhaps if by "community" you mean the "run everything on VMs on the cloud" people...
dexen 5y ago
>"won't fix, we just reboot servers when they crash" >understanding dark magic That is the original Unix black magic: KISS and do the simplest possible thing that works and when in doubt use brute force and so forth. That is the "C" language of infrastructure. In the particular in the scenario we're discussing, there's always a pool of several servers ready to handle a HTTP request (or DB query, or multimedia transformation, etc.). Rebooting a small % of this pool, rather than debugging every single memory leak is a pragmatic, cost-effective solution. "Zero bugs" is a very abstract and detached goal; "cost effectiveness" is a very realistic and wholesome goal.
solididiot 5y ago
You have only this much bandwidth between business logic, the last trendy framework/language thrust upon you, the billion overengineering "good practices" adding clutter to your code base and the zillion intricacies of the latest kubernetes platform that "makes your life easier". Most devs have way more than they can work with a single human mind on their plates. Accusing them of missing out on the low level kernel stuff that is about 5 or 6 levels of indirection deeper than the stuff they work day in and day out is like accusing pilots of insufficient knowledge of organic chemistry to understand the jet fuel composition.
elesbao 5y ago
They have not infected, that's not fair. they just don't care and shouldn't be blamed just because you care too much. Good that you and others care and contribute to improve tools and libraries.
ninkendo 5y ago
I feel like the overall direction of the software world has been aiming to maximize developer productivity -- at the expense of rock solid reliability -- for quite some time now. You can think of garbage collection as being an early admittance that it's far simpler to just clean things up in bulk on a periodic basis, than it is to keep track of every resource you're allocating, and making sure that it's freed at the earliest possible time. Restarting a service after it's been running for hours, because of memory leaks, seems like a broader implementation of a similar notion... despite best efforts, sometimes services just accumulate cruft, and so long as you can properly drain clients to another server before restarting, is it really that bad to have to restart it periodically? I think of it as an extra long GC pause that frees everything up. There's a sweet spot where "good enough" really is good enough, and I don't lament it all that much.
skyde 5y ago
it’s a good idea to assume server will fail (power supply, hard drive, network card) because we you use 3k of them they do. But it’s a not a good excuse to be wasteful! which is why I like that Rust and Go are getting more popular

anonymoushn 5y ago

Devs don't generally have the time to take good care of these things while meeting the goals of the business, especially because they probably work at a company with 0 sysadmins.

cuddlecake 5y ago

As a current developer, what pisses me off about anyone is sweeping statements like this :(

silisili 5y ago
I'm also a current developer. This isn't a slight against developers, but the prevailing mindset. And that mindset is little more than 'move fast and keep rebooting things'
- zerocount 5y ago
  Yep, the quickest solution with the least amount of effort, unfortunately.
  - jeltz 5y ago
    But the amount of technical debt created by always using the leadt effort solution can be huge.
jiveturkey 5y ago
Please, where do you work such that you are afforded adequate time to debug deep OS level issues? That it's acceptable to code to handle the 1-10^12 type error? I want to work there too!

lillecarl 5y ago

I mean, while we shouldn't leak but the systems should be able to deal with it right?

silisili 5y ago
Different mindsets. I was cutting my teeth on a set of 12 static servers/processes, with no autoscaling or auto reboots. Memory leaks were rare but reported. Usually FD leaks were culprits. So learned a ton about how to write really correct applications, and today it just feels like none of that matters. And in complete fairness, with today's ops tools, it probably doesn't.
- Arainach 5y ago
  At scale, server failures are going to happen. Maybe it's a power outage. Maybe it's a failed drive. Maybe it's a kernel crash. Only one of those can be remedied (not even prevented) with careful debugger investigation, so you're going to have to do the work to be able to handle such failures gracefully. At that point, unless the leaks are causing outages so rapidly that the machines can't handle, then rebooting becomes not that big of a deal, while wasting senior eng days trying to track them down is a real cost.
  - hvgk 5y ago
    There are some nasty escalation scenarios I’ve seen in that space. Cost increases as does workload backlogs and latency. Eventually something breaks and causes an outage. All leaks should be resolved lest you end up with floods.
numpad0 5y ago
So long as it isn’t running anything realtime and process restart cost isn’t a problem (realtime == consistent run time, predictable behavior, != “fast”)

dmitriid 5y ago

> As a former linux expert and sysadmin turned developer, I think this attitude is what pissed me off most about current developers Where by "current" you mean "always has been" (I remember this being a solution to early Rails problems, for example)

samhw 5y ago
He's getting it from the line of code in everyone's head that goes `let imageOfWorld = exaggeratedAntithesis(self.ideals)`

onion2k 5y ago

"My app has a memory leak. Please can I have the engineering time to debug an operating system." wouldn't fly at any company. It's not that developers don't care; it's that developers don't have any choice.

ajuc 5y ago
I'm pretty sure I did this several times and I worked in a company of 15 people where our whole IT department was me and the other guy. Depends on the domain. For example we had our devices for impaired vision people stop responding to touch events on our cheap OEM touchscreens when German locale was enabled. Turns out linux driver used locale-specific code to read calibration configuration so each event position was scaled by orders of magnitude :) The funny thing is that then they fixed that in the driver and if you had commas in the config file as a workaround - you get a kernel panic :)
- onion2k 5y ago
  That's a problem where you had no real choice to do anything else by the sounds of it though. You couldn't just "turn it off and on again" to fix it. In the case of a server where you can just have health check that reboots the server if memory gets a bit too low, it's far simpler just to automate a power cycle. It's not laziness or incompetence; it's understanding where to spend your effort best.
  - ajuc 5y ago
    If we had memory leaks caused by drivers we would have to debug them all the way as well, restarting the device every few hours wasn't an option.
    - onion2k 5y ago
      Exactly. We all choose the most cost effective solution - it's a balance between engineering effort and how effective the fix is. In the case of a touch screen, rebootingit every few hours isn't as effective as debugging the underlying cause. In the case of a web server having a memory leak, killing a pod and bringing up a new one automatically when it gets unhealthy definitely is. It's not surprising that each dev team settles on different solutions to each problem.

deleted 5y ago

[deleted]

hvgk 5y ago

That’s about right. I’ve got a dev who’s container just exits 0 ten times a day. He told me kubernetes takes care of it so it’s not a priority.

_def 5y ago

Of course I can't speak about your experiences, but I think the reason for this are mostly business goals. At least I have the impression that lots of developers care for technical excellence, but often that is not desirable from a business perspective. That said, I really wish this wasn't the case. I miss the days when efficient software really mattered (just look at modern tv firmwares, or even mobile stuff).

silisili 5y ago
You're probably right. I still remember coming in to 'take over' a new product one (crazy) guy had written. It had no safe guards at all - no queueing, kept everything in memory, no retries or really error handling other than printlines. I remember pointing out 'what if x happens', where x is a really realistic scenario, and the author(and VP) kept saying 'oh thats a caviar problem.' When I asked what that meant, they said "by the time we have to worry about that scenario, we'll be so rich we'll all be eating caviar." Apparently retrying network requests or getting say, 50 requests at a time would make us all millionaires or something. Predictably, the thing fell over multiple times per day until we trashed it and started from scratch...and I still have no idea what caviar tastes like.

marcan_42 5y ago

They don't care, until a kernel panic causes disk corruption in just the right way to cause a cross-security-boundary information leak that also ends up breaking an entire global pipeline, or a rare buffer overread in a ubiquitous piece of code causes once in a petabyte mismatches in cross-checked processes with compliance implications. Source: xoogler, developed a reputation for black magic debugging, debugged both of those things. Correctness still matters, no matter how big you are or how much you think auto-restarts help.

jasmas 5y ago
Nah they still don't care
- nisegami 5y ago
  Never stay at one place long enough for these things to come up /s
- bluGill 5y ago
  That depends on how noticed it gets. My company makes embedded systems that in some failure cases can kill - we have a history of losing large lawsuits over the base 70 years (before computers of course). As such we have a powerful safety review system in place because million dollar lawsuits make CEOs care - it isn't perfect but there are a lot of things that we don't do that our customers would love.
discardable_dan 5y ago
Yes, they really don't care unless it results in a site going down. An industrial giant is happy to throw infinite machines at a problem until it is resolved. The issue isn't in their behavior, it's in Linux's implementation being so poor that they have to. But what are the alternatives? Should Google/Facebook/Amazon throw infinite developer time resolving kernel issues? Should they walk away from Linux?
- mhoad 5y ago
  I mean one of them at least went ahead and wrote (technically is in the process of writing) an entire operating system from scratch in response. I say this as someone who uses and has used linux as a daily driver for years now but honestly I don’t feel good about the future of the platform in the medium term at all. If anyone has some time to throw at understanding why this is probably the single most sober and rational set of arguments I’ve found of the topic yet https://m.youtube.com/watch?v=WtJ9T_IJOPE https://m.youtube.com/watch?v=WtJ9T_IJOPE

einpoklum 5y ago

> Come to find out most developers don't know and don't care about any of it Can you better define "it"? > and have infected the whole community with the idea that none of it matters and servers and processes are just cattle. Perhaps if by "community" you mean the "run everything on VMs on the cloud" people...

dexen 5y ago

>"won't fix, we just reboot servers when they crash" >understanding dark magic That is the original Unix black magic: KISS and do the simplest possible thing that works and when in doubt use brute force and so forth. That is the "C" language of infrastructure. In the particular in the scenario we're discussing, there's always a pool of several servers ready to handle a HTTP request (or DB query, or multimedia transformation, etc.). Rebooting a small % of this pool, rather than debugging every single memory leak is a pragmatic, cost-effective solution. "Zero bugs" is a very abstract and detached goal; "cost effectiveness" is a very realistic and wholesome goal.

solididiot 5y ago

You have only this much bandwidth between business logic, the last trendy framework/language thrust upon you, the billion overengineering "good practices" adding clutter to your code base and the zillion intricacies of the latest kubernetes platform that "makes your life easier". Most devs have way more than they can work with a single human mind on their plates. Accusing them of missing out on the low level kernel stuff that is about 5 or 6 levels of indirection deeper than the stuff they work day in and day out is like accusing pilots of insufficient knowledge of organic chemistry to understand the jet fuel composition.

elesbao 5y ago

They have not infected, that's not fair. they just don't care and shouldn't be blamed just because you care too much. Good that you and others care and contribute to improve tools and libraries.

ninkendo 5y ago

I feel like the overall direction of the software world has been aiming to maximize developer productivity -- at the expense of rock solid reliability -- for quite some time now. You can think of garbage collection as being an early admittance that it's far simpler to just clean things up in bulk on a periodic basis, than it is to keep track of every resource you're allocating, and making sure that it's freed at the earliest possible time. Restarting a service after it's been running for hours, because of memory leaks, seems like a broader implementation of a similar notion... despite best efforts, sometimes services just accumulate cruft, and so long as you can properly drain clients to another server before restarting, is it really that bad to have to restart it periodically? I think of it as an extra long GC pause that frees everything up. There's a sweet spot where "good enough" really is good enough, and I don't lament it all that much.

skyde 5y ago

it’s a good idea to assume server will fail (power supply, hard drive, network card) because we you use 3k of them they do. But it’s a not a good excuse to be wasteful! which is why I like that Rust and Go are getting more popular