5 ms·

"My app has a memory leak. Please can I have the engineering time to debug an operating system." wouldn't fly at any company. It's not that developers don't car

"My app has a memory leak. Please can I have the engineering time to debug an operating system." wouldn't fly at any company. It's not that developers don't care; it's that developers don't have any choice.

ajuc 5y ago
I'm pretty sure I did this several times and I worked in a company of 15 people where our whole IT department was me and the other guy. Depends on the domain. For example we had our devices for impaired vision people stop responding to touch events on our cheap OEM touchscreens when German locale was enabled. Turns out linux driver used locale-specific code to read calibration configuration so each event position was scaled by orders of magnitude :) The funny thing is that then they fixed that in the driver and if you had commas in the config file as a workaround - you get a kernel panic :)
- onion2k 5y ago
  That's a problem where you had no real choice to do anything else by the sounds of it though. You couldn't just "turn it off and on again" to fix it. In the case of a server where you can just have health check that reboots the server if memory gets a bit too low, it's far simpler just to automate a power cycle. It's not laziness or incompetence; it's understanding where to spend your effort best.
  - ajuc 5y ago
    If we had memory leaks caused by drivers we would have to debug them all the way as well, restarting the device every few hours wasn't an option.
    - onion2k 5y ago
      Exactly. We all choose the most cost effective solution - it's a balance between engineering effort and how effective the fix is. In the case of a touch screen, rebootingit every few hours isn't as effective as debugging the underlying cause. In the case of a web server having a memory leak, killing a pod and bringing up a new one automatically when it gets unhealthy definitely is. It's not surprising that each dev team settles on different solutions to each problem.
      - ajuc 5y ago
        It still feels dirty though.
  - xorcist 5y ago
    It's not that simple. Intermittently rebooting touchscreens make for a bad user experience. User experience is one of those things that doesn't matter until it suddently does, often when a competitor sells a product that doesn't reboot in the middle of a user operation.
    - deleted 5y ago
      [deleted]
- unwind 5y ago
  Wow, what an interesting bug! Was it a case of parsing "1.000" vs "1,000" or something like that?
  - ajuc 5y ago
    Yup, the matrix for transforming hardware coordinates into software coordinates was saved in a config file using locale-specific number format. When user changed locale it remained in the previous format and if the change was English->German the matrix elements suddenly changed from 1.234 to 1234 :) the details are here: https://github.com/jiixyj/libinput/commit/98793f6b435abf8c17b84102de1d7c59ba8e3f54 https://github.com/jiixyj/libinput/commit/98793f6b435abf8c17... The funny thing is we had the issue once - reported it and added a workaround (added special handling for these config files that used locale-specific calibration tables), then a year later we updated the linux version and had the issue the opposite way till we removed our workaround :) But it was even more fun cause it bricked the devices (kernel panic on libinput startup = restart loop forever) :)
herbst 5y ago
The last company I worked at had several people employed that had a max 20% time for open source rule. To do more or less exactly what you just said. If all companies would don't care, open source wouldn't be there where it is.
- onion2k 5y ago
  There's a huge difference between giving engineers 20% time to work on open source code, and choosing to fix a memory leak by debugging the OS rather than using a health monitor to automatically restart an app. For a start, there's no guarantee that the devs will succeed. It might be a huge waste of time. That's a massive risk.
  - xorcist 5y ago
    "Debugging the OS" sounds not just a little bit pretentious. Perhaps call it "troubleshooting" instead, because that's what's being done. That the problem was found in a driver doesn't make it any different from faulty shell script or a faulty hard disk.
  - ajuc 5y ago
    > That's a massive risk. Leaving a leak in your code base is a risk too. What if it scales with something outside of your control and suddenly you have to restart every 15 seconds?
    - onion2k 5y ago
      Add more pods. Its literally the answer to every devOps problem. (This is a joke btw.) (... or is it?)
      - 123pie123 5y ago
        for some.. sadly not a joke
  - herbst 5y ago
    Sure there is, however our company and debug culture also did not take restarts as a solution for memory leaks either. Sometimes quality matters more than money
bamboozled 5y ago
At some point we have to ask ourselves, who does care? Especially if you're running open source. Yes there are sponsored open source developers, but do the people who don't care even submit a bug report more than half the time?
- hvgk 5y ago
  I did always submit bug reports and patches. Most of the time some GitHub robot closes my ticket or PR after it is ignored for a month. That’s the new status quo: keep your ticket queue short by auto closing it.
  - bamboozled 5y ago
    This made me laugh a lot.
  - jeltz 5y ago
    Yes I noticed recently that this is very common in the Kubernetes community whicj is very annoying.
detaro 5y ago
Very much depends on the domain. Not for a generic server-/web-app normally, but in other software fields not surprising. But of course "current developers" only work on webapps ;)
- oblio 5y ago
  > But of course "current developers" only work on webapps ;) The rule of thumb is that the vast, overwhelming majority of developers are working on business apps, especially Line Of Business (LOB) internal apps nobody ever sees. It's hard to get exact numbers for this stuff but I wouldn't be surprised if the ratio of base infrastructure developers (kernel, shell, drivers, compilers, runtimes, frameworks, libraries etc.) to app developers is 1:100.
  - detaro 5y ago
    Not sure if a contrast between "base infrastructure" and "apps" in response to "only on webapps" is supposed to be a joke or not. Just because its not a webapp it doesn't mean it's "base infrastructure".
    - oblio 5y ago
      I said apps, not just webapps. I'm not sure I see your point in this context.
Aissen 5y ago
That's either that, or pay the most expensive support contract from your OS vendor. You can also ignore it, but you'll end up running into a wall at some point… hopefully after you left the company ?
ckdarby 5y ago
I'm only 12 years in the industry and that is young compared to some others here but I'll give my perspective. I would honestly say that all developers outside of companies who activity are going to run out of cash in 6 months have the time and are given the choice. When management says, we don't have the time the reality is the proposed time given goes beyond a reasonable allocation of time, the max capacity given and or managers bullshit radar amount. Great developers also are able to factor these tasks into their own ticket time to make the time for the issue. I've been in countless meetings where individuals will complain about the state of X, how this is a hack, or we should really fix that. Plenty of times I've seen, "You've got two days of dedicated time to identify, solve and implement." This is met with various results, individuals taking the time, don't really do any work and consider it free time. They spend two days just trying to understand even how all the components fit together. Even if you revisit in a month with another two days they'll spend the same two days relearning the same information because they did not retain or make notes to be able to pickup where they left off. On a rare occasion someone will come in and fix it or move the needle further for the next person. All of these issues tend to funnel into these types of developers and from my sampling they're ~1 per 100 developers at a company.
saagarjha 5y ago
I literally do this all the time. My job is to make the software I work on perform well, and sometimes that means debugging the operating system and figuring out why it’s not doing that. Assuming I bring results by doing this (and I do!) nobody is going to tell me to stop: they honestly could not care less with how I went about my job.
matheusmoreira 5y ago
You'd think professional engineers would have enough autonomy to make decisions like these...

ajuc 5y ago

I'm pretty sure I did this several times and I worked in a company of 15 people where our whole IT department was me and the other guy. Depends on the domain. For example we had our devices for impaired vision people stop responding to touch events on our cheap OEM touchscreens when German locale was enabled. Turns out linux driver used locale-specific code to read calibration configuration so each event position was scaled by orders of magnitude :) The funny thing is that then they fixed that in the driver and if you had commas in the config file as a workaround - you get a kernel panic :)

onion2k 5y ago
That's a problem where you had no real choice to do anything else by the sounds of it though. You couldn't just "turn it off and on again" to fix it. In the case of a server where you can just have health check that reboots the server if memory gets a bit too low, it's far simpler just to automate a power cycle. It's not laziness or incompetence; it's understanding where to spend your effort best.
- ajuc 5y ago
  If we had memory leaks caused by drivers we would have to debug them all the way as well, restarting the device every few hours wasn't an option.
  - onion2k 5y ago
    Exactly. We all choose the most cost effective solution - it's a balance between engineering effort and how effective the fix is. In the case of a touch screen, rebootingit every few hours isn't as effective as debugging the underlying cause. In the case of a web server having a memory leak, killing a pod and bringing up a new one automatically when it gets unhealthy definitely is. It's not surprising that each dev team settles on different solutions to each problem.
    - ajuc 5y ago
      It still feels dirty though.
- xorcist 5y ago
  It's not that simple. Intermittently rebooting touchscreens make for a bad user experience. User experience is one of those things that doesn't matter until it suddently does, often when a competitor sells a product that doesn't reboot in the middle of a user operation.
  - deleted 5y ago
    [deleted]
unwind 5y ago
Wow, what an interesting bug! Was it a case of parsing "1.000" vs "1,000" or something like that?
- ajuc 5y ago
  Yup, the matrix for transforming hardware coordinates into software coordinates was saved in a config file using locale-specific number format. When user changed locale it remained in the previous format and if the change was English->German the matrix elements suddenly changed from 1.234 to 1234 :) the details are here: https://github.com/jiixyj/libinput/commit/98793f6b435abf8c17b84102de1d7c59ba8e3f54 https://github.com/jiixyj/libinput/commit/98793f6b435abf8c17... The funny thing is we had the issue once - reported it and added a workaround (added special handling for these config files that used locale-specific calibration tables), then a year later we updated the linux version and had the issue the opposite way till we removed our workaround :) But it was even more fun cause it bricked the devices (kernel panic on libinput startup = restart loop forever) :)

herbst 5y ago

The last company I worked at had several people employed that had a max 20% time for open source rule. To do more or less exactly what you just said. If all companies would don't care, open source wouldn't be there where it is.

onion2k 5y ago
There's a huge difference between giving engineers 20% time to work on open source code, and choosing to fix a memory leak by debugging the OS rather than using a health monitor to automatically restart an app. For a start, there's no guarantee that the devs will succeed. It might be a huge waste of time. That's a massive risk.
- xorcist 5y ago
  "Debugging the OS" sounds not just a little bit pretentious. Perhaps call it "troubleshooting" instead, because that's what's being done. That the problem was found in a driver doesn't make it any different from faulty shell script or a faulty hard disk.
- ajuc 5y ago
  > That's a massive risk. Leaving a leak in your code base is a risk too. What if it scales with something outside of your control and suddenly you have to restart every 15 seconds?
  - onion2k 5y ago
    Add more pods. Its literally the answer to every devOps problem. (This is a joke btw.) (... or is it?)
    - 123pie123 5y ago
      for some.. sadly not a joke
- herbst 5y ago
  Sure there is, however our company and debug culture also did not take restarts as a solution for memory leaks either. Sometimes quality matters more than money

bamboozled 5y ago

At some point we have to ask ourselves, who does care? Especially if you're running open source. Yes there are sponsored open source developers, but do the people who don't care even submit a bug report more than half the time?

hvgk 5y ago
I did always submit bug reports and patches. Most of the time some GitHub robot closes my ticket or PR after it is ignored for a month. That’s the new status quo: keep your ticket queue short by auto closing it.
- bamboozled 5y ago
  This made me laugh a lot.
- jeltz 5y ago
  Yes I noticed recently that this is very common in the Kubernetes community whicj is very annoying.

detaro 5y ago

Very much depends on the domain. Not for a generic server-/web-app normally, but in other software fields not surprising. But of course "current developers" only work on webapps ;)

oblio 5y ago
> But of course "current developers" only work on webapps ;) The rule of thumb is that the vast, overwhelming majority of developers are working on business apps, especially Line Of Business (LOB) internal apps nobody ever sees. It's hard to get exact numbers for this stuff but I wouldn't be surprised if the ratio of base infrastructure developers (kernel, shell, drivers, compilers, runtimes, frameworks, libraries etc.) to app developers is 1:100.
- detaro 5y ago
  Not sure if a contrast between "base infrastructure" and "apps" in response to "only on webapps" is supposed to be a joke or not. Just because its not a webapp it doesn't mean it's "base infrastructure".
  - oblio 5y ago
    I said apps, not just webapps. I'm not sure I see your point in this context.

Aissen 5y ago

That's either that, or pay the most expensive support contract from your OS vendor. You can also ignore it, but you'll end up running into a wall at some point… hopefully after you left the company ?

ckdarby 5y ago

I'm only 12 years in the industry and that is young compared to some others here but I'll give my perspective. I would honestly say that all developers outside of companies who activity are going to run out of cash in 6 months have the time and are given the choice. When management says, we don't have the time the reality is the proposed time given goes beyond a reasonable allocation of time, the max capacity given and or managers bullshit radar amount. Great developers also are able to factor these tasks into their own ticket time to make the time for the issue. I've been in countless meetings where individuals will complain about the state of X, how this is a hack, or we should really fix that. Plenty of times I've seen, "You've got two days of dedicated time to identify, solve and implement." This is met with various results, individuals taking the time, don't really do any work and consider it free time. They spend two days just trying to understand even how all the components fit together. Even if you revisit in a month with another two days they'll spend the same two days relearning the same information because they did not retain or make notes to be able to pickup where they left off. On a rare occasion someone will come in and fix it or move the needle further for the next person. All of these issues tend to funnel into these types of developers and from my sampling they're ~1 per 100 developers at a company.

saagarjha 5y ago

I literally do this all the time. My job is to make the software I work on perform well, and sometimes that means debugging the operating system and figuring out why it’s not doing that. Assuming I bring results by doing this (and I do!) nobody is going to tell me to stop: they honestly could not care less with how I went about my job.

matheusmoreira 5y ago

You'd think professional engineers would have enough autonomy to make decisions like these...