4 ms·

Funnily enough, in one previous company I worked, which used Linux exclusively, the software was very buggy, and the servers were rebooted daily. We jokingly ca

by 21 9y ago

Funnily enough, in one previous company I worked, which used Linux exclusively, the software was very buggy, and the servers were rebooted daily. We jokingly called that the "Windows solution". When we proposed adding a daily reboot to cron, the tech lead (which encouraged practices which lead to this low quality software) retorted that "this in not Windows, it doesn't need constant reboots", totally missing the point that just using Linux doesn't make you a developer of reliable software.

noir_lord 9y ago
I used to have a service that was like that, it was third party and only handled a specific function so I had a cron job that would reboot it at 4am every day. Sometimes the hammer approach is the only solution.
- ianai 9y ago
  Was the software loading kernel modules? I'm curious why a more simple kill process routine wouldn't work.
  - LoSboccacc 9y ago
    Having been in a similar situation, many enterprise software require a precise start order and shutdown, sometime even depending to certain system services or some arcane stuff that then breaks in unpredictable times and often the only sane way to restart the environment is to bring it down whole
    - noir_lord 9y ago
      Exactly this, it was a horrible piece of enterprise software that depended on other similary horrible pieces of crap. The only way to get it into a known good state when it shit the bed was to reboot the machine and then let it sort itself out. We rapidly moved away from using it. Funnily enough I'm in a similar situation at the new job with Jasper Reports, god damn if that thing isn't everything bad about Enterprise Java(TM).
superhans2 9y ago
Sounds familiar. I had a client once, who had to reboot one particular router every day because it stopped working after 8h uptime. So one of their employees did that every morning by logging into some server via remote desktop to click the "reboot" button. I asked why they don't use some kind of cron job to automate that task, they just said "it doesn't work that way, you have to do it manually".
- in9 9y ago
  Some people are anti-automation. I am an economist, turned into "data scientist" since I've learned to program in the past 5 years (hate that name...). At a macro consultancy firm I worked, everybody lost it when I suggested that we moved our manually downloaded data from a bunch of excel spreadsheets to a proper database (28 years of macroeconomic data) so that we could programatically extract data for online reports we sent/hosted. They said I was being lazy...
  - tudorconstantin 9y ago
    I hope you took that as a compliment. Laziness is a virtue in programming, according to Larry Wall: http://threevirtues.com/ http://threevirtues.com/
takeda 9y ago
Am I idealist by wondering that people think adding a cron job is a good solution? Why not trying to fix the broken software?
- loonattic 9y ago
  Sometimes, it's just not possible nor reasonably doable. As seen above, with proprietary drivers...
kbenson 9y ago
> "this in not Windows, it doesn't need constant reboots", totally missing the point that just using Linux doesn't make you a developer of reliable software. Yes... but no software should require the Linux OS to reboot unless you're running custom or known buggy kernel modules, or you've triggered a spiral of death through swap usage. If your program is misbehaving, kill the program and restart it. Restarting the OS daily is like noticing that your car uses a lot of gas, and deciding that every time you fill it up you'll get an oil change too. You might need to fill up a lot, but the oil change is overkill and not really affecting the situation in one way or another.
disconnected 9y ago
I think that if you are around in this business long enough, you will eventually experience something like this. We had a Linux server that had some issue with a network driver that was causing kernel oops with long uptimes (about once a week). This one was particularly nasty because the oops would disable the network interface, meaning that you couldn't ssh to diagnose the fault (or, more likely, reboot), which meant that you had to drive to the premises and kick the server in the guts. Of course, Murphy's law ensured that the server would fail at the worst possible time. Late Friday, weekends, your mom's birthday, etc... Not fun, at all. The solution was to write a cron job to reboot every day at ~4:30AM. Stupid? Yes. But we all agreed that it sure beat the alternative (driving, kicking, sobbing). The driver was eventually fixed by the vendor, and this "hack" became unnecessary.
IntelMiner 9y ago
I had a fun issue for about 6 months with a Cable (DOCSIS 3.0) modem I'd purchased. A Motorola/Arris Surfboard 6183 The modem would randomly seem to keel over with some unknown fault. Causing my internet speed to drop from 300Mbps down to 0.25Mbps, ping to Google.com for instance would then spike from 5ms to 1900ms (or more) Curiously the upload speed would stay pegged at 30Mbps however! After a few days of this happening, I picked up a Chinese "Smart Switch" that ran OpenWRT, and set up a small shell script to simply ping Google.com, then cycle the modem if the average ping results exceeded a certain threshold (I think 100ms?) It would also record the exact date and time and log that, so I could try and correlate the issue. Unfortunately it seemed to be utterly random, without any rhyme or reason. Since I worked at Comcast at the time, I tried to gather more data on the issue internally. Eventually writing a report that totaled something around 10 pages. From what I gathered: There were no physical signal deviations when the device would "hang". The device would respond normally to SNMP requests etc, everything on Comcast's side appeared normal. The device had some internal fault with its software that was causing problems (kernel bug perhaps?) I contacted Motorola/Arris for support, and was advised that the warranty specifically excludes Software faults(!) and then kindly recommended to "upgrade" to the newer SB6190 model. Unfortunately being a Cable modem, the firmware is completely controlled by the ISP. Since there were only 25,000~ SB6183's on Comcast's network at the time, and even fewer on the speed tier that I had, there was not enough data to report the issue back to Motorola/Arris through Comcast Eventually a software update was pushed out which corrected the issue roughly 6 months later

by 21 9y ago

noir_lord 9y ago

I used to have a service that was like that, it was third party and only handled a specific function so I had a cron job that would reboot it at 4am every day. Sometimes the hammer approach is the only solution.

ianai 9y ago
Was the software loading kernel modules? I'm curious why a more simple kill process routine wouldn't work.
- LoSboccacc 9y ago
  Having been in a similar situation, many enterprise software require a precise start order and shutdown, sometime even depending to certain system services or some arcane stuff that then breaks in unpredictable times and often the only sane way to restart the environment is to bring it down whole
  - noir_lord 9y ago
    Exactly this, it was a horrible piece of enterprise software that depended on other similary horrible pieces of crap. The only way to get it into a known good state when it shit the bed was to reboot the machine and then let it sort itself out. We rapidly moved away from using it. Funnily enough I'm in a similar situation at the new job with Jasper Reports, god damn if that thing isn't everything bad about Enterprise Java(TM).

superhans2 9y ago

Sounds familiar. I had a client once, who had to reboot one particular router every day because it stopped working after 8h uptime. So one of their employees did that every morning by logging into some server via remote desktop to click the "reboot" button. I asked why they don't use some kind of cron job to automate that task, they just said "it doesn't work that way, you have to do it manually".

in9 9y ago
Some people are anti-automation. I am an economist, turned into "data scientist" since I've learned to program in the past 5 years (hate that name...). At a macro consultancy firm I worked, everybody lost it when I suggested that we moved our manually downloaded data from a bunch of excel spreadsheets to a proper database (28 years of macroeconomic data) so that we could programatically extract data for online reports we sent/hosted. They said I was being lazy...
- tudorconstantin 9y ago
  I hope you took that as a compliment. Laziness is a virtue in programming, according to Larry Wall: http://threevirtues.com/ http://threevirtues.com/

takeda 9y ago

Am I idealist by wondering that people think adding a cron job is a good solution? Why not trying to fix the broken software?

loonattic 9y ago
Sometimes, it's just not possible nor reasonably doable. As seen above, with proprietary drivers...

kbenson 9y ago

> "this in not Windows, it doesn't need constant reboots", totally missing the point that just using Linux doesn't make you a developer of reliable software. Yes... but no software should require the Linux OS to reboot unless you're running custom or known buggy kernel modules, or you've triggered a spiral of death through swap usage. If your program is misbehaving, kill the program and restart it. Restarting the OS daily is like noticing that your car uses a lot of gas, and deciding that every time you fill it up you'll get an oil change too. You might need to fill up a lot, but the oil change is overkill and not really affecting the situation in one way or another.

disconnected 9y ago

I think that if you are around in this business long enough, you will eventually experience something like this. We had a Linux server that had some issue with a network driver that was causing kernel oops with long uptimes (about once a week). This one was particularly nasty because the oops would disable the network interface, meaning that you couldn't ssh to diagnose the fault (or, more likely, reboot), which meant that you had to drive to the premises and kick the server in the guts. Of course, Murphy's law ensured that the server would fail at the worst possible time. Late Friday, weekends, your mom's birthday, etc... Not fun, at all. The solution was to write a cron job to reboot every day at ~4:30AM. Stupid? Yes. But we all agreed that it sure beat the alternative (driving, kicking, sobbing). The driver was eventually fixed by the vendor, and this "hack" became unnecessary.

IntelMiner 9y ago

I had a fun issue for about 6 months with a Cable (DOCSIS 3.0) modem I'd purchased. A Motorola/Arris Surfboard 6183 The modem would randomly seem to keel over with some unknown fault. Causing my internet speed to drop from 300Mbps down to 0.25Mbps, ping to Google.com for instance would then spike from 5ms to 1900ms (or more) Curiously the upload speed would stay pegged at 30Mbps however! After a few days of this happening, I picked up a Chinese "Smart Switch" that ran OpenWRT, and set up a small shell script to simply ping Google.com, then cycle the modem if the average ping results exceeded a certain threshold (I think 100ms?) It would also record the exact date and time and log that, so I could try and correlate the issue. Unfortunately it seemed to be utterly random, without any rhyme or reason. Since I worked at Comcast at the time, I tried to gather more data on the issue internally. Eventually writing a report that totaled something around 10 pages. From what I gathered: There were no physical signal deviations when the device would "hang". The device would respond normally to SNMP requests etc, everything on Comcast's side appeared normal. The device had some internal fault with its software that was causing problems (kernel bug perhaps?) I contacted Motorola/Arris for support, and was advised that the warranty specifically excludes Software faults(!) and then kindly recommended to "upgrade" to the newer SB6190 model. Unfortunately being a Cable modem, the firmware is completely controlled by the ISP. Since there were only 25,000~ SB6183's on Comcast's network at the time, and even fewer on the speed tier that I had, there was not enough data to report the issue back to Motorola/Arris through Comcast Eventually a software update was pushed out which corrected the issue roughly 6 months later