3 ms·

> One of Stefano Marinelli's NetBSD boxes sat quietly serving for a decade, because everyone forgot about it. This is how Unix is meant to be. I remember someo

by hk1337 2y ago

> One of Stefano Marinelli's NetBSD boxes sat quietly serving for a decade, because everyone forgot about it. This is how Unix is meant to be. I remember someone mentioning they did this with Linux back in 1994 or 1995. Not for a decade obviously but it had been running for at least a year with no reboots or needing maintenance

lproven 2y ago
There's an order of magnitude difference between running for a year and a decade... That's a lot.
- appendix-rock 2y ago
  Yep. Not rebooting for a decade is in most cases an irresponsible ‘advanced rookie’ move. I’m more than aware of the greybeard-era uptime fetishism. I’ve more than dabbled in it. I’ve typed /exec uptime into my IRC client more times than I dare admit. But come on… All that’s been said about security updates etc aside (some of which can be mitigated with that fancy in-place kernel update stuff), If something hasn’t rebooted in 10 years I’m going to be a bit nervous about what happens when it does reboot. If it’s in an uptime fetishist environment, chances are that it’ll be rebooted at a time that’s…inconvenient to say the least. Are you SURE that nothing has changed in that time? Some people are! Moreso than others at least. But that’s extra work, and my bet is most places with these high-uptime machines aren’t putting that work In, or think they are and are doing it poorly.
- Delk 2y ago
  A couple of years ago at work, we came across a Linux (virtual) server related to a customer project that had been running for some 2.5 or 3 years since the last reboot. Pretty much by accident. I think we rebooted it for good measure (applying security updates, making sure it still booted cleanly, etc.) but there was no immediate need for doing that just to keep it functional. It's entirely possible that BSD is more stable and lends itself better to running for really long times uninterrupted. But Linux systems running for years really aren't unheard of, and by no means is one year the top of the range. Whether you should do that is another question. Over the course of a decade, there could well be even kernel-level vulnerabilities discovered, let alone ones in other services running on top of it. You might have a system running without a reboot for years as long as you make sure to update (and restart) user space services as needed. But leaving an entire server unattended for years doesn't sound like a good idea generally. That may not be as much of a concern if what the box is running is a limited set of services or functionality with little exposed surface. But that then comes more down to "what you're doing with it" rather than "which OS you're using". The generally less conservative development culture around Linux leans more towards moving fast and breaking things, although generally while trying to avoid the latter. Perhaps that makes things like low-level OS vulnerabilities or whether the system still restarts cleanly after a decade more important in the Linux land, and what counts as prudent administration in Linux might be less of a concern in BSD. But if you can have a BSD box running for a decade, with some particular set of services, in an internal network(?), and then compare that to someone else's report of a Linux box running for (at least) a year, in 1994 or 1995, probably running an entirely different workload, in a different environment (perhaps externally exposed?), and with no indication of why it may or may not have been restarted after that time, that's not really a fair comparison either.
- hk1337 2y ago
  Well, yeah but also at this time Linux had not even been around half a decade. My thought was more that maybe a lot of the issues people are having with Linux to push them to BSD is what we, developers collectively, have done to Linux over the last 2-3 decades.
  - lproven 2y ago
    > Well, yeah but also at this time Linux had not even been around half a decade. Ah, yes, good point! > My thought was more that maybe a lot of the issues people are having with Linux to push them to BSD is what we, developers collectively, have done to Linux over the last 2-3 decades. Indeed so. Especially in the last decade or so: snap, Flatpak, Wayland, systemd, etc. It's being made gratuitously more and more complex to meet the demands of the main commercial users -- running cloud servers -- even if this makes like much more complicated for individual users. Which is why I advocate the BSDs as an alternative, but man, they are all forbidding and offputting to beginners/novices, and some of them don't even realise why and how.
eesmith 2y ago
The classic urban legend is of a Novell server rediscovered four years after being walled in, https://www.theregister.com/2001/04/12/missing_novell_server_discovered_after/ https://www.theregister.com/2001/04/12/missing_novell_server... . See also https://skeptics.stackexchange.com/questions/32502/did-a-computer-server-get-accidentally-walled-in https://skeptics.stackexchange.com/questions/32502/did-a-com... . The Reg also has articles like https://www.theregister.com/2016/01/20/486_fleet_still_in_production/ https://www.theregister.com/2016/01/20/486_fleet_still_in_pr... 'Eighteen year old server trumped by functional 486 fleet!'
swiftcoder 2y ago
The problem with a box that has been running the same critical service for years on end, is that nobody knows if it would actually boot back up successfully in the event of a reboot/crash/power cut/etc... I inherited administration over a number of one-off critical linux systems back in ~2010 that had 6+ years of uptime. Spent a long time analysing their contents, and then building replacement stacks with redundancy alongside them, and carefully cutting traffic across... At the end of the day, you need to be able to reboot/upgrade servers regularly (even if you don't in practice do it very often).
- eru 2y ago
  Yes, you should semi-regularly restart everything (both OS and server processes), so you know you can bring them back up.
  - kemotep 2y ago
    Even restoring from the latest backups, running the two in parallel as you hand off the services and then shutdown the old one. You confirm that backups work, and that deploying the service is still well documented/automations work.
    - eru 2y ago
      Yes, the only backups you can count on working are those you practice restoring from.
- gtvwill 2y ago
  I just spent the last 4 days recovering payroll systems and data from machines that have no backup. Turns out they were storing all the payroll data on an external hdd in a folder called "dead hdd recovery" that was from the last time I recovered a dead system for them about 4 years ago. They weren't ment to keep using that external hdd or folder but they did. They also weren't interested in other forms of redundancy the last time the payroll machine died. This time however the building had a fire go through the main office and took out a quarter of the facility. I finally convinced them to atleast get the data on one drive for a start. First fire they'd had in 40 years of operating. I still can't convince them to switch to a cloud based payroll/book keeping system but hey atleast it's now backed up across two terminals, one that's kept offsite and a 3rd location of onedrive in the cloud. It's been a fun few days to say the least. Astounded the drives survived whilst their chassis melted around them (samsung ssds and asus pn51-e1 mini pcs). They got insanely lucky.

by hk1337 2y ago

lproven 2y ago

There's an order of magnitude difference between running for a year and a decade... That's a lot.

appendix-rock 2y ago
Yep. Not rebooting for a decade is in most cases an irresponsible ‘advanced rookie’ move. I’m more than aware of the greybeard-era uptime fetishism. I’ve more than dabbled in it. I’ve typed /exec uptime into my IRC client more times than I dare admit. But come on… All that’s been said about security updates etc aside (some of which can be mitigated with that fancy in-place kernel update stuff), If something hasn’t rebooted in 10 years I’m going to be a bit nervous about what happens when it does reboot. If it’s in an uptime fetishist environment, chances are that it’ll be rebooted at a time that’s…inconvenient to say the least. Are you SURE that nothing has changed in that time? Some people are! Moreso than others at least. But that’s extra work, and my bet is most places with these high-uptime machines aren’t putting that work In, or think they are and are doing it poorly.
Delk 2y ago
A couple of years ago at work, we came across a Linux (virtual) server related to a customer project that had been running for some 2.5 or 3 years since the last reboot. Pretty much by accident. I think we rebooted it for good measure (applying security updates, making sure it still booted cleanly, etc.) but there was no immediate need for doing that just to keep it functional. It's entirely possible that BSD is more stable and lends itself better to running for really long times uninterrupted. But Linux systems running for years really aren't unheard of, and by no means is one year the top of the range. Whether you should do that is another question. Over the course of a decade, there could well be even kernel-level vulnerabilities discovered, let alone ones in other services running on top of it. You might have a system running without a reboot for years as long as you make sure to update (and restart) user space services as needed. But leaving an entire server unattended for years doesn't sound like a good idea generally. That may not be as much of a concern if what the box is running is a limited set of services or functionality with little exposed surface. But that then comes more down to "what you're doing with it" rather than "which OS you're using". The generally less conservative development culture around Linux leans more towards moving fast and breaking things, although generally while trying to avoid the latter. Perhaps that makes things like low-level OS vulnerabilities or whether the system still restarts cleanly after a decade more important in the Linux land, and what counts as prudent administration in Linux might be less of a concern in BSD. But if you can have a BSD box running for a decade, with some particular set of services, in an internal network(?), and then compare that to someone else's report of a Linux box running for (at least) a year, in 1994 or 1995, probably running an entirely different workload, in a different environment (perhaps externally exposed?), and with no indication of why it may or may not have been restarted after that time, that's not really a fair comparison either.
hk1337 2y ago
Well, yeah but also at this time Linux had not even been around half a decade. My thought was more that maybe a lot of the issues people are having with Linux to push them to BSD is what we, developers collectively, have done to Linux over the last 2-3 decades.
- lproven 2y ago
  > Well, yeah but also at this time Linux had not even been around half a decade. Ah, yes, good point! > My thought was more that maybe a lot of the issues people are having with Linux to push them to BSD is what we, developers collectively, have done to Linux over the last 2-3 decades. Indeed so. Especially in the last decade or so: snap, Flatpak, Wayland, systemd, etc. It's being made gratuitously more and more complex to meet the demands of the main commercial users -- running cloud servers -- even if this makes like much more complicated for individual users. Which is why I advocate the BSDs as an alternative, but man, they are all forbidding and offputting to beginners/novices, and some of them don't even realise why and how.

eesmith 2y ago

The classic urban legend is of a Novell server rediscovered four years after being walled in, https://www.theregister.com/2001/04/12/missing_novell_server_discovered_after/ https://www.theregister.com/2001/04/12/missing_novell_server... . See also https://skeptics.stackexchange.com/questions/32502/did-a-computer-server-get-accidentally-walled-in https://skeptics.stackexchange.com/questions/32502/did-a-com... . The Reg also has articles like https://www.theregister.com/2016/01/20/486_fleet_still_in_production/ https://www.theregister.com/2016/01/20/486_fleet_still_in_pr... 'Eighteen year old server trumped by functional 486 fleet!'

swiftcoder 2y ago

The problem with a box that has been running the same critical service for years on end, is that nobody knows if it would actually boot back up successfully in the event of a reboot/crash/power cut/etc... I inherited administration over a number of one-off critical linux systems back in ~2010 that had 6+ years of uptime. Spent a long time analysing their contents, and then building replacement stacks with redundancy alongside them, and carefully cutting traffic across... At the end of the day, you need to be able to reboot/upgrade servers regularly (even if you don't in practice do it very often).

eru 2y ago
Yes, you should semi-regularly restart everything (both OS and server processes), so you know you can bring them back up.
- kemotep 2y ago
  Even restoring from the latest backups, running the two in parallel as you hand off the services and then shutdown the old one. You confirm that backups work, and that deploying the service is still well documented/automations work.
  - eru 2y ago
    Yes, the only backups you can count on working are those you practice restoring from.
gtvwill 2y ago
I just spent the last 4 days recovering payroll systems and data from machines that have no backup. Turns out they were storing all the payroll data on an external hdd in a folder called "dead hdd recovery" that was from the last time I recovered a dead system for them about 4 years ago. They weren't ment to keep using that external hdd or folder but they did. They also weren't interested in other forms of redundancy the last time the payroll machine died. This time however the building had a fire go through the main office and took out a quarter of the facility. I finally convinced them to atleast get the data on one drive for a start. First fire they'd had in 40 years of operating. I still can't convince them to switch to a cloud based payroll/book keeping system but hey atleast it's now backed up across two terminals, one that's kept offsite and a 3rd location of onedrive in the cloud. It's been a fun few days to say the least. Astounded the drives survived whilst their chassis melted around them (samsung ssds and asus pn51-e1 mini pcs). They got insanely lucky.