4 ms·

I have three servers still running since 2014, I know this because they run Ubuntu 14. They've never been upgraded, they've only been rebooted a few times by l

by buro9 3y ago

I have three servers still running since 2014, I know this because they run Ubuntu 14. They've never been upgraded, they've only been rebooted a few times by linode during maintenance. I have no way of building them again, I have no backups, and yet... They are the production servers for a web app with 250k monthly users (not a commercial venture). Seemingly I'm fine with this... They've just been that reliable. A python so old I've no migration path, a django so old I've no migration path. But yet they keep working The only thing updated in a decade is an API built in go, which still compiles on latest go despite being written for pre 1.0 go. And I did replace the postgres server as that had outgrown it's instance. But everything else, a decade old and still fine. I logged in the other day to discover that SSH didn't initially work as ssh+RSA has been superseded and needed new ssh config just to keep connecting. There are so many things on these servers that no longer make sense, graphite monitoring that goes nowhere, new relic integration I disabled years ago, linode Longview long deprecated. And yet still the servers work as load balancers, and app servers for ancient python programs.

ornornor 3y ago
> 250k users Very bold… How do you keep it secure with such ancient software? What happens when it goes down and you don’t have backups? I can’t imagine a service that’s both unimportant and has so many users. So many questions.
- count 3y ago
  Plausible deniability for other operations out of the same system?
- buro9 3y ago
  > How do you keep it secure with such ancient software? What happens when it goes down and you don’t have backups? Only 3 ports are open: 80 443 <another for ssh that isn't 20> The nginx is just a proxy to the frontend and a file cache, I'm confident I could rebuild it in under an hour even without access to what is there today. Also confident that if it's compromised it can't do much real harm, I will nuke it. The django is just a thin front end that calls an API, I'm confident that when it gets compromised it can't do further damage as it has no direct access to the DB. And TBH, if/when it dies it will be a kick up the butt to reimplementing this in Go (I've about 60% of that already done but I only look at it once a year for a few evenings) > I can’t imagine a service that’s both unimportant and has so many users. So many questions. It's a platform with 300 forums on it. And I guess if it goes down I discover either the users didn't care (no big loss) or the users really care (and perhaps now the donations would be sufficient to cover the eng time for me to get someone to complete the frontend rewrite in Go). The whole thing is provided for free, and I figure people get what they pay for. I used to feel very emotionally attached, but now I'm quite YOLO about it. I believe these things should be ephemeral, if it turns out it has run it's time then so be it.
  - treesknees 3y ago
    Hackers love servers like yours. It's old and they know nobody is monitoring it. If they can get in and use it to launch attacks as a proxy or part of a botnet without touching your running services, they have a nice home for a long time and you'll never notice. Maybe you don't care about the data running on that server. It's still irresponsible and a disservice to the rest of the Internet to run an unpatched Internet-facing server.
    - buro9 3y ago
      There's a million routers that are unpatched and easier to get into than this. It's not entirely unmonitored, Linode send bandwidth usage warnings and iops warnings, and my users have been the best downtime signal. I'm very fine with these servers. They belong to hobby web, a low threshold for making something for yourself and others should be encouraged, but also the burden should be carried lightly, and this burden is carried very lightly and if it falls I'm fine with it.
justsomehnguy 3y ago
> I have no backups, and yet There is nothing what stops you from, at least, dd'ing to some other server, even if at Linode, too. > I logged in the other day to discover that SSH didn't initially work as ssh+RSA has been superseded and needed new ssh config just to keep connecting. Yeah, I have a 2T PKI, originally from 2012R2. I needed to re-issue one certificate and I needed an older OpenSSL binary to split it to pem+key pair.

by buro9 3y ago

ornornor 3y ago

> 250k users Very bold… How do you keep it secure with such ancient software? What happens when it goes down and you don’t have backups? I can’t imagine a service that’s both unimportant and has so many users. So many questions.

count 3y ago
Plausible deniability for other operations out of the same system?
buro9 3y ago
> How do you keep it secure with such ancient software? What happens when it goes down and you don’t have backups? Only 3 ports are open: 80 443 <another for ssh that isn't 20> The nginx is just a proxy to the frontend and a file cache, I'm confident I could rebuild it in under an hour even without access to what is there today. Also confident that if it's compromised it can't do much real harm, I will nuke it. The django is just a thin front end that calls an API, I'm confident that when it gets compromised it can't do further damage as it has no direct access to the DB. And TBH, if/when it dies it will be a kick up the butt to reimplementing this in Go (I've about 60% of that already done but I only look at it once a year for a few evenings) > I can’t imagine a service that’s both unimportant and has so many users. So many questions. It's a platform with 300 forums on it. And I guess if it goes down I discover either the users didn't care (no big loss) or the users really care (and perhaps now the donations would be sufficient to cover the eng time for me to get someone to complete the frontend rewrite in Go). The whole thing is provided for free, and I figure people get what they pay for. I used to feel very emotionally attached, but now I'm quite YOLO about it. I believe these things should be ephemeral, if it turns out it has run it's time then so be it.
- treesknees 3y ago
  Hackers love servers like yours. It's old and they know nobody is monitoring it. If they can get in and use it to launch attacks as a proxy or part of a botnet without touching your running services, they have a nice home for a long time and you'll never notice. Maybe you don't care about the data running on that server. It's still irresponsible and a disservice to the rest of the Internet to run an unpatched Internet-facing server.
  - buro9 3y ago
    There's a million routers that are unpatched and easier to get into than this. It's not entirely unmonitored, Linode send bandwidth usage warnings and iops warnings, and my users have been the best downtime signal. I'm very fine with these servers. They belong to hobby web, a low threshold for making something for yourself and others should be encouraged, but also the burden should be carried lightly, and this burden is carried very lightly and if it falls I'm fine with it.

justsomehnguy 3y ago

> I have no backups, and yet There is nothing what stops you from, at least, dd'ing to some other server, even if at Linode, too. > I logged in the other day to discover that SSH didn't initially work as ssh+RSA has been superseded and needed new ssh config just to keep connecting. Yeah, I have a 2T PKI, originally from 2012R2. I needed to re-issue one certificate and I needed an older OpenSSL binary to split it to pem+key pair.