4 ms·
Not long, but perhaps not for the reasons you think. With the code base frozen, no new incompatibilities can be introduced. That's how software (including opera
by notacoward 4y ago
Not long, but perhaps not for the reasons you think. With the code base frozen, no new incompatibilities can be introduced. That's how software (including operating system) releases worked for many years, and they did work. VM/container images are still basically this way, and they tend to keep running quite well too. One problem with both of these models is that existing security vulnerabilities don't get fixed, but that's not code rot. It's a serious but separate issue.
No, the way I think a frozen codebase will rear its head is when things that haven't been restarted in too long start to misbehave. When new updates are being pushed out frequently by a busy team, every machine (container, process, whatever) is being restarted regularly. People don't even realize what they're avoiding each time. Resource leaks, deadlocks, and timer issues can remain undetected a long time in that environment, then surface when the longevity increases. Even worse, some things tend to fail after a predictable amount of time, i.e. across most or all machines at once in a cluster.
I first saw this in 1990 at Encore, when our shiny new SMP version of UNIX became capable of staying up for a whole month at a time. Because of problems like those mentioned above, getting from one month to two took almost as long as getting from zero to one. I've seen the same scenario play out on many other systems since, and there's no reason to believe the systems within Twitter are immune.
Stasis is not the same as stability.
- bydlocoder 4y agoDude, they use machine learning to tune JVM and OS scheduler, they'll handle restarts just fine.
- solarkraft 4y agoJust put some Tesla autopilot engineers on it.
- orangepanda 4y agoTo accelerate server crashes?
- notacoward 4y ago"They didn't get run over the last two times they ran across the freeway, so running across the freeway is perfectly safe." They developed those methods of dealing with JVM and scheduler issues because those problems rose in prominence against the background of a zillion other problems. They became the longest poles, most worth spending time and effort on. But those other problems still exist, with more appearing every day. Yes, even without code change, because environments and workloads change too. Anyone who has actually been responsible for large complex systems, not just in the sense of working at the same team/company and riding the coat-tails of those who actually understood it, knows that they require continued attention and always will. Entropy exists.
- thrwyoilarticle 4y agoI spoke to a HFT software engineer who said how useful the limited market opening hours can be - memory leaks are a non-issue, just kill it at the end of the day.
- dangerface 4y agoThis is very true. I learned this the hard way when I moved from making games that only ran for an hour to making server stuff and wondering why I couldn't keep a site up for more than a day and how are you supposed to debug if it takes a day for it to reach failure point. I haven't had the issue in a long time but I can see that being an issue if you are used to break stuff and release quick.