15 ms·

This will be an interesting experiment in large scale code rot. How long do you think twitter.com will work with the code base frozen?

fotta 4y ago
The TLS cert expires on December 12.
- post_break 4y ago
  Oh man I just smiled like the Grinch.
  - jfoutz 4y ago
    Me too. I'm not proud of that. but it looks like somebody had their eyes on that issue Expires: Tuesday, November 14, 2023 at 3:59:59 PM Pacific Standard Time
    - fotta 4y ago
      Aww man. Glad I grabbed a screenshot before it got rotated.
- gonzo41 4y ago
  I had a job once, when TLS was new and half the internet was not using some form of SSL. We had to manually do keyswaps on a stack of machines to create a trust network for all the certs. IT really sucked. I made a note of the certificate expiry data and a calendar reminder 3 years into the future to start looking for a new job if I hadn't already.
  - 3-cheese-sundae 4y ago
    Hopefully that reminder never slipped pasta your attention.
    - gonzo41 4y ago
      Yeah!, for context, I saw and fixed that spelling mistake of calendar / colander. This week really has been a wild ride. Can't text fast enough.
  - nothrowaways 4y ago
    I hope you don't end up in a job where someone left on cert calendar reminder.
- dilyevsky 4y ago
  There is absolutely no way in hell those aren’t autorenewed. I think the answer is “quite a long time” actually. Eventually disk/machine rot in DCs will take over but it can take many months or more
  - SoftTalker 4y ago
    Everything I run at work is 5-10 years old or older. It all just hums along. Occasionally I replace a hard drive or a power supply.
    - sockgrant 4y ago
      You get massive DoS attacks pretty regularly? State sponsored hackers?
    - kccoder 4y ago
      Now multiply that frequency against the ratio of machines Twitter has vs you.
    - XorNot 4y ago
      On a modest Hadoop cluster of 300 machines with 20 disks a piece, we replaced 3 hard disks a week.
      - dilyevsky 4y ago
        Spindles? That sounds like a ridiculous amount burn rate - it should be no more than 5-10% a YEAR. Backblaze reports only 1.6% quarterly failure rate https://www.backblaze.com/b2/hard-drive-test-data.html https://www.backblaze.com/b2/hard-drive-test-data.html
        dilyevsky 4y ago
        Er actually missed its total 6k disks seems about right then
    - matwood 4y ago
- dragontamer 4y ago
  The certificate looks like it was created on Nov 14th and is valid until Nov 14th 2023 to me.
  - fotta 4y ago
    Yeah it looks like they rotated it. It used to be this one: https://crt.sh/?q=DA26F72EACAC75CCEE794140441020848697AEB0 https://crt.sh/?q=DA26F72EACAC75CCEE794140441020848697AEB0 The new one isn’t even indexed on crt.sh yet.
- herpderperator 4y ago
  My browser said that too, but that certificate may be cached from a while ago. The terminal gives a different set of dates: $ openssl s_client -servername twitter.com -connect twitter.com:443 2>/dev/null | openssl x509 -noout -dates notBefore=Nov 14 00:00:00 2022 GMT notAfter=Nov 14 23:59:59 2023 GMT Edit: Opening my browser in incognito now shows the new expiry too. The non-incognito version continues to show the old expiry.
  - fotta 4y ago
    I think they rotated it. The new one isn’t in crt.sh yet, but for posterity’s sake this is the old one https://crt.sh/?q=DA26F72EACAC75CCEE794140441020848697AEB0 https://crt.sh/?q=DA26F72EACAC75CCEE794140441020848697AEB0
- deleted 4y ago
  [deleted]
notacoward 4y ago
Not long, but perhaps not for the reasons you think. With the code base frozen, no new incompatibilities can be introduced. That's how software (including operating system) releases worked for many years, and they did work. VM/container images are still basically this way, and they tend to keep running quite well too. One problem with both of these models is that existing security vulnerabilities don't get fixed, but that's not code rot. It's a serious but separate issue. No, the way I think a frozen codebase will rear its head is when things that haven't been restarted in too long start to misbehave. When new updates are being pushed out frequently by a busy team, every machine (container, process, whatever) is being restarted regularly. People don't even realize what they're avoiding each time. Resource leaks, deadlocks, and timer issues can remain undetected a long time in that environment, then surface when the longevity increases. Even worse, some things tend to fail after a predictable amount of time, i.e. across most or all machines at once in a cluster. I first saw this in 1990 at Encore, when our shiny new SMP version of UNIX became capable of staying up for a whole month at a time. Because of problems like those mentioned above, getting from one month to two took almost as long as getting from zero to one. I've seen the same scenario play out on many other systems since, and there's no reason to believe the systems within Twitter are immune. Stasis is not the same as stability.
- bydlocoder 4y ago
  Dude, they use machine learning to tune JVM and OS scheduler, they'll handle restarts just fine.
  - solarkraft 4y ago
    Just put some Tesla autopilot engineers on it.
    - orangepanda 4y ago
      To accelerate server crashes?
  - notacoward 4y ago
    "They didn't get run over the last two times they ran across the freeway, so running across the freeway is perfectly safe." They developed those methods of dealing with JVM and scheduler issues because those problems rose in prominence against the background of a zillion other problems. They became the longest poles, most worth spending time and effort on. But those other problems still exist, with more appearing every day. Yes, even without code change, because environments and workloads change too. Anyone who has actually been responsible for large complex systems, not just in the sense of working at the same team/company and riding the coat-tails of those who actually understood it, knows that they require continued attention and always will. Entropy exists.
themenomen 4y ago
Until the AWS credit card gives up.
- JyB 4y ago
  Is twitter completely on AWS?
jillesvangurp 4y ago
Here's a crazy idea: 1) get rid of most of the existing code base and simply re-implement and migrate existing content and remaining users to the new thing. 2) get some small team together that does this. 3) stop bleeding cash while that plan is being set into motion. 4) keep the existing code base limping along while that is being put into motion. I'm pretty sure that that's the plan Elon Musk is executing. It's simple, brutal, nasty, and probably not very well thought through. And yes he's not being very nice about it (to put it mildly). But it's not completely crazy if you can get over how obnoxious he's being. He's clearly not interested in keeping the current team. He's never going to win them over at this point and from his point of view they were part of the problem, not the solution in any case. He bought a company that was basically not profitable, bleeding cash, and fresh out of ideas on how to fix any of that for many years. And he's now on the spot to keep that train wreck going. So, first order of business is to just cut the cost. That means decimating the team to a fraction of its size. The Twitter team is basically rage quitting at this point so that's solved. Again, I don't like how he's doing it but undeniably, there are going to be vastly less people on the payroll very soon and cost will be a lot lower. The law suits that follow will be interesting though. But my guess is that he'll just buy those off and that will be it. Of those that remain, he can cherry pick a team to salvage/build whatever he needs and then re-enforce it with some outsiders. Building a micro blogging platform is not rocket science and Elon Musk is actually deeply into rocket science. So, I actually get his gut feeling that this simply should not require thousands of engineers obsessing about all sorts of arcane software bits and pieces that somehow implement Twitter. It's a micro blogging platform with a fairly simplistic feature set. And he just called bullshit on how that was done at Twitter. And we're talking about a person with a long track record of having absolutely no tolerance for bull shit. Again, not liking his methods but I can see his line of reasoning here. He has no respect for the Twitter team and he actually wants them gone. He wants to rebuild Twitter in a way that is cost effective and interesting to him. And he is not a risk averse person. So, in his mind that means a small team that can execute fast. I kind of agree with that part. Will this work? I don't know. I see no technical reason that this plan could not work. But the big question is if the Twitter brand name will be worth anything by the time he's done or if he's just going to shut down the whole thing and walk away from it. He can certainly afford to do that. But letting this thing coast along just never was going to be a thing.
yreg 4y ago
What code freeze? Surely Elon will keep pushing the team to merge and deploy changes fast as long as there are any engineers left. They will break it themselves.

fotta 4y ago

The TLS cert expires on December 12.

post_break 4y ago
Oh man I just smiled like the Grinch.
- jfoutz 4y ago
  Me too. I'm not proud of that. but it looks like somebody had their eyes on that issue Expires: Tuesday, November 14, 2023 at 3:59:59 PM Pacific Standard Time
  - fotta 4y ago
    Aww man. Glad I grabbed a screenshot before it got rotated.
gonzo41 4y ago
I had a job once, when TLS was new and half the internet was not using some form of SSL. We had to manually do keyswaps on a stack of machines to create a trust network for all the certs. IT really sucked. I made a note of the certificate expiry data and a calendar reminder 3 years into the future to start looking for a new job if I hadn't already.
- 3-cheese-sundae 4y ago
  Hopefully that reminder never slipped pasta your attention.
  - gonzo41 4y ago
    Yeah!, for context, I saw and fixed that spelling mistake of calendar / colander. This week really has been a wild ride. Can't text fast enough.
- nothrowaways 4y ago
  I hope you don't end up in a job where someone left on cert calendar reminder.
dilyevsky 4y ago
There is absolutely no way in hell those aren’t autorenewed. I think the answer is “quite a long time” actually. Eventually disk/machine rot in DCs will take over but it can take many months or more
- SoftTalker 4y ago
  Everything I run at work is 5-10 years old or older. It all just hums along. Occasionally I replace a hard drive or a power supply.
  - sockgrant 4y ago
    You get massive DoS attacks pretty regularly? State sponsored hackers?
  - kccoder 4y ago
    Now multiply that frequency against the ratio of machines Twitter has vs you.
  - XorNot 4y ago
    On a modest Hadoop cluster of 300 machines with 20 disks a piece, we replaced 3 hard disks a week.
    - dilyevsky 4y ago
      Spindles? That sounds like a ridiculous amount burn rate - it should be no more than 5-10% a YEAR. Backblaze reports only 1.6% quarterly failure rate https://www.backblaze.com/b2/hard-drive-test-data.html https://www.backblaze.com/b2/hard-drive-test-data.html
      - dilyevsky 4y ago
        Er actually missed its total 6k disks seems about right then
  - matwood 4y ago
dragontamer 4y ago
The certificate looks like it was created on Nov 14th and is valid until Nov 14th 2023 to me.
- fotta 4y ago
  Yeah it looks like they rotated it. It used to be this one: https://crt.sh/?q=DA26F72EACAC75CCEE794140441020848697AEB0 https://crt.sh/?q=DA26F72EACAC75CCEE794140441020848697AEB0 The new one isn’t even indexed on crt.sh yet.
herpderperator 4y ago
My browser said that too, but that certificate may be cached from a while ago. The terminal gives a different set of dates: $ openssl s_client -servername twitter.com -connect twitter.com:443 2>/dev/null | openssl x509 -noout -dates notBefore=Nov 14 00:00:00 2022 GMT notAfter=Nov 14 23:59:59 2023 GMT Edit: Opening my browser in incognito now shows the new expiry too. The non-incognito version continues to show the old expiry.
- fotta 4y ago
  I think they rotated it. The new one isn’t in crt.sh yet, but for posterity’s sake this is the old one https://crt.sh/?q=DA26F72EACAC75CCEE794140441020848697AEB0 https://crt.sh/?q=DA26F72EACAC75CCEE794140441020848697AEB0
deleted 4y ago
[deleted]

notacoward 4y ago

Not long, but perhaps not for the reasons you think. With the code base frozen, no new incompatibilities can be introduced. That's how software (including operating system) releases worked for many years, and they did work. VM/container images are still basically this way, and they tend to keep running quite well too. One problem with both of these models is that existing security vulnerabilities don't get fixed, but that's not code rot. It's a serious but separate issue. No, the way I think a frozen codebase will rear its head is when things that haven't been restarted in too long start to misbehave. When new updates are being pushed out frequently by a busy team, every machine (container, process, whatever) is being restarted regularly. People don't even realize what they're avoiding each time. Resource leaks, deadlocks, and timer issues can remain undetected a long time in that environment, then surface when the longevity increases. Even worse, some things tend to fail after a predictable amount of time, i.e. across most or all machines at once in a cluster. I first saw this in 1990 at Encore, when our shiny new SMP version of UNIX became capable of staying up for a whole month at a time. Because of problems like those mentioned above, getting from one month to two took almost as long as getting from zero to one. I've seen the same scenario play out on many other systems since, and there's no reason to believe the systems within Twitter are immune. Stasis is not the same as stability.

bydlocoder 4y ago
Dude, they use machine learning to tune JVM and OS scheduler, they'll handle restarts just fine.
- solarkraft 4y ago
  Just put some Tesla autopilot engineers on it.
  - orangepanda 4y ago
    To accelerate server crashes?
- notacoward 4y ago
  "They didn't get run over the last two times they ran across the freeway, so running across the freeway is perfectly safe." They developed those methods of dealing with JVM and scheduler issues because those problems rose in prominence against the background of a zillion other problems. They became the longest poles, most worth spending time and effort on. But those other problems still exist, with more appearing every day. Yes, even without code change, because environments and workloads change too. Anyone who has actually been responsible for large complex systems, not just in the sense of working at the same team/company and riding the coat-tails of those who actually understood it, knows that they require continued attention and always will. Entropy exists.

themenomen 4y ago

Until the AWS credit card gives up.

JyB 4y ago
Is twitter completely on AWS?

jillesvangurp 4y ago

Here's a crazy idea: 1) get rid of most of the existing code base and simply re-implement and migrate existing content and remaining users to the new thing. 2) get some small team together that does this. 3) stop bleeding cash while that plan is being set into motion. 4) keep the existing code base limping along while that is being put into motion. I'm pretty sure that that's the plan Elon Musk is executing. It's simple, brutal, nasty, and probably not very well thought through. And yes he's not being very nice about it (to put it mildly). But it's not completely crazy if you can get over how obnoxious he's being. He's clearly not interested in keeping the current team. He's never going to win them over at this point and from his point of view they were part of the problem, not the solution in any case. He bought a company that was basically not profitable, bleeding cash, and fresh out of ideas on how to fix any of that for many years. And he's now on the spot to keep that train wreck going. So, first order of business is to just cut the cost. That means decimating the team to a fraction of its size. The Twitter team is basically rage quitting at this point so that's solved. Again, I don't like how he's doing it but undeniably, there are going to be vastly less people on the payroll very soon and cost will be a lot lower. The law suits that follow will be interesting though. But my guess is that he'll just buy those off and that will be it. Of those that remain, he can cherry pick a team to salvage/build whatever he needs and then re-enforce it with some outsiders. Building a micro blogging platform is not rocket science and Elon Musk is actually deeply into rocket science. So, I actually get his gut feeling that this simply should not require thousands of engineers obsessing about all sorts of arcane software bits and pieces that somehow implement Twitter. It's a micro blogging platform with a fairly simplistic feature set. And he just called bullshit on how that was done at Twitter. And we're talking about a person with a long track record of having absolutely no tolerance for bull shit. Again, not liking his methods but I can see his line of reasoning here. He has no respect for the Twitter team and he actually wants them gone. He wants to rebuild Twitter in a way that is cost effective and interesting to him. And he is not a risk averse person. So, in his mind that means a small team that can execute fast. I kind of agree with that part. Will this work? I don't know. I see no technical reason that this plan could not work. But the big question is if the Twitter brand name will be worth anything by the time he's done or if he's just going to shut down the whole thing and walk away from it. He can certainly afford to do that. But letting this thing coast along just never was going to be a thing.

yreg 4y ago

What code freeze? Surely Elon will keep pushing the team to merge and deploy changes fast as long as there are any engineers left. They will break it themselves.