7 ms·

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even do

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to. So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building. But eventually people are going to start turning on stoves again… and then…

woodruffw 4y ago
This is correct, but also: a sudden decline in maintenance is a kind of change in its own right. Even automated processes have humans in the loop and manual sign-offs; there's always some cronjob or short-lived certificate somewhere that a human was dutifully maintaining. Those things aren't going to fail any sooner than they would have anyways, but they're going to fail a lot harder due to the loss of institutional knowledge.
- RijilV 4y ago
  Welp, their TLS cert expires second week of the new year. I really hope for them that’s an automated process.
jandrese 4y ago
Except that he at the same time demanded that people invent an entirely new dish by the end of the week, and now they are scrambling to try to figure it out. Already the DMCA auto-takedown bot is apparently broken and people are posting entire movies on Twitter. I would expect other peripheral systems to start breaking down as nobody is maintaining them even as other parts of the system are being changed.
- bboygravity 4y ago
  oh noes! Not the DMCA auto-takedown bot! Said no-one in the entire world except a hand-full of Hollywood studio owners. That bot shouldn't have existed in the first place, but I know that that falls under "just world fallacy" and is a naive thought.
  - emodendroket 4y ago
    Well it is exposing them to significant legal risk if they no longer comply, isn't it?
  - woodruffw 4y ago
    I interpreted the GP's comment less as a moral claim ("the DMCA bot is good") and more as a claim that the DMCA bot's failure is a strong indicator of internal instability (given that it sits directly at the intersection between Twitter's profit interests and microservices architecture). Put another way: being unable keep a little bot running, one that keeps an entire industry happy, doesn't bode well for other components of the service.
    - edgyquant 4y ago
      No it’s proof that people will take anything and run with it. This bot likely had low priority and that’s all
      - woodruffw 4y ago
        Run with what? It seems self-evident that the bot was considered low priority, since it isn’t working anymore. But nobody is disputing that: they’re saying that the fact that it is low priority does not bode well.
      - yourapostasy 4y ago
        > This bot likely had low priority and that’s all If it was a prerequisite to land $100M ARR from all the media properties’ marketing budgets to advertise the multi-billion dollar pipelines of the movie and entertainment industry, that lil’ bot was the gate to $11,415 per hour of revenue at risk if its uptime failed to sufficiently please the attorneys and auditors from those customers.
  - toast0 4y ago
    I mean, does Twitter want to be a party to a copyright lawsuit? If not, following legitimate looking DMCA notifications (and legitimate looking DMCA counter-notifications) and responding to suponeas as necessary gets you an affirmative defense for copyright infringement. You may not like it, but having a bot do that probably saves a lot of legal hassle.
- kaba0 4y ago
  2FA as well
- edgyquant 4y ago
  Are we in support of DMCA now?
  - deleted 4y ago
    [deleted]
  - thrwyoilarticle 4y ago
    I don't think you really think that's what OP is saying.
afavour 4y ago
> Twitter isn’t shipping many new features right now Is that true? I thought one of Elon’s big pushes was launching the whole Blue Tick subscription thing. That doesn’t feel like a small feature.
- EdwardDiego 4y ago
  Given that Twitter already offered premium API access, they've got billing in place, so now they add a new form that, once your credit card is verified, flicks a boolean on an account that was previously flicked by another process. It might not be small, but it's not exactly huge.
  - fomine3 4y ago
    Blue is paid by in app purchase that seems to haven't implemented until that. Its functionality is far from premium API access.
    - EdwardDiego 4y ago
      As I said, a new form. They already had billing integration, even if you're accessing it via a new route, and they already had a boolean on your account. Like I said, not small, but not exactly huge either.
      - sundvor 4y ago
        Except there was a crapload of social engineering testing around this feature that was just skipped altogether. Suddenly that simple change had the potential for catastrophic consequences.
        EdwardDiego 4y ago
        Yep, doing it right would've been harder, but Musk was after doing it right now.
miked85 4y ago
> Twitter isn’t shipping many new features right now When was the last major new feature? The site has always seemed pretty stagnant.
- emodendroket 4y ago
  They outright built me-too versions of both Clubhouse and IG Stories, and a lot of stuff trying to encourage people to be nice.
Bukhmanizer 4y ago
I was just about to make the same analogy. The chance that anything will break when there are no new commits is way lower

woodruffw 4y ago

This is correct, but also: a sudden decline in maintenance is a kind of change in its own right. Even automated processes have humans in the loop and manual sign-offs; there's always some cronjob or short-lived certificate somewhere that a human was dutifully maintaining. Those things aren't going to fail any sooner than they would have anyways, but they're going to fail a lot harder due to the loss of institutional knowledge.

RijilV 4y ago
Welp, their TLS cert expires second week of the new year. I really hope for them that’s an automated process.

jandrese 4y ago

Except that he at the same time demanded that people invent an entirely new dish by the end of the week, and now they are scrambling to try to figure it out. Already the DMCA auto-takedown bot is apparently broken and people are posting entire movies on Twitter. I would expect other peripheral systems to start breaking down as nobody is maintaining them even as other parts of the system are being changed.

bboygravity 4y ago
oh noes! Not the DMCA auto-takedown bot! Said no-one in the entire world except a hand-full of Hollywood studio owners. That bot shouldn't have existed in the first place, but I know that that falls under "just world fallacy" and is a naive thought.
- emodendroket 4y ago
  Well it is exposing them to significant legal risk if they no longer comply, isn't it?
- woodruffw 4y ago
  I interpreted the GP's comment less as a moral claim ("the DMCA bot is good") and more as a claim that the DMCA bot's failure is a strong indicator of internal instability (given that it sits directly at the intersection between Twitter's profit interests and microservices architecture). Put another way: being unable keep a little bot running, one that keeps an entire industry happy, doesn't bode well for other components of the service.
  - edgyquant 4y ago
    No it’s proof that people will take anything and run with it. This bot likely had low priority and that’s all
    - woodruffw 4y ago
      Run with what? It seems self-evident that the bot was considered low priority, since it isn’t working anymore. But nobody is disputing that: they’re saying that the fact that it is low priority does not bode well.
    - yourapostasy 4y ago
      > This bot likely had low priority and that’s all If it was a prerequisite to land $100M ARR from all the media properties’ marketing budgets to advertise the multi-billion dollar pipelines of the movie and entertainment industry, that lil’ bot was the gate to $11,415 per hour of revenue at risk if its uptime failed to sufficiently please the attorneys and auditors from those customers.
- toast0 4y ago
  I mean, does Twitter want to be a party to a copyright lawsuit? If not, following legitimate looking DMCA notifications (and legitimate looking DMCA counter-notifications) and responding to suponeas as necessary gets you an affirmative defense for copyright infringement. You may not like it, but having a bot do that probably saves a lot of legal hassle.
kaba0 4y ago
2FA as well
edgyquant 4y ago
Are we in support of DMCA now?
- deleted 4y ago
  [deleted]
- thrwyoilarticle 4y ago
  I don't think you really think that's what OP is saying.

afavour 4y ago

> Twitter isn’t shipping many new features right now Is that true? I thought one of Elon’s big pushes was launching the whole Blue Tick subscription thing. That doesn’t feel like a small feature.

EdwardDiego 4y ago
Given that Twitter already offered premium API access, they've got billing in place, so now they add a new form that, once your credit card is verified, flicks a boolean on an account that was previously flicked by another process. It might not be small, but it's not exactly huge.
- fomine3 4y ago
  Blue is paid by in app purchase that seems to haven't implemented until that. Its functionality is far from premium API access.
  - EdwardDiego 4y ago
    As I said, a new form. They already had billing integration, even if you're accessing it via a new route, and they already had a boolean on your account. Like I said, not small, but not exactly huge either.
    - sundvor 4y ago
      Except there was a crapload of social engineering testing around this feature that was just skipped altogether. Suddenly that simple change had the potential for catastrophic consequences.
      - EdwardDiego 4y ago
        Yep, doing it right would've been harder, but Musk was after doing it right now.

miked85 4y ago

> Twitter isn’t shipping many new features right now When was the last major new feature? The site has always seemed pretty stagnant.

emodendroket 4y ago
They outright built me-too versions of both Clubhouse and IG Stories, and a lot of stuff trying to encourage people to be nice.

Bukhmanizer 4y ago

I was just about to make the same analogy. The chance that anything will break when there are no new commits is way lower