4 ms·

Elon Musk is disconnecting random Twitter-servers just to see what happens

20 points by ahiknsr 4y ago

brad0 4y ago
It reminds me somewhat of chaos testing. Killing random components can be useful, maybe not like this though.
- Oras 4y ago
  Chaos Monkey by Netflix https://netflix.github.io/chaosmonkey/ https://netflix.github.io/chaosmonkey/
  - themodelplumber 4y ago
    Chaos CEO by Overwhelm
  - mindcrash 4y ago
    Or Chaos Mesh if you want to break (and learn more about how to properly set up) Kubernetes clusters. https://chaos-mesh.org/ https://chaos-mesh.org/
- nunez 4y ago
  Yup this is classic scream testing
_the_inflator 4y ago
I will never buy a Tesla. If this is his best practice approach, I am speechless. Even if you do things like that, do it in a sane, scientific manner as a test, and not to sound like a fool who plays with to many buttons as superadmin.
- LargoLasskhyfv 4y ago
  Why? http://principlesofchaos.org/ http://principlesofchaos.org/ are real.
  - fsociety 4y ago
    If you disconnect the server without a plan, you are likely to miss downstream results if they occur. Better to treat it as a science experiment and have a plan.
    - LargoLasskhyfv 4y ago
      Bouncing around on my pogo stick, I randomly hit switches. Jumping harder, I go higher, zooming out. Watching for glitches far below. bright patterns, stops, jams, smoke, serenely drawing another toke. Any plans I doubt.
- encryptluks2 4y ago
  Systems are supposed to be able to handle random servers going offline. While it might be best to post a notice to users, it isn't unheard of to ensure your system is working as intended.
vital 4y ago
This may explain why, according to my anecdotal case, Twitter web app "Report Tweet" flows and Notifications are currently broken (for example, Report an issue modal "done" click starts acting like browsers back nav, randomly). Some Twitter features have been really wonky for the last 2 days or so.
deleted 4y ago
[deleted]
more_corn 4y ago
Bro, redundancy. If you want a resilient service your operations team will build in automatic redundancy. They’ll even build in auto-failover so they don’t have to wake up at 2am to fix something if a pdu fails. You don’t even have to ask for that. You just hire well and they do it because it’s the correct way to do it. But if you fire the majority of your operations team and then go unplugging racks you’re going to reach a point where you’ve taken out the redundancy and the whole service falls over on its face. Whatever, I shouldn’t have to explain these topics to you. A) you should know them B) if you don’t know them after high level work at Tesla, spacex and PayPal you obviously can’t learn them (and god help the people doing reliability work at those companies) and C) I don’t really care if your little empire crumbles to dust Go nuts boyo, maybe the money you save on your power bill can make up that extra billion a year your ill-considered acquisition is costing the company. That’s a joke it can’t. The only hope was advertising. The people you drive away with your antics. Perhaps next time bother to learn the business model of a company you plan on acquiring before you gleefully kill the goose that lays the golden egg.
LinuxBender 4y ago
When I ran a datacenter in California, our company was acquired by a company in Texas. The Texans came out and started pulling that crap. I advised the customers the causes of their outages and it stopped given it impacted their rather expensive service level agreements. I suppose in this case Elon can absorb any revenue losses but surely there is a better way to ease-in a chaos monkey test in a more professional and strategic manor. One could come up with a tech-refresh plan that implements a design retrofit for anything that does not have n+1 automated redundancy.
cactusplant7374 4y ago
Musk is in Sacramento (according to his jet). I think he’s planning to shut down Twitter’s data center to save cash.
lolc 4y ago
Hitting the switch tests assumptions and builds confidence. That's all great when one knows the redundancy setup. In this case one doesn't, so one shouldn't. For scrappy startups where the main investor has admin access, the mean time to shut down is measured in months.
sacnoradhq 4y ago
To a non-engineering manager or non-SRE, this may seem absurd. We do these in a more planned way where I work and call them "fire drills". Everything prod should be HA and evict-able from a particular server, rack, and/or DC floor and survive. If it's not, it's a sign something is wrong. The goal isn't fun, it's to find corner cases and places that aren't HA by meatcloud, chaos monkey-style. Ideally, lights-out PDUs and Ethernet switch port control can simulate "pulling plugs" in an automated way and do so remotely and more cheaply. If you bought a company, maybe you'd do this once to get a baseline, but with the express intention of not doing it for fun or to be a jerk.
prirun 4y ago
This is like a person unfamiliar with a large program going in and randomly removing lines of code, then trying the program. If the program still runs, obviously those lines of code were unnecessary, right? Damn he's dumb. He needs a nanny.

20 points by ahiknsr 4y ago

brad0 4y ago

It reminds me somewhat of chaos testing. Killing random components can be useful, maybe not like this though.

Oras 4y ago
Chaos Monkey by Netflix https://netflix.github.io/chaosmonkey/ https://netflix.github.io/chaosmonkey/
- themodelplumber 4y ago
  Chaos CEO by Overwhelm
- mindcrash 4y ago
  Or Chaos Mesh if you want to break (and learn more about how to properly set up) Kubernetes clusters. https://chaos-mesh.org/ https://chaos-mesh.org/
nunez 4y ago
Yup this is classic scream testing

_the_inflator 4y ago

I will never buy a Tesla. If this is his best practice approach, I am speechless. Even if you do things like that, do it in a sane, scientific manner as a test, and not to sound like a fool who plays with to many buttons as superadmin.

LargoLasskhyfv 4y ago
Why? http://principlesofchaos.org/ http://principlesofchaos.org/ are real.
- fsociety 4y ago
  If you disconnect the server without a plan, you are likely to miss downstream results if they occur. Better to treat it as a science experiment and have a plan.
  - LargoLasskhyfv 4y ago
    Bouncing around on my pogo stick, I randomly hit switches. Jumping harder, I go higher, zooming out. Watching for glitches far below. bright patterns, stops, jams, smoke, serenely drawing another toke. Any plans I doubt.
encryptluks2 4y ago
Systems are supposed to be able to handle random servers going offline. While it might be best to post a notice to users, it isn't unheard of to ensure your system is working as intended.

vital 4y ago

This may explain why, according to my anecdotal case, Twitter web app "Report Tweet" flows and Notifications are currently broken (for example, Report an issue modal "done" click starts acting like browsers back nav, randomly). Some Twitter features have been really wonky for the last 2 days or so.

deleted 4y ago

[deleted]

more_corn 4y ago

Bro, redundancy. If you want a resilient service your operations team will build in automatic redundancy. They’ll even build in auto-failover so they don’t have to wake up at 2am to fix something if a pdu fails. You don’t even have to ask for that. You just hire well and they do it because it’s the correct way to do it. But if you fire the majority of your operations team and then go unplugging racks you’re going to reach a point where you’ve taken out the redundancy and the whole service falls over on its face. Whatever, I shouldn’t have to explain these topics to you. A) you should know them B) if you don’t know them after high level work at Tesla, spacex and PayPal you obviously can’t learn them (and god help the people doing reliability work at those companies) and C) I don’t really care if your little empire crumbles to dust Go nuts boyo, maybe the money you save on your power bill can make up that extra billion a year your ill-considered acquisition is costing the company. That’s a joke it can’t. The only hope was advertising. The people you drive away with your antics. Perhaps next time bother to learn the business model of a company you plan on acquiring before you gleefully kill the goose that lays the golden egg.

LinuxBender 4y ago

When I ran a datacenter in California, our company was acquired by a company in Texas. The Texans came out and started pulling that crap. I advised the customers the causes of their outages and it stopped given it impacted their rather expensive service level agreements. I suppose in this case Elon can absorb any revenue losses but surely there is a better way to ease-in a chaos monkey test in a more professional and strategic manor. One could come up with a tech-refresh plan that implements a design retrofit for anything that does not have n+1 automated redundancy.

cactusplant7374 4y ago

Musk is in Sacramento (according to his jet). I think he’s planning to shut down Twitter’s data center to save cash.

lolc 4y ago

Hitting the switch tests assumptions and builds confidence. That's all great when one knows the redundancy setup. In this case one doesn't, so one shouldn't. For scrappy startups where the main investor has admin access, the mean time to shut down is measured in months.

sacnoradhq 4y ago

To a non-engineering manager or non-SRE, this may seem absurd. We do these in a more planned way where I work and call them "fire drills". Everything prod should be HA and evict-able from a particular server, rack, and/or DC floor and survive. If it's not, it's a sign something is wrong. The goal isn't fun, it's to find corner cases and places that aren't HA by meatcloud, chaos monkey-style. Ideally, lights-out PDUs and Ethernet switch port control can simulate "pulling plugs" in an automated way and do so remotely and more cheaply. If you bought a company, maybe you'd do this once to get a baseline, but with the express intention of not doing it for fun or to be a jerk.

prirun 4y ago

This is like a person unfamiliar with a large program going in and randomly removing lines of code, then trying the program. If the program still runs, obviously those lines of code were unnecessary, right? Damn he's dumb. He needs a nanny.