7 ms·

Your site is getting hacked, you don't know how the hackers are doing it, what do you do ops wise? Take the whole site down for a few hours? Because the entire

Your site is getting hacked, you don't know how the hackers are doing it, what do you do ops wise? Take the whole site down for a few hours? Because the entire platform is compromised, how do you handle that?

rsanheim 6y ago
Yes, of course. Take the site down if you don't have a read only mode or something. You are losing millions in trust every minute this hack goes on.
- cryptoz 6y ago
  Indeed, already billions in trust lost so far, guessing by the ~4+% after-hours TWTR drop.
  - throwaway43234 6y ago
    To be fair, it's still higher than yesterday's low. It's not like TWTR is known to increase over time anyways.
- eternalban 6y ago
  This is possibly a blessing in disguise. Obama and Biden's accounts have been hacked as well so this basically just burned Twitter as an international political platform. Following that thought, it is entirely possible the whole point of the hack is to discredit Twitter and the bitcoin bit is just smoke.
- jcims 6y ago
  They just disabled posts from verified users.
  - mlindner 6y ago
    Yes, but it took them nearly 2 hours to do that, in the middle of the work day no less.
    - swagonomixxx 6y ago
      Its that epic WFH productivity!
      - azernik 6y ago
        I doubt this is a productivity issue or an infrastructure issue - shutting off write access is a major business and reputational loss, and I can easily see cultural factors pushing people not to take that step.
        kabacha 6y ago
        Multi-billion company with thousands of employees of a shitty micro-blogging website can't make their site read-only lol Just shows how massively overvalued these projects are — crash and burn.
  - blisseyGo 6y ago
    Was this for verified users only? Or is that only verified users were targeted?
axaxs 6y ago
Should have went read-only when the flood started. If they didn't have the forethought to have a read-only mode, then yes, show a failwhale while they investigate.
Nextgrid 6y ago
Worst case scenario, shut down the app servers, load-balancers or even the network equipment that connects the platform to the Internet.
tetha 6y ago
More of a b2b context. However, we've had an unannounced pentester achieve RCE on our systems. Not a fun situation. At that point, we were forced by our contracts, and data protection laws, and a CEO aware of all of these, to shut the affected productive system down. We stopped all services, set the firewalls of our hoster to only accept traffic from our office and that's it, while figuring out wtf happened. Those measures overall reduce the situation to a known situation again. If someone in our office is hostile.. that's another issue. After a bit of analysis, we figured out the IPs attacking us and we blacklisted those on the firewall of the other production systems. Eventually things cleared up to be a pentest no one told us about. If the attack had moved into these other systems, we'd have to extend the nuclear solution to those systems too. At that point, we'd have to lockout some 30k+ FTE users. I think we'd be able to make national news with that for our customers. Except.. not good news.
- htrp 6y ago
  When you say unannounced pentester, how the hell did that happen? Usually, isn't someone in the escalation chain aware of these types of things?
  - tetha 6y ago
    A manager at a customer told a pentester to take our system without telling anyone. As simple as that. The pentester did. We axed their system. This was elevated in ridiculousness, because said manager was backpedaling really, really hard after we contacted the pen-testing company as well as the customers senior management. However, all attempts at re-instating the system were swiftly blocked by the customers security policies and security teams. So, the system stayed down for a solid amount of time. After all, the customer insisted on us participating in their security workflows for that system under their security teams control. And from their companies point of view, this was an external hostile attack -- since the manager didn't tell anyone.
deleted 6y ago
[deleted]
Aperocky 6y ago
If you can't have a log trail that establish how someone tweeted something, might as well shut down then. It should become very apparent how this is done through the correct levels of logging. Unless of course twitter backend firefighting team consists of hasty tooling that writes directly to production table with no oversight (which also sounds like a possibility)..

rsanheim 6y ago

Yes, of course. Take the site down if you don't have a read only mode or something. You are losing millions in trust every minute this hack goes on.

cryptoz 6y ago
Indeed, already billions in trust lost so far, guessing by the ~4+% after-hours TWTR drop.
- throwaway43234 6y ago
  To be fair, it's still higher than yesterday's low. It's not like TWTR is known to increase over time anyways.
eternalban 6y ago
This is possibly a blessing in disguise. Obama and Biden's accounts have been hacked as well so this basically just burned Twitter as an international political platform. Following that thought, it is entirely possible the whole point of the hack is to discredit Twitter and the bitcoin bit is just smoke.
jcims 6y ago
They just disabled posts from verified users.
- mlindner 6y ago
  Yes, but it took them nearly 2 hours to do that, in the middle of the work day no less.
  - swagonomixxx 6y ago
    Its that epic WFH productivity!
    - azernik 6y ago
      I doubt this is a productivity issue or an infrastructure issue - shutting off write access is a major business and reputational loss, and I can easily see cultural factors pushing people not to take that step.
      - kabacha 6y ago
        Multi-billion company with thousands of employees of a shitty micro-blogging website can't make their site read-only lol Just shows how massively overvalued these projects are — crash and burn.
- blisseyGo 6y ago
  Was this for verified users only? Or is that only verified users were targeted?

axaxs 6y ago

Should have went read-only when the flood started. If they didn't have the forethought to have a read-only mode, then yes, show a failwhale while they investigate.

Nextgrid 6y ago

Worst case scenario, shut down the app servers, load-balancers or even the network equipment that connects the platform to the Internet.

tetha 6y ago

More of a b2b context. However, we've had an unannounced pentester achieve RCE on our systems. Not a fun situation. At that point, we were forced by our contracts, and data protection laws, and a CEO aware of all of these, to shut the affected productive system down. We stopped all services, set the firewalls of our hoster to only accept traffic from our office and that's it, while figuring out wtf happened. Those measures overall reduce the situation to a known situation again. If someone in our office is hostile.. that's another issue. After a bit of analysis, we figured out the IPs attacking us and we blacklisted those on the firewall of the other production systems. Eventually things cleared up to be a pentest no one told us about. If the attack had moved into these other systems, we'd have to extend the nuclear solution to those systems too. At that point, we'd have to lockout some 30k+ FTE users. I think we'd be able to make national news with that for our customers. Except.. not good news.

htrp 6y ago
When you say unannounced pentester, how the hell did that happen? Usually, isn't someone in the escalation chain aware of these types of things?
- tetha 6y ago
  A manager at a customer told a pentester to take our system without telling anyone. As simple as that. The pentester did. We axed their system. This was elevated in ridiculousness, because said manager was backpedaling really, really hard after we contacted the pen-testing company as well as the customers senior management. However, all attempts at re-instating the system were swiftly blocked by the customers security policies and security teams. So, the system stayed down for a solid amount of time. After all, the customer insisted on us participating in their security workflows for that system under their security teams control. And from their companies point of view, this was an external hostile attack -- since the manager didn't tell anyone.

deleted 6y ago

[deleted]

Aperocky 6y ago

If you can't have a log trail that establish how someone tweeted something, might as well shut down then. It should become very apparent how this is done through the correct levels of logging. Unless of course twitter backend firefighting team consists of hasty tooling that writes directly to production table with no oversight (which also sounds like a possibility)..