8 ms·
If there is such a level of privilege in Twitter's stack, that says a great deal about their technology. Insiders must not be able to act as users except in pre
by jeffbee 6y ago
If there is such a level of privilege in Twitter's stack, that says a great deal about their technology. Insiders must not be able to act as users except in prescribed ways requiring two-person control, logged and 100% audited. Glass-breaking privilege escalation should set off every pager in the company.
- shakezula 6y agothis assumes two things: that there is a security model that would prevent this attack that they should have implemented, and that alarms _weren't_ set off. Both of those are weak assumptions.
- GVIrish 6y agoI don't think parent was assuming those measures are implemented. They were saying that they should be implemented and if they are not, it betrays seriously poor security posture at Twitter.
- somehnguy 6y agoSorry, but would you mind expanding slightly on how you would implement such a system? In my understanding once you remove all the layers of abstraction as some point it's a bunch of databases and data stores. Someone has to manage them. Why wouldn't a breach of those users be able to do whatever they want? And a higher level, someone is writing the code to implement such a stringent access system. Why wouldn't a breach of those users (or a rogue employee) be able to accomplish bad things?
- jeffbee 6y agoGlad you asked. "There is a database and some guy is the DBA" is a very outdated architecture that can get you passing grades as an undergraduate and that's about all its good for. You should not take as a given that the right to modify datastores falls ultimately upon some individual. It is possible to permanently discard this ability, and organizations should strive for that.
- somehnguy 6y agoThats not what I meant, sorry. How do you implement such a system? So theres a team to manage the datastores, but that changes nothing that on some level someone somewhere has root passwords and/or filesystem access and/or ability to modify the fleet. We all know access controls and multiple operators are good, yeah. But at the heart of it is still a bunch of linux machines that have to be managed and deployed to. Which as far as I know has no mechanism for check with operator x before running command from operator 0.
- adamantoise 6y agoAWS KMS has a great whitepaper explaining how they do it here: https://d0.awsstatic.com/whitepapers/KMS-Cryptographic-Details.pdf https://d0.awsstatic.com/whitepapers/KMS-Cryptographic-Detai... The tl;dr is that they use hardware security modules (HSMs) with quorum-based access controls. Any administrative actions such as deploying software or changing the list of authorized operators requires a quorum of operators to sign a command for that action using their respective private keys. While this system was designed specifically around protecting customers' private keys, you could imagine a similar system around large databases.
- cardamomo 6y agoYour comment got me thinking: what does Twitter's infrastructure look like. This is from 2017, so I'm sure it's changed since then, but I found it interesting: https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale.html https://blog.twitter.com/engineering/en_us/topics/infrastruc...
- jeffbee 6y ago> someone somewhere has root passwords Not necessary > or filesystem access Also no > or ability to modify the fleet. Not that either. It feel like the conversation around these things is stuck in the far past. Large-scale organizations can and have driven the number of people with root passwords to zero. "Filesystem access" shouldn't be as easy as you're implying and it also shouldn't be of any use, since everything in the files ought to be separately encrypted with keys that can only be unwrapped by authorized systems. Even the last thing you said about Linux systems starting processes ... even a minor application of imagination can lead you to think of an init daemon that can enforce the pedigree of every process on the machine.
- belorn 6y ago> requiring two-person control, logged and 100% audited That would be good from a security perspective, but it would cost additional training, require more support staff, increase response time between request and resolve, make the system more complex and possible fragile, and take development resources away from profit centers. Most companies has likely, at best, the same security at their internal support center as their accounting department, and given how common CEO fraud is, it mean social engineering will likely continue to be a major attack vector for a long time.