5 ms·

This excerpt is frightening: > About half of the company's 500,000 servers run on outdated software that does not support basic security features such as encry

by Signez 4y ago

This excerpt is frightening: > About half of the company's 500,000 servers run on outdated software that does not support basic security features such as encryption for stored data or regular security updates by vendors

tppiotrowski 4y ago
I wonder if they're running Ubuntu on 32-bit hardware
- raverbashing 4y ago
  or RHEL 6
  - taf2 4y ago
    RHEL5 more likely based on when twitter was founded
  - imron 4y ago
    I wish this was a joke. I know of systems running multibillion dollar companies that are still using rhel6.
  - discodave 4y ago
    Hahahaha... Wait until you hear about the large cloud provider running RHEL5... (I worked at said provider).
kerng 4y ago
I wish companies generally would be more transparent - I'd imagine this be the norm at most companies.
secondcoming 4y ago
Why bother hacking Twitter when it'd be cheaper to bribe an employee to get all the information you want: > allows too many of its staff access to the platform's central controls and most sensitive information without adequate oversight It'd be even easier if you find an employee who's on the same political team as you.
- Mindwipe 4y ago
  The likelihood is that bad actors do. It's one of the reasons I disliked Twitter forcing the use of mobile numbers for 2FA, they're just not sufficiently trustworthy. And I have an account under my real name! If I were a political dissident etc that just feels like an insane idea.
- vineyardmike 4y ago
  Or worse: https://www.usnews.com/news/technology/articles/2022-08-23/india-forced-twitter-to-put-agent-on-payroll-whistleblower-says https://www.usnews.com/news/technology/articles/2022-08-23/i...
netsharc 4y ago
Just do it the Zuck way: "If you make an FB app, you can read all user's data and their friends' data, but click here to promise that you won't do that and you won't use the data to subvert democracies...".
- adrianmsmith 4y ago
  To be fair this hasn't been the case for many years.
rightbyte 4y ago
It is also frightening that they need half a million servers.
- jonahbenton 4y ago
  The JVM is a hungry beast.
  - qualudeheart 4y ago
    - hunter-gatherer 4y ago
      C has such a bad wrap with the HN crowd...
      - memling 4y ago
        > C has such a bad wrap with the HN crowd... why?
        encryptluks2 4y ago
        hn_throwaway_99 4y ago
        It really doesn't. After all, many (most?) other languages like Java and JavaScript are implemented primarily in C and/or C++. Where it gets deserved opprobrium is that it has no memory safety features, and thus inherently contributes to gobs of security vulnerabilities, and there are safer alternatives now, like Rust. C is basically "portable assembly", and it's rarely the right tool for the job these days.
        _vdpp 4y ago
        Can only patch so many buffer overflows, off-by-one errors, format string vulnerabilities, integer overflows, race conditions, use-after-free errors, etc, before it gets to be a bit tiring. Safer alternatives exist.
encryptluks2 4y ago
First, servers generally run on operating systems. No one with any serious knowledge would use the phrase run on software. Second, does this guy have any actual tech knowledge at all? He doesn't list what operating system they are running or what security updates he is expecting. It doesn't sound great but I assure you I've probably seen worse on systems used by the literal federal government to conduct official business and store sensitive information on. All government cares about is having remediation plans in place.
- vngzs 4y ago
  > Second, does this guy have any actual tech knowledge at all? He doesn't list what operating system they are running or what security updates he is expecting. "This guy": https://en.wikipedia.org/wiki/Peiter_Zatko https://en.wikipedia.org/wiki/Peiter_Zatko
  - encryptluks2 4y ago
    Then he should be in an even better position to specify what the actual issues are in details and not some abstract garbage. You could summarize the information there as.. "Momma, servers bad. Need encryption. Need updates."
    - koheripbal 4y ago
      They are intentionally vague for legal and security reasons.
      - encryptluks2 4y ago
        What legal and security reasons exactly?
        batch12 4y ago
        Publishing a detailed report of infrastructure and specific CVEs would be irresponsible and malicious. If that is off the table the only thing left is ambiguity. Also, the audience is important. They are going for maximum outrage, not glassy eyes.
jonahbenton 4y ago
The "does not support basic security features such as encryption for stored data" unquoted line of reporting is almost certainly not what Mudge wrote and is likely not literally true. That 500k servers in Twitter infra are missing patches certainly is true and what was likely in the original was a statement that stored data that should have been encrypted at rest was not, and/or that acceptable standards for data at rest encryption, a relatively rapidly moving freight train, were not maintained.
- hn_throwaway_99 4y ago
  I have discovered that there are vastly different definitions of "encryption for stored data" that can mean critically different things for security. One definition is "the underlying disk is encrypted". This is true, by default, of virtually all cloud environments these days. But it really only protects you against physical access to the storage media, which actually is far from the top threat. The other, more useful/meaningful definition, is "we encrypt everything at the application layer before it is placed into the DB, and all decryption requests are logged by user". For example, using an envelope encryption scheme to encrypt data before it is stored in a DB, and upon retrieval decrypting the data with a call to something like KMS. In that environment you can literally give readonly DB access to all your developers and not have to worry about PII being exposed. If hackers somehow got access to your DB, they wouldn't be able to read sensitive data, and if they also managed to get access to your KMS credentials, any attempts to decrypt the data would be tracked and logged. My point is that when many companies say "we encrypt your data", they are usually just talking about the first thing, but that doesn't really provide that much additional security. The second definition is really what you should be doing.
  - olliej 4y ago
    I think the good comparison that people encounter day-to-day is full disk encryption. It's the default on macOS, the only option on iOS, (those are the two platforms I use), and I assume the case on windows and android. The thing is FDE essentially only protects your data when your machine is powered off. Once your machine is booted and you've logged in any block level encryption ceases to be relevant, because to get to the point of running your machine has to have loaded in the relevant key material to decrypt. From that point on user space code no longer sees a difference between encrypted and decrypted drives. In other words FDE is not relevant is you lose a powered on device (post login if relevant to the platform), and you're the kind of person people are actively targeting (I recall recently? the content of someone's phone or such being dumped by the FBI because they grabbed it while it was being used). That's why modern OS's have different key classes, there's the lowest level which is just FDE, but you can have higher levels where requesting key material essentially just gives you a handle to that material. Then the OS, or preferably hardware with a much less complex OS, manages those handles and invalidates them according to policy rules. e.g you may want your phone to have access to your address book while your phone is locked, which does not mean you need your call history available as well. The policies provided by OSs tend to be fairly simple because it's better to have an easy to understand API that is easy to use and hard to screw up than a more "powerful" API that is easy to screw up and hard to use (the latter resulting in people simply not encrypting things at all). e.g iOS/macOS only has the following file protections when you create files: "NSFileProtectionComplete", "NSFileProtectionCompleteUnlessOpen", "NSFileProtectionCompleteUntilFirstUserAuthentication", "NSFileProtectionNone", but they're very easy to understand.[1] I tried to find the android equivalent but I don't know the terminology that's used and I just get linked to instructions on using AES, so if someone could link the correct doc I'd appreciate it. [1] https://support.apple.com/guide/security/data-protection-classes-secb010e978a/web https://support.apple.com/guide/security/data-protection-cla... and https://support.apple.com/guide/security/keychain-data-protection-secb0694df1a/web https://support.apple.com/guide/security/keychain-data-prote...
antegamisou 4y ago
Wasn't it them that had a bug that exposed users' passwords in plain text a few years back?
- sytringy05 4y ago
  I think they were logging them in clear text, so server side, prolly ended up in splunk or elasticsearch.
sylens 4y ago
I think it's also important to recognize how much of a "check the box" security control encryption at rest has become for many vendors/GRC teams. A lot of times, the encryption at rest control only has the capability to prevent somebody from physically detaching the disk and trying to mount it with their own machine and access the data that way. In a world where many companies now run their workloads on public cloud providers who keep their hardware in distributed cages in secure datacenters, this isn't the security control many assume it is. If you're trying to prevent an actor who has gained a foothold on a box/network from seeing plaintext data that is actually in use by the actual production system at that very moment, you're looking for a much stronger type of control - probably some sort of client-side encryption or obfuscation/tokenization
- physicles 4y ago
  Finally, someone points out the emperor has no clothes. When customers first started requesting encryption at rest, it didn’t make any sense to me — the threats it mitigates aren’t worth worrying about if you’re using public cloud. So it is just a checkbox then.
zionic 4y ago
I suspect this will be the norm going forward. Big tech was taken over by bean counters long ago, the fact that it’s all running on duct tape and popsicle sticks under the hood will come back to bite us when we have a digital Pearl Harbor event. China will invade Taiwan and the first shot won’t be physical, it will be activating the 30 years of assets they grew in AWS/GCP/cloudfare/level3/AT&T/Etc Most of their HR/engineering departments are completely retarded. They’ll hire any H1B who passes l33t code that accepts $50k under market rate then give them repo access in a few weeks. Our soulless megacorps are beyond easy to penetrate by hostile intelligence. The CIA/NSA/FBI, you know the groups who we pay billions per year for and they take half my income to fund will of course not catch any of this. The FBI is too busy manufacturing domestic terrorist, the NSA is too busy hacking American companies, and the CIA is too busy importing drugs to actually secure our country from foreign attack. Why? Because it’s been so long since we were actually attacked they believe it can’t happen so why not loot Rome in the mean time?

by Signez 4y ago

tppiotrowski 4y ago

I wonder if they're running Ubuntu on 32-bit hardware

raverbashing 4y ago
or RHEL 6
- taf2 4y ago
  RHEL5 more likely based on when twitter was founded
- imron 4y ago
  I wish this was a joke. I know of systems running multibillion dollar companies that are still using rhel6.
- discodave 4y ago
  Hahahaha... Wait until you hear about the large cloud provider running RHEL5... (I worked at said provider).

kerng 4y ago

I wish companies generally would be more transparent - I'd imagine this be the norm at most companies.

secondcoming 4y ago

Why bother hacking Twitter when it'd be cheaper to bribe an employee to get all the information you want: > allows too many of its staff access to the platform's central controls and most sensitive information without adequate oversight It'd be even easier if you find an employee who's on the same political team as you.

Mindwipe 4y ago
The likelihood is that bad actors do. It's one of the reasons I disliked Twitter forcing the use of mobile numbers for 2FA, they're just not sufficiently trustworthy. And I have an account under my real name! If I were a political dissident etc that just feels like an insane idea.
vineyardmike 4y ago
Or worse: https://www.usnews.com/news/technology/articles/2022-08-23/india-forced-twitter-to-put-agent-on-payroll-whistleblower-says https://www.usnews.com/news/technology/articles/2022-08-23/i...

netsharc 4y ago

Just do it the Zuck way: "If you make an FB app, you can read all user's data and their friends' data, but click here to promise that you won't do that and you won't use the data to subvert democracies...".

adrianmsmith 4y ago
To be fair this hasn't been the case for many years.

rightbyte 4y ago

It is also frightening that they need half a million servers.

jonahbenton 4y ago
The JVM is a hungry beast.
- qualudeheart 4y ago
  - hunter-gatherer 4y ago
    C has such a bad wrap with the HN crowd...
    - memling 4y ago
      > C has such a bad wrap with the HN crowd... why?
      - encryptluks2 4y ago
      - hn_throwaway_99 4y ago
        It really doesn't. After all, many (most?) other languages like Java and JavaScript are implemented primarily in C and/or C++. Where it gets deserved opprobrium is that it has no memory safety features, and thus inherently contributes to gobs of security vulnerabilities, and there are safer alternatives now, like Rust. C is basically "portable assembly", and it's rarely the right tool for the job these days.
      - _vdpp 4y ago
        Can only patch so many buffer overflows, off-by-one errors, format string vulnerabilities, integer overflows, race conditions, use-after-free errors, etc, before it gets to be a bit tiring. Safer alternatives exist.

encryptluks2 4y ago

First, servers generally run on operating systems. No one with any serious knowledge would use the phrase run on software. Second, does this guy have any actual tech knowledge at all? He doesn't list what operating system they are running or what security updates he is expecting. It doesn't sound great but I assure you I've probably seen worse on systems used by the literal federal government to conduct official business and store sensitive information on. All government cares about is having remediation plans in place.

vngzs 4y ago
> Second, does this guy have any actual tech knowledge at all? He doesn't list what operating system they are running or what security updates he is expecting. "This guy": https://en.wikipedia.org/wiki/Peiter_Zatko https://en.wikipedia.org/wiki/Peiter_Zatko
- encryptluks2 4y ago
  Then he should be in an even better position to specify what the actual issues are in details and not some abstract garbage. You could summarize the information there as.. "Momma, servers bad. Need encryption. Need updates."
  - koheripbal 4y ago
    They are intentionally vague for legal and security reasons.
    - encryptluks2 4y ago
      What legal and security reasons exactly?
      - batch12 4y ago
        Publishing a detailed report of infrastructure and specific CVEs would be irresponsible and malicious. If that is off the table the only thing left is ambiguity. Also, the audience is important. They are going for maximum outrage, not glassy eyes.

jonahbenton 4y ago

The "does not support basic security features such as encryption for stored data" unquoted line of reporting is almost certainly not what Mudge wrote and is likely not literally true. That 500k servers in Twitter infra are missing patches certainly is true and what was likely in the original was a statement that stored data that should have been encrypted at rest was not, and/or that acceptable standards for data at rest encryption, a relatively rapidly moving freight train, were not maintained.

hn_throwaway_99 4y ago
I have discovered that there are vastly different definitions of "encryption for stored data" that can mean critically different things for security. One definition is "the underlying disk is encrypted". This is true, by default, of virtually all cloud environments these days. But it really only protects you against physical access to the storage media, which actually is far from the top threat. The other, more useful/meaningful definition, is "we encrypt everything at the application layer before it is placed into the DB, and all decryption requests are logged by user". For example, using an envelope encryption scheme to encrypt data before it is stored in a DB, and upon retrieval decrypting the data with a call to something like KMS. In that environment you can literally give readonly DB access to all your developers and not have to worry about PII being exposed. If hackers somehow got access to your DB, they wouldn't be able to read sensitive data, and if they also managed to get access to your KMS credentials, any attempts to decrypt the data would be tracked and logged. My point is that when many companies say "we encrypt your data", they are usually just talking about the first thing, but that doesn't really provide that much additional security. The second definition is really what you should be doing.
- olliej 4y ago
  I think the good comparison that people encounter day-to-day is full disk encryption. It's the default on macOS, the only option on iOS, (those are the two platforms I use), and I assume the case on windows and android. The thing is FDE essentially only protects your data when your machine is powered off. Once your machine is booted and you've logged in any block level encryption ceases to be relevant, because to get to the point of running your machine has to have loaded in the relevant key material to decrypt. From that point on user space code no longer sees a difference between encrypted and decrypted drives. In other words FDE is not relevant is you lose a powered on device (post login if relevant to the platform), and you're the kind of person people are actively targeting (I recall recently? the content of someone's phone or such being dumped by the FBI because they grabbed it while it was being used). That's why modern OS's have different key classes, there's the lowest level which is just FDE, but you can have higher levels where requesting key material essentially just gives you a handle to that material. Then the OS, or preferably hardware with a much less complex OS, manages those handles and invalidates them according to policy rules. e.g you may want your phone to have access to your address book while your phone is locked, which does not mean you need your call history available as well. The policies provided by OSs tend to be fairly simple because it's better to have an easy to understand API that is easy to use and hard to screw up than a more "powerful" API that is easy to screw up and hard to use (the latter resulting in people simply not encrypting things at all). e.g iOS/macOS only has the following file protections when you create files: "NSFileProtectionComplete", "NSFileProtectionCompleteUnlessOpen", "NSFileProtectionCompleteUntilFirstUserAuthentication", "NSFileProtectionNone", but they're very easy to understand.[1] I tried to find the android equivalent but I don't know the terminology that's used and I just get linked to instructions on using AES, so if someone could link the correct doc I'd appreciate it. [1] https://support.apple.com/guide/security/data-protection-classes-secb010e978a/web https://support.apple.com/guide/security/data-protection-cla... and https://support.apple.com/guide/security/keychain-data-protection-secb0694df1a/web https://support.apple.com/guide/security/keychain-data-prote...

antegamisou 4y ago

Wasn't it them that had a bug that exposed users' passwords in plain text a few years back?

sytringy05 4y ago
I think they were logging them in clear text, so server side, prolly ended up in splunk or elasticsearch.

sylens 4y ago

I think it's also important to recognize how much of a "check the box" security control encryption at rest has become for many vendors/GRC teams. A lot of times, the encryption at rest control only has the capability to prevent somebody from physically detaching the disk and trying to mount it with their own machine and access the data that way. In a world where many companies now run their workloads on public cloud providers who keep their hardware in distributed cages in secure datacenters, this isn't the security control many assume it is. If you're trying to prevent an actor who has gained a foothold on a box/network from seeing plaintext data that is actually in use by the actual production system at that very moment, you're looking for a much stronger type of control - probably some sort of client-side encryption or obfuscation/tokenization

physicles 4y ago
Finally, someone points out the emperor has no clothes. When customers first started requesting encryption at rest, it didn’t make any sense to me — the threats it mitigates aren’t worth worrying about if you’re using public cloud. So it is just a checkbox then.

zionic 4y ago

I suspect this will be the norm going forward. Big tech was taken over by bean counters long ago, the fact that it’s all running on duct tape and popsicle sticks under the hood will come back to bite us when we have a digital Pearl Harbor event. China will invade Taiwan and the first shot won’t be physical, it will be activating the 30 years of assets they grew in AWS/GCP/cloudfare/level3/AT&T/Etc Most of their HR/engineering departments are completely retarded. They’ll hire any H1B who passes l33t code that accepts $50k under market rate then give them repo access in a few weeks. Our soulless megacorps are beyond easy to penetrate by hostile intelligence. The CIA/NSA/FBI, you know the groups who we pay billions per year for and they take half my income to fund will of course not catch any of this. The FBI is too busy manufacturing domestic terrorist, the NSA is too busy hacking American companies, and the CIA is too busy importing drugs to actually secure our country from foreign attack. Why? Because it’s been so long since we were actually attacked they believe it can’t happen so why not loot Rome in the mean time?