9 ms·

> When we learned about this, we immediately investigated and fixed it. At that time, we had no evidence to suggest someone had taken advantage of the vulnerabi

by bink 4y ago

> When we learned about this, we immediately investigated and fixed it. At that time, we had no evidence to suggest someone had taken advantage of the vulnerability. > In July 2022, we learned through a press report that someone had potentially leveraged this and was offering to sell the information they had compiled. After reviewing a sample of the available data for sale, we confirmed that a bad actor had taken advantage of the issue before it was addressed. Yikes. Sounds like they either didn't dig deep enough to see if it was exploited or they don't keep records long enough to be sure.

londons_explore 4y ago
Probably the latter - all companies operating in the EU have had short (ie. 30 days) retention policies on anything user-identifiable (ie. http logs) for a while now. But if they didn't keep sufficient logs, they should have alerted the users back then, not now.
- bink 4y ago
  AFAIK there is an exception for security purposes. They could be hashing or "anonymizing" the IPs and keep the data longer.
  - tick_tock_tick 4y ago
    No that's not valid at all! You must remove any trace of your ability to backwards engineering the IPs. Hashing isn't sufficient since it's so easy to run over the whole IPv4 space. This is one of the trade offs.
    - zaarn 4y ago
      You can if the hash collides within the IPv4 address space; ie it's a hash of less than about 16 bits. Enough to let your roughly see if something fishy is going on but you can't reverse engineer to any specific IP, only a set of 64 thousand.
      - tick_tock_tick 4y ago
        That isn't good enough. By taking that hash and old request data combined with your current request logs it's enough to de-anonymization a significant portion of those logs making you not in compliance.
        zaarn 4y ago
        Data from the past few days we do have a legitimate interest in; protecting our network. If someone is spamming us we need to be able to find out who did it and the only way to do that is deanonymized logs to begin with. Atleast in my workplace we have worked with the DPA to ensure that we are in compliance and there is no issue in keeping around 7 days of IP logs without further anonymization. All or long term logs are hashed below the bit minimum, and that can't be paired with old request data as easily since we strip all but major version identifiers from User Agents, for example.
    - heurisko 4y ago
      Wouldn't salting mitigate this?
      - 0x457 4y ago
        If something uniquely identifies someone, it's considered a PII and a salted (but still useful) hash of the IP address is that. At least under GDPR. That means you will need to throw away the salt and have different salt for every instance. At that point, you might as well replace with a random string, and that isn't very useful. "In the context of the European GDPR the Article 29 Working Party has stated that while the technique of salting and then hashing data “reduce[s] the likelihood of deriving the input value,” because “calculating the original attribute value hidden behind the result of a salted hash function may still be feasible within reasonable means,” the salted-hashed output should be considered pseudonymized data that remains subject to the GDPR." Under CCPA, I think that is enough, HOWEVER, business must implement business processes that specifically prohibit reidentification. So again, not useful at all in this case. The question should be is IP address a PII or not. Under CCPA and GDPR it is, but only if it “identifies, relates to, describes, is reasonably capable of being associated with, or could reasonably be linked, directly or indirectly, with a particular consumer or household.”
    - 0x0 4y ago
      You could probably make the argument that you need to store http logs with cleartext IP addresses for more than 30 days for operational security and fraud detection reasons. I would certainly consider 180+ days of cleartext IP addresses quite necessary to be able to react to any security or abuse incidents.
  - deleted 4y ago
    [deleted]
  - AtNightWeCode 4y ago
    For security reasons IP addresses needs to be available in plain text. There is no time limit for how long time you can store the data, but you need to be able to motivate why.
- grogenaut 4y ago
  AFAIK there are exceptions for many purposes, taxes, law enforcement, "critical business functions", etc of the 30 day window. Tax records, which can be quite PII and personal, need to be kept for ~7 years in the US for instance. Anything that needs to go to law enforcement stays around until the court case is over which can be longer.
addingnumbers 4y ago
"We have no evidence that this was exploited" is a standard psychological trick they pull in vulnerability announcements to give an unfounded impression that it hasn't been exploited.
- hunter2_ 4y ago
  I always wonder who "we" refers to in that usage, legally speaking. Does it refer only to a subset of employees / board members who are authorized to speak for the company? Because then even if someone analyzing logs sees something damning, if middle management is trained to stop that knowledge from reaching the top, then those speaking for the company can continue saying "we" didn't know it.
  - lrvick 4y ago
    I have 100% seen this happen.
    - johndhi 4y ago
      really? what do you mean 'middle management is trained to keep that from getting to the top'? intentional malfeasance? where I work people are trying their best but dealing with complex systems, memories, and methods of communication. because of this, security issues are sometimes missed, sometimes poorly communicated, and sometimes poorly remediated.
      - hobs 4y ago
        Almost all companies operate with an extremely low level of trust and most places are blame, shame, and ultimately game the system all the way down. Hiding something often takes years to uncover and by then management has moved on, maybe even to their second company!
      - t-3 4y ago
        Probably not 'trained' as much as 'heavily incentivized'. Nobody wants to be the messenger that gets shot for bringing bad news. Much easier to cover up and tell the big boss what they want to hear as long as you can.
nolok 4y ago
You seem to believe "we had no evidence to suggest someone had taken advantage of the vulnerability" implies "we looked for any evidence of it", it doesn't, not in that case nor in any similar situation.
grogenaut 4y ago
Or they don't monitor that system for that type of access at all and so literally don't know.
AvImd 4y ago
https://astralcodexten.substack.com/p/the-phrase-no-evidence-is-a-red-flag https://astralcodexten.substack.com/p/the-phrase-no-evidence...
- brian_cloutier 4y ago
  This link is not particularly relevant, as it talks about how the phrase "no evidence" is used within a specific community and that community has little overlap with the community which writes press releases after security incidents. Security incident response teams do not have the same strange distinction between "real" evidence and the non-published non-peer-reviewed evidence which cannot be relied on or even really mentioned.
hertzrat 4y ago
Out of curiosity, why is it only 5M and not 500M? You would think the same vulnerability applied to every server, not just one or one cluster, if they are using automated deployments
- uncletammy 4y ago
  Could be a time intensive exploit. Maybe they didn't have enough time to mine the other 455M.
- alex_young 4y ago
  The rest are bot accounts right?
- LegitShady 4y ago
  Doing it slowly over time to not raise an alarm and collect the information, rather than twitter noticing a massive upticks in password resets that don’t go through?

by bink 4y ago

londons_explore 4y ago

Probably the latter - all companies operating in the EU have had short (ie. 30 days) retention policies on anything user-identifiable (ie. http logs) for a while now. But if they didn't keep sufficient logs, they should have alerted the users back then, not now.

bink 4y ago
AFAIK there is an exception for security purposes. They could be hashing or "anonymizing" the IPs and keep the data longer.
- tick_tock_tick 4y ago
  No that's not valid at all! You must remove any trace of your ability to backwards engineering the IPs. Hashing isn't sufficient since it's so easy to run over the whole IPv4 space. This is one of the trade offs.
  - zaarn 4y ago
    You can if the hash collides within the IPv4 address space; ie it's a hash of less than about 16 bits. Enough to let your roughly see if something fishy is going on but you can't reverse engineer to any specific IP, only a set of 64 thousand.
    - tick_tock_tick 4y ago
      That isn't good enough. By taking that hash and old request data combined with your current request logs it's enough to de-anonymization a significant portion of those logs making you not in compliance.
      - zaarn 4y ago
        Data from the past few days we do have a legitimate interest in; protecting our network. If someone is spamming us we need to be able to find out who did it and the only way to do that is deanonymized logs to begin with. Atleast in my workplace we have worked with the DPA to ensure that we are in compliance and there is no issue in keeping around 7 days of IP logs without further anonymization. All or long term logs are hashed below the bit minimum, and that can't be paired with old request data as easily since we strip all but major version identifiers from User Agents, for example.
  - heurisko 4y ago
    Wouldn't salting mitigate this?
    - 0x457 4y ago
      If something uniquely identifies someone, it's considered a PII and a salted (but still useful) hash of the IP address is that. At least under GDPR. That means you will need to throw away the salt and have different salt for every instance. At that point, you might as well replace with a random string, and that isn't very useful. "In the context of the European GDPR the Article 29 Working Party has stated that while the technique of salting and then hashing data “reduce[s] the likelihood of deriving the input value,” because “calculating the original attribute value hidden behind the result of a salted hash function may still be feasible within reasonable means,” the salted-hashed output should be considered pseudonymized data that remains subject to the GDPR." Under CCPA, I think that is enough, HOWEVER, business must implement business processes that specifically prohibit reidentification. So again, not useful at all in this case. The question should be is IP address a PII or not. Under CCPA and GDPR it is, but only if it “identifies, relates to, describes, is reasonably capable of being associated with, or could reasonably be linked, directly or indirectly, with a particular consumer or household.”
  - 0x0 4y ago
    You could probably make the argument that you need to store http logs with cleartext IP addresses for more than 30 days for operational security and fraud detection reasons. I would certainly consider 180+ days of cleartext IP addresses quite necessary to be able to react to any security or abuse incidents.
- deleted 4y ago
  [deleted]
- AtNightWeCode 4y ago
  For security reasons IP addresses needs to be available in plain text. There is no time limit for how long time you can store the data, but you need to be able to motivate why.
grogenaut 4y ago
AFAIK there are exceptions for many purposes, taxes, law enforcement, "critical business functions", etc of the 30 day window. Tax records, which can be quite PII and personal, need to be kept for ~7 years in the US for instance. Anything that needs to go to law enforcement stays around until the court case is over which can be longer.

addingnumbers 4y ago

"We have no evidence that this was exploited" is a standard psychological trick they pull in vulnerability announcements to give an unfounded impression that it hasn't been exploited.

hunter2_ 4y ago
I always wonder who "we" refers to in that usage, legally speaking. Does it refer only to a subset of employees / board members who are authorized to speak for the company? Because then even if someone analyzing logs sees something damning, if middle management is trained to stop that knowledge from reaching the top, then those speaking for the company can continue saying "we" didn't know it.
- lrvick 4y ago
  I have 100% seen this happen.
  - johndhi 4y ago
    really? what do you mean 'middle management is trained to keep that from getting to the top'? intentional malfeasance? where I work people are trying their best but dealing with complex systems, memories, and methods of communication. because of this, security issues are sometimes missed, sometimes poorly communicated, and sometimes poorly remediated.
    - hobs 4y ago
      Almost all companies operate with an extremely low level of trust and most places are blame, shame, and ultimately game the system all the way down. Hiding something often takes years to uncover and by then management has moved on, maybe even to their second company!
    - t-3 4y ago
      Probably not 'trained' as much as 'heavily incentivized'. Nobody wants to be the messenger that gets shot for bringing bad news. Much easier to cover up and tell the big boss what they want to hear as long as you can.

nolok 4y ago

You seem to believe "we had no evidence to suggest someone had taken advantage of the vulnerability" implies "we looked for any evidence of it", it doesn't, not in that case nor in any similar situation.

grogenaut 4y ago

Or they don't monitor that system for that type of access at all and so literally don't know.

AvImd 4y ago

https://astralcodexten.substack.com/p/the-phrase-no-evidence-is-a-red-flag https://astralcodexten.substack.com/p/the-phrase-no-evidence...

brian_cloutier 4y ago
This link is not particularly relevant, as it talks about how the phrase "no evidence" is used within a specific community and that community has little overlap with the community which writes press releases after security incidents. Security incident response teams do not have the same strange distinction between "real" evidence and the non-published non-peer-reviewed evidence which cannot be relied on or even really mentioned.

hertzrat 4y ago

Out of curiosity, why is it only 5M and not 500M? You would think the same vulnerability applied to every server, not just one or one cluster, if they are using automated deployments

uncletammy 4y ago
Could be a time intensive exploit. Maybe they didn't have enough time to mine the other 455M.
alex_young 4y ago
The rest are bot accounts right?
LegitShady 4y ago
Doing it slowly over time to not raise an alarm and collect the information, rather than twitter noticing a massive upticks in password resets that don’t go through?