7 ms·

Okta’s Investigation of the January 2022 Compromise

282 points by photon12 5y ago

angryGhost 5y ago
oh no, my company just rolled out Okta too...
JohnCClarke 5y ago
The first 71 minutes of the response timeline are exemplary. But then two gaps emerge: 1. (Bad) Why did it take an entire day to notify the contractor? 2. (Much, much worse) Why is there no mention of any investigation into all the historical actions taken by that support agent?
kyleee 5y ago
david bradbury will probably fail upwards, must be nice
- bobnamob 5y ago
  https://en.wiktionary.org/wiki/do_a_Bradbury https://en.wiktionary.org/wiki/do_a_Bradbury
  - disillusioned 5y ago
    That is deeply ironic and also hilarious in this context.
  - hendiatris 5y ago
    Here’s video: https://youtu.be/fAADWfJO2qM https://youtu.be/fAADWfJO2qM
ZeroCool2u 5y ago
"Although that individual attempt was unsuccessful, out of an abundance of caution, we reset the account and notified Sitel who engaged a leading forensic firm to perform an investigation." Really don't like that they're still unwilling to actually say the name of the 'leading forensic firm'.
- shbooms 5y ago
  All while happily throwing Sitel under the bus. This whole communication from Okta is a total debacle. They are trying desperately to under play the event and deflect blame in the poor transparency after the event all while not once mentioning that they plan on making any remediations or changing anything in their day to day operations after this incident to prevent it from happening again.
bytelines 5y ago
> This is an application built with least privilege in mind Uh huh, makes sense > Named SuperUser Uhh... It lists all the operations that it can't do, but not what it can do. Can they download a private SAML certificate? Can they impersonate a user? Can they configure SSO and MFA settings? Can they download audit logs?
- tgsovlerkhgsel 5y ago
  > Can they download a private SAML certificate? Oh, that's a good one. Definitely something that the software should not allow, because I can't see a legitimate reason for this (allowing to download the certificate is fine, but not the key).
  - bostik 5y ago
    This was my topmost question too. The report very cleanly omits any and all mentions of SAML signing certificates. Solar Winds was the first known incident to escalate to so called "Golden SAML" attack. If the support staff had access to signing certificates, then that would open the door to a wide-scale exploitation of Okta's clients. A shower of Golden SAMLs, if you like.
    - zaroth 5y ago
      Caution to fellow readers: Put down your drink before reading the last line of this post.
oefrha 5y ago
> The majority of support engineering tasks are performed using an internally-built application called SuperUser or SU for short, which is used to perform basic management functions of Okta customer tenants. Pretty ominous name. I wouldn’t hand out “super user” accounts to support engineers from contracting firms for “basic duties in handling inbound support queries”.
- rstat1 5y ago
  Unless there's some amount of "lying-by-omission" going on here (or some amount of me not understanding what was said), it doesn't sound like support staff get actual super user access to me. It just sounds like a poorly named internal app.
  - oefrha 5y ago
    That’s what I meant by ominous.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
qbasic_forever 5y ago
I don't understand how the CSO can write this: "In this post, I want to provide a timeline and my perspective on what has transpired, and where we are today with this investigation. I hope that it will illuminate why I am confident in our conclusions that the Okta service has not been breached and there are no corrective actions that need to be taken by our customers." And then go on to write paragraphs of detail and a timeline that explicitly shows for a five day period an unauthorized user had full super user access to the service. He explicitly says, "The report from the forensic firm highlighted that there was a five-day window of time between January 16-21, 2022 when the threat actor had access to the Sitel environment, which we validated with our own analysis." This is some unbelievable double speak to try to claim that Okta was not breached. Any trust I had in this company is completely in the toilet, just based on this response. How is the CEO not responding to this too? The hole just keeps getting dug deeper the more they say.
- Buttons840 5y ago
  Sounds like the main point of disagreement is the definition of a single word, "breached". Otherwise, everyone is in agreement about what happened, right? I agree with your definition of the word for whatever it's worth. At this point, it would be helpful for Okta to stop using the term "breached", because apparently they aren't using the word in the same way everyone else is, and it's a point of contention.
  - Closi 5y ago
    Everyone else is using the plain English definition because it’s accurate. Okta is intentionally using their own definition to down-play what’s happened. A breach is simply ‘to overcome defences’ (ie to get access to something you shouldn’t). In this case their defence against someone else accessing the super user application was the support employee and their credentials, but this defence was clearly overcome by the hackers. I agree that they need to stop using the word.
  - kerng 5y ago
    Agreed. It's obvious that lawyers are deeply involved already at this point. The most likely reason they dont wanna use the common sense term "breached" is because it implies legally a breach of contract, which means liability and getting sued and having to show up in front of congress. Very sad to see their response be so intransparent and flawed. I wish technical people would be more involved in writing these responses, not lawyers.
the_duke 5y ago
What a weird and potentially misleading statement. * They stress that the compromised account wasn't able to "create/delete users or download customer databases", but not what it could do. Could it change passwords of accounts and add 2fa methods, allowing them to take over rarely/never accessed users? Disable 2fa? Change account permissions? List user accounts and metadata to build a user account DB for further attacks? The application is named "SuperUser" ... * It took public posting of a screenshot to trigger an audit of access logs, two months after the compromise was detected! * "Only" 2.5% of customers were accessed. That's supposed to be a good thing? Those were certainly the most valuable targets. * Concludes everything is just fine and no corrective actions need to be taken, but affected customers might want to do their own analysis... Huh? Sounds a lot like damage control.
- Cthulhu_ 5y ago
  > Sounds a lot like damage control. Yup, the first line made my hackles stand up; surely there's no security incidents raised for a user-invoked operation like adding MFA to a subcontractor's account? Audit logging is fine, but that's a user-initiated operation that already requires (if they have their shit in order) username, password, existing MFA if applicable, and an e-mail confirmation. To the user. It's all by the user. Does Github raise the alarms if I change something in my MFA settings?
  - throw1230 5y ago
    It looks like it's got flagged because the key was added from a new location
  - ensignavenger 5y ago
    The MFA tokens you use on Github are provided by you. The MFA tokens used by Okta employees/contractors may be provided by Okta, therefore Okta may know the token that was attempted to be added was not provided by them, and that is why it may have been blocked and a security alert generated. GitHub can't do that unless they are the only ones providing the tokens to their users.
tgsovlerkhgsel 5y ago
It's interesting how you need to read what they don't write to actually figure out what they're saying: They are unable to create or delete users. They cannot download customer databases. They cannot access our source code repositories. So they can likely arbitrarily access/impersonate or at least password reset existing users, and probably reconfigure the accounts in creative ways. For transparency, these customers will receive a report that shows the actions performed on their Okta tenant by Sitel during that period of time. We think this is the best way to let customers assess the situation for themselves. So they have no idea which of the actions were or weren't legitimate and make it their customers' problem.
- numbsafari 5y ago
  At least they are finally notifying.
PixelPaul 5y ago
What the best free place to subscribe to, to get notified of hacks like this? Some place that is quick at getting them added/listed and notifying people. As I often hear about it in the news first which is day+ after it’s released and not soon enough
- sofixa 5y ago
  Depends on the type of hack. Haveibeenpwned is a classic, but i don't think it covers cases like this, it's more for username/password/PII breaches.
- angryGhost 5y ago
  I'd say twitter perhaps but you have to follow the right sources
oxfordmale 5y ago
As communicated in stern words to Okta, my company unnecessarily spend many people hours on this. IT had to investigate if we were impacted by this, and on top of that issued a password reset for the entire company. A swift communication by Okta could have avoided this all together. It seems they care more about their shareholders than their customers.
- EMIRELADERO 5y ago
  > It seems they care more about their shareholders than their customers. isn't this how publicly-traded companies are supposed to work? I agree on critizicing that approach and capitalism model, but I don't understand how that isn't common knowledge here.
  - philjohn 5y ago
    It could be argued that if you don't care about your customers, you won't be in business long enough to please your shareholders.
  - oxfordmale 5y ago
    Of course the sole purpose of a publicly traded company is to maximise the revenue for its shareholders, however, you can take a long term or short term approach on this. Okta appeared to have kept this under wraps to prevent a shareholders backlash (short term approach) However, as a result they achieved the opposite, as the share price is still down this morning. This may of course be a temporarily glitch, however, I can see it resulting in a temporary loss of revenue. If I would be evaluating Okta versus a different solution right now, this may well sway my decision.
    - JshWright 5y ago
      I suspect there are plenty of current customers that are considering a change, not just prospects.
    - vel0city 5y ago
      It doesn't have to be. If the shareholders want the company to otherwise cease operations and throw a big ice cream party for all the shareholders every Friday they can choose to do that. There's nothing that forces a public company to focus only on maximizing shareholder value, they just have to be open and honest about the goals of the company and try and meet the shareholder expectations.
stavros 5y ago
Oh man, it sucks that they were hacked, but at least they weren't hacked.
philjohn 5y ago
"were taken from a Sitel support engineer’s computer upon which an attacker had obtained remote access using RDP. This device was owned and managed by Sitel. The scenario here is analogous to walking away from your computer at a coffee shop" It really is not analogous at all. That RDP was enabled, let alone could be accessed from outside the network is worrying. To me this would appear that, to save a buck, they outsource a lot of functions that then meant customer security was partly out of their hands, and relied upon another company having their security ducks in a row. I'm sure in their marketing materials they boast about state-of-the-art security, but that's only as good as the weakest point in the chain.
- athenot 5y ago
  Maybe they meant: The scenario here is analogous to walking away from your computer at a coffee shop... with your computer unlocked and logged in to various things
  - philjohn 5y ago
    We can extend that - "after we've promised our customers that a computer would never be left unlocked and logged in to sensitive things in a public place, but would instead be behind multiple locked doors." It's really not the "see, this is something you might do! It's not so bad!" out they thought it would be.
nstart 5y ago
The entire message has a tone of being entirely true but not representative of the entire truth. And that just leaves us all hanging with more questions because it doesn't tell us what we really need to know. 2.5% of all customers were accessed by all Sintel employees for the period in question. How many customers did the particular affected Sintel employee access? They assessed all the actions that took place by the Sintel employees. Were any of them sensitive? The tool doesn't allow them to create or delete users. Does it allow them to modify users so that an attacker can take over an account? The attacker had access to "Jira, Slack, Splunk, RingCentral, and support tickets through Salesforce". Doesn't sound bad on its own but have those tools been evaluated to ensure they don't have sensitive information? Do we know what the Sintel employee in question accessed from each of those tools for the time period? The employee's machine was logged into using an RDP session. Was the employee's machine assessed to ensure they weren't storing other sensitive information? While not allowed, a lot of support engineers will quickly drop in "temporary" sensitive stuff into random notepads and what not. Was this assessed? Also, why does a support engineer have a machine that has the ability to enable an inward bound RDP session at all?? How was that not the first trigger since this is a known attack vector for support agents. Given the possibly small blast radius of this issue, Okta had all the opportunity to turn this into a communications win for them and just build a multiple of trust really quickly. All these half communications have utterly botched it and that's just really disappointing. We'll still continue to use them because the switching costs just don't justify moving off Okta. But Okta has really taken a hit in their "trust bank".
- dordoka 5y ago
  LAPSUS$ has stated that Okta had sensitive info in Slack, including AWS Keys (they also confirmed they had access to 8.6K channels).
  - tyingq 5y ago
    I see a Splunk icon in the app list also, which is often a good source of that sort of thing.
- staticelf 5y ago
HL33tibCe7 5y ago
After seeing this incident, and particularly Okta’s response to this, I will make it a personal mission to ensure that Okta is not used anywhere that I work.
taubek 5y ago
Is there a list of services that use Okta?
thematrixturtle 5y ago
> The sharing of these screenshots is embarrassing for myself and the whole Okta team. It speaks volumes that their embarrassment is so important that it was the second sentence of the whole investigation, while there is literally not a single word of apology to the actual customers who were compromised.
- Jxl180 5y ago
  They are individually reaching out to the actual customers who were compromised. You are making quite an assumption without seeing the individual emails.
  - thematrixturtle 5y ago
    I'm not assuming anything, I'm commenting on the public post here.
jbrownbridge 5y ago
So if I'm reading this right, Okta was aware of a "compromise" of one of their sub-processors that impacted an unknown number of their customers/end users. They then waited more than 2 months before performing their own rudimentary analysis of the audit log to see what actions that sub-processor may have taken during the "compromise". Their CSO writes, "Over the past 24 hours we have analyzed more than 125,000 log entries to ascertain what actions were performed by Sitel during the relevant period. We have determined that the maximum potential impact is 366 (approximately 2.5% of) customers whose Okta tenant was accessed by Sitel." IANAL and these are only my opinions but it seems like: (a) Their DPO chose not to notify (GDPR Art. 33) without having the full picture or thought waiting several months for sub-processor's report was a justifiable reason for delaying notification (b) They failed to perform their own basic forensic activities in light of a "compromise" and only reviewed logs on March 22nd (c) Have terrible taste in naming their support app "Super User" In my opinion they are also down playing the importance of the data that may have been compromised. For example do the hackers now know which accounts have MFAs attached to and which don't. What the password policies are (e.g. strength, number attempts, etc.)
- chippiewill 5y ago
  The compromised tenant looks like it was specifically _not_ one of the EMEA ones so GDPR wouldn't be relevant here.
  - EwanToo 5y ago
    If the tenant had 1 or more European employees in their system, then yes GDPR is likely relevant.
  - jbrownbridge 5y ago
    I think the issue is that they just wouldn’t know. They didn’t know which customers were impacted. They didn’t know which users personal data might have been compromised. They most likely don’t have the ability to determine whether a user is a EU resident or not as this information would reside with their customers HR systems which all points to having to notify to avoid the legal complications.
hericium 5y ago
David Bradbury, grow a backbone. You were breached and you did not notify your customers.
luciusdomitius 5y ago
I don't understand how OKTA is +4.34%/1M and 10.26%/5D. This is bat-shit crazy lol. Anyway I put a 10,000USD 5x sell at 166.19, let's see how it goes from here :D :D :D
- luciusdomitius 5y ago
  hahha. downvote at will. I just made $2k with literally twenty mouse clicks.
  - __app_dev__ 5y ago
    Congrats! If you bought PUTS at market open today (or yesterday) you could have made 500% today. I'm excited about this and in the future will be checking stock price first thing when I see these smaller publicly listed tech companies getting hit with big hacks. For MSFT (and companies of that size) this type of news doesn't move the stock that much but with current volatility there will be plenty of great plays in the future for smaller companies.
Traubenfuchs 5y ago
tl;dr - Companies cheerfully handed over their golden skeleton and city (company) keys over to a third party service provider that offers SSO and now, after they got hacked by some kiddy that writes like a teenager, we found out that they also just cook with water (or less: api keys in slack). Now the public company communication is a reputation crushing web of lies, inaccuracies and whining.
- luciusdomitius 5y ago
  The stock hasn't crashed yet. I see opportunity.
  - __app_dev__ 5y ago
    Could have made 500% on PUTS today alone though.
dang 5y ago
Recent and related: New Updated Okta Statement on Lapsus$ - https://news.ycombinator.com/item?id=30774193 https://news.ycombinator.com/item?id=30774193 - March 2022 (24 comments) Updated Okta Statement on Lapsus$ - https://news.ycombinator.com/item?id=30769537 https://news.ycombinator.com/item?id=30769537 - March 2022 (220 comments) Also: DEV-0537 (LAPSUS$) Criminal actor targeting organizations - https://news.ycombinator.com/item?id=30774406 https://news.ycombinator.com/item?id=30774406 - March 2022 (0 comments) Lapsus$ hackers leak 37GB of Microsoft's alleged source code - https://news.ycombinator.com/item?id=30763623 https://news.ycombinator.com/item?id=30763623 - March 2022 (117 comments)
deleted 5y ago
[deleted]
nijave 5y ago
I remember a couple years ago we were using Okta for AWS federated access. One day a bunch of role associations disappeared. Ended up badgering support for a couple weeks since the audit logs were empty before they finally admitted it was a "bad migration" Apparently they had some internal issue with orphaned records or something and deployed a code "fix" that ended up deleting legitimate records in a way that wasn't auditable from the customer side. I think it took about 4-6 weeks of trying to escalate through support and account reps until we actually got an answer. It was also very surprising to see role associations just disappear without a trace (we associated Okta groups with AWS IAM roles in the Okta integration)
dfsegoat 5y ago
> "I am greatly disappointed by the long period of time that transpired between our notification...once WE received the Sitel summary report WE should have moved more swiftly to understand its implications." This is an example of total non-ownership. He is the CSO. It should be "I" or "My" and not diffusing responsibility onto his team with "We". In my book you should celebrate your successes as a team ("we", "our") but failures are ALWAYS on leaders ("I", "my").
deepzn 5y ago
Again, sounds very defensive and not honest. Like rather than say the account had least priveleges and did not have "God" level access, you should state what authorities they did have, for eg: resetting pws, mfa's, etc...
nkotov 5y ago
What a weird and dumb response. The fact that they keep updating their response will result in a backfire.
wizwit999 5y ago
I'm curious about Okta's customer facing audit logs. It seems they do offer audit logs [1] [2] but I can't find the documentation on what all events are included and if actions by support are (which they should). Google for example includes transparency events in it's audit log [3], CloudTrail also shows actions taken by AWS support. I've talked about this before [4] but just having internal audit logs doesn't cut it nowadays, even in this case it took two months to check the audit logs, you should give your customers access to their audit logs, ideally in near realtime so they can do proactive monitoring. [1] https://help.okta.com/en/prod/Content/Topics/Reports/Reports_SysLog.htm https://help.okta.com/en/prod/Content/Topics/Reports/Reports... [2] https://developer.okta.com/docs/reference/api/system-log/ https://developer.okta.com/docs/reference/api/system-log/ [3] https://developers.google.com/admin-sdk/reports/v1/appendix/activity/access-transparency https://developers.google.com/admin-sdk/reports/v1/appendix/... [4] https://apptrail.com/blog/2022/03/07/internal-vs-customer-facing-audit-logs https://apptrail.com/blog/2022/03/07/internal-vs-customer-fa...
- samcat116 5y ago
  Apparently support actions can be found with "user.session.impersonation" from what I read.
dbenhur 5y ago
"Prior to Okta, David was the Senior Vice President and Chief Security Officer at Symantec" So, his prior experience is as CSO at a company who's principal business is selling fake security products. Ho, boy.

282 points by photon12 5y ago

oh no, my company just rolled out Okta too...

The first 71 minutes of the response timeline are exemplary. But then two gaps emerge: 1. (Bad) Why did it take an entire day to notify the contractor? 2. (Much, much worse) Why is there no mention of any investigation into all the historical actions taken by that support agent?

kyleee 5y ago

david bradbury will probably fail upwards, must be nice

bobnamob 5y ago
https://en.wiktionary.org/wiki/do_a_Bradbury https://en.wiktionary.org/wiki/do_a_Bradbury
- disillusioned 5y ago
  That is deeply ironic and also hilarious in this context.
- hendiatris 5y ago
  Here’s video: https://youtu.be/fAADWfJO2qM https://youtu.be/fAADWfJO2qM

ZeroCool2u 5y ago

"Although that individual attempt was unsuccessful, out of an abundance of caution, we reset the account and notified Sitel who engaged a leading forensic firm to perform an investigation." Really don't like that they're still unwilling to actually say the name of the 'leading forensic firm'.

shbooms 5y ago
All while happily throwing Sitel under the bus. This whole communication from Okta is a total debacle. They are trying desperately to under play the event and deflect blame in the poor transparency after the event all while not once mentioning that they plan on making any remediations or changing anything in their day to day operations after this incident to prevent it from happening again.

bytelines 5y ago

> This is an application built with least privilege in mind Uh huh, makes sense > Named SuperUser Uhh... It lists all the operations that it can't do, but not what it can do. Can they download a private SAML certificate? Can they impersonate a user? Can they configure SSO and MFA settings? Can they download audit logs?

tgsovlerkhgsel 5y ago
> Can they download a private SAML certificate? Oh, that's a good one. Definitely something that the software should not allow, because I can't see a legitimate reason for this (allowing to download the certificate is fine, but not the key).
- bostik 5y ago
  This was my topmost question too. The report very cleanly omits any and all mentions of SAML signing certificates. Solar Winds was the first known incident to escalate to so called "Golden SAML" attack. If the support staff had access to signing certificates, then that would open the door to a wide-scale exploitation of Okta's clients. A shower of Golden SAMLs, if you like.
  - zaroth 5y ago
    Caution to fellow readers: Put down your drink before reading the last line of this post.

oefrha 5y ago

> The majority of support engineering tasks are performed using an internally-built application called SuperUser or SU for short, which is used to perform basic management functions of Okta customer tenants. Pretty ominous name. I wouldn’t hand out “super user” accounts to support engineers from contracting firms for “basic duties in handling inbound support queries”.

rstat1 5y ago
Unless there's some amount of "lying-by-omission" going on here (or some amount of me not understanding what was said), it doesn't sound like support staff get actual super user access to me. It just sounds like a poorly named internal app.
- oefrha 5y ago
  That’s what I meant by ominous.

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

qbasic_forever 5y ago

I don't understand how the CSO can write this: "In this post, I want to provide a timeline and my perspective on what has transpired, and where we are today with this investigation. I hope that it will illuminate why I am confident in our conclusions that the Okta service has not been breached and there are no corrective actions that need to be taken by our customers." And then go on to write paragraphs of detail and a timeline that explicitly shows for a five day period an unauthorized user had full super user access to the service. He explicitly says, "The report from the forensic firm highlighted that there was a five-day window of time between January 16-21, 2022 when the threat actor had access to the Sitel environment, which we validated with our own analysis." This is some unbelievable double speak to try to claim that Okta was not breached. Any trust I had in this company is completely in the toilet, just based on this response. How is the CEO not responding to this too? The hole just keeps getting dug deeper the more they say.

Buttons840 5y ago
Sounds like the main point of disagreement is the definition of a single word, "breached". Otherwise, everyone is in agreement about what happened, right? I agree with your definition of the word for whatever it's worth. At this point, it would be helpful for Okta to stop using the term "breached", because apparently they aren't using the word in the same way everyone else is, and it's a point of contention.
- Closi 5y ago
  Everyone else is using the plain English definition because it’s accurate. Okta is intentionally using their own definition to down-play what’s happened. A breach is simply ‘to overcome defences’ (ie to get access to something you shouldn’t). In this case their defence against someone else accessing the super user application was the support employee and their credentials, but this defence was clearly overcome by the hackers. I agree that they need to stop using the word.
- kerng 5y ago
  Agreed. It's obvious that lawyers are deeply involved already at this point. The most likely reason they dont wanna use the common sense term "breached" is because it implies legally a breach of contract, which means liability and getting sued and having to show up in front of congress. Very sad to see their response be so intransparent and flawed. I wish technical people would be more involved in writing these responses, not lawyers.

the_duke 5y ago

What a weird and potentially misleading statement. * They stress that the compromised account wasn't able to "create/delete users or download customer databases", but not what it could do. Could it change passwords of accounts and add 2fa methods, allowing them to take over rarely/never accessed users? Disable 2fa? Change account permissions? List user accounts and metadata to build a user account DB for further attacks? The application is named "SuperUser" ... * It took public posting of a screenshot to trigger an audit of access logs, two months after the compromise was detected! * "Only" 2.5% of customers were accessed. That's supposed to be a good thing? Those were certainly the most valuable targets. * Concludes everything is just fine and no corrective actions need to be taken, but affected customers might want to do their own analysis... Huh? Sounds a lot like damage control.

Cthulhu_ 5y ago
> Sounds a lot like damage control. Yup, the first line made my hackles stand up; surely there's no security incidents raised for a user-invoked operation like adding MFA to a subcontractor's account? Audit logging is fine, but that's a user-initiated operation that already requires (if they have their shit in order) username, password, existing MFA if applicable, and an e-mail confirmation. To the user. It's all by the user. Does Github raise the alarms if I change something in my MFA settings?
- throw1230 5y ago
  It looks like it's got flagged because the key was added from a new location
- ensignavenger 5y ago
  The MFA tokens you use on Github are provided by you. The MFA tokens used by Okta employees/contractors may be provided by Okta, therefore Okta may know the token that was attempted to be added was not provided by them, and that is why it may have been blocked and a security alert generated. GitHub can't do that unless they are the only ones providing the tokens to their users.

tgsovlerkhgsel 5y ago

It's interesting how you need to read what they don't write to actually figure out what they're saying: They are unable to create or delete users. They cannot download customer databases. They cannot access our source code repositories. So they can likely arbitrarily access/impersonate or at least password reset existing users, and probably reconfigure the accounts in creative ways. For transparency, these customers will receive a report that shows the actions performed on their Okta tenant by Sitel during that period of time. We think this is the best way to let customers assess the situation for themselves. So they have no idea which of the actions were or weren't legitimate and make it their customers' problem.

numbsafari 5y ago
At least they are finally notifying.

PixelPaul 5y ago

What the best free place to subscribe to, to get notified of hacks like this? Some place that is quick at getting them added/listed and notifying people. As I often hear about it in the news first which is day+ after it’s released and not soon enough

sofixa 5y ago
Depends on the type of hack. Haveibeenpwned is a classic, but i don't think it covers cases like this, it's more for username/password/PII breaches.
angryGhost 5y ago
I'd say twitter perhaps but you have to follow the right sources

oxfordmale 5y ago

As communicated in stern words to Okta, my company unnecessarily spend many people hours on this. IT had to investigate if we were impacted by this, and on top of that issued a password reset for the entire company. A swift communication by Okta could have avoided this all together. It seems they care more about their shareholders than their customers.

EMIRELADERO 5y ago
> It seems they care more about their shareholders than their customers. isn't this how publicly-traded companies are supposed to work? I agree on critizicing that approach and capitalism model, but I don't understand how that isn't common knowledge here.
- philjohn 5y ago
  It could be argued that if you don't care about your customers, you won't be in business long enough to please your shareholders.
- oxfordmale 5y ago
  Of course the sole purpose of a publicly traded company is to maximise the revenue for its shareholders, however, you can take a long term or short term approach on this. Okta appeared to have kept this under wraps to prevent a shareholders backlash (short term approach) However, as a result they achieved the opposite, as the share price is still down this morning. This may of course be a temporarily glitch, however, I can see it resulting in a temporary loss of revenue. If I would be evaluating Okta versus a different solution right now, this may well sway my decision.
  - JshWright 5y ago
    I suspect there are plenty of current customers that are considering a change, not just prospects.
  - vel0city 5y ago
    It doesn't have to be. If the shareholders want the company to otherwise cease operations and throw a big ice cream party for all the shareholders every Friday they can choose to do that. There's nothing that forces a public company to focus only on maximizing shareholder value, they just have to be open and honest about the goals of the company and try and meet the shareholder expectations.

stavros 5y ago

Oh man, it sucks that they were hacked, but at least they weren't hacked.

philjohn 5y ago

"were taken from a Sitel support engineer’s computer upon which an attacker had obtained remote access using RDP. This device was owned and managed by Sitel. The scenario here is analogous to walking away from your computer at a coffee shop" It really is not analogous at all. That RDP was enabled, let alone could be accessed from outside the network is worrying. To me this would appear that, to save a buck, they outsource a lot of functions that then meant customer security was partly out of their hands, and relied upon another company having their security ducks in a row. I'm sure in their marketing materials they boast about state-of-the-art security, but that's only as good as the weakest point in the chain.

athenot 5y ago
Maybe they meant: The scenario here is analogous to walking away from your computer at a coffee shop... with your computer unlocked and logged in to various things
- philjohn 5y ago
  We can extend that - "after we've promised our customers that a computer would never be left unlocked and logged in to sensitive things in a public place, but would instead be behind multiple locked doors." It's really not the "see, this is something you might do! It's not so bad!" out they thought it would be.

nstart 5y ago

The entire message has a tone of being entirely true but not representative of the entire truth. And that just leaves us all hanging with more questions because it doesn't tell us what we really need to know. 2.5% of all customers were accessed by all Sintel employees for the period in question. How many customers did the particular affected Sintel employee access? They assessed all the actions that took place by the Sintel employees. Were any of them sensitive? The tool doesn't allow them to create or delete users. Does it allow them to modify users so that an attacker can take over an account? The attacker had access to "Jira, Slack, Splunk, RingCentral, and support tickets through Salesforce". Doesn't sound bad on its own but have those tools been evaluated to ensure they don't have sensitive information? Do we know what the Sintel employee in question accessed from each of those tools for the time period? The employee's machine was logged into using an RDP session. Was the employee's machine assessed to ensure they weren't storing other sensitive information? While not allowed, a lot of support engineers will quickly drop in "temporary" sensitive stuff into random notepads and what not. Was this assessed? Also, why does a support engineer have a machine that has the ability to enable an inward bound RDP session at all?? How was that not the first trigger since this is a known attack vector for support agents. Given the possibly small blast radius of this issue, Okta had all the opportunity to turn this into a communications win for them and just build a multiple of trust really quickly. All these half communications have utterly botched it and that's just really disappointing. We'll still continue to use them because the switching costs just don't justify moving off Okta. But Okta has really taken a hit in their "trust bank".

dordoka 5y ago
LAPSUS$ has stated that Okta had sensitive info in Slack, including AWS Keys (they also confirmed they had access to 8.6K channels).
- tyingq 5y ago
  I see a Splunk icon in the app list also, which is often a good source of that sort of thing.
staticelf 5y ago

HL33tibCe7 5y ago

After seeing this incident, and particularly Okta’s response to this, I will make it a personal mission to ensure that Okta is not used anywhere that I work.

taubek 5y ago

Is there a list of services that use Okta?

thematrixturtle 5y ago

> The sharing of these screenshots is embarrassing for myself and the whole Okta team. It speaks volumes that their embarrassment is so important that it was the second sentence of the whole investigation, while there is literally not a single word of apology to the actual customers who were compromised.

Jxl180 5y ago
They are individually reaching out to the actual customers who were compromised. You are making quite an assumption without seeing the individual emails.
- thematrixturtle 5y ago
  I'm not assuming anything, I'm commenting on the public post here.

jbrownbridge 5y ago

So if I'm reading this right, Okta was aware of a "compromise" of one of their sub-processors that impacted an unknown number of their customers/end users. They then waited more than 2 months before performing their own rudimentary analysis of the audit log to see what actions that sub-processor may have taken during the "compromise". Their CSO writes, "Over the past 24 hours we have analyzed more than 125,000 log entries to ascertain what actions were performed by Sitel during the relevant period. We have determined that the maximum potential impact is 366 (approximately 2.5% of) customers whose Okta tenant was accessed by Sitel." IANAL and these are only my opinions but it seems like: (a) Their DPO chose not to notify (GDPR Art. 33) without having the full picture or thought waiting several months for sub-processor's report was a justifiable reason for delaying notification (b) They failed to perform their own basic forensic activities in light of a "compromise" and only reviewed logs on March 22nd (c) Have terrible taste in naming their support app "Super User" In my opinion they are also down playing the importance of the data that may have been compromised. For example do the hackers now know which accounts have MFAs attached to and which don't. What the password policies are (e.g. strength, number attempts, etc.)

chippiewill 5y ago
The compromised tenant looks like it was specifically _not_ one of the EMEA ones so GDPR wouldn't be relevant here.
- EwanToo 5y ago
  If the tenant had 1 or more European employees in their system, then yes GDPR is likely relevant.
- jbrownbridge 5y ago
  I think the issue is that they just wouldn’t know. They didn’t know which customers were impacted. They didn’t know which users personal data might have been compromised. They most likely don’t have the ability to determine whether a user is a EU resident or not as this information would reside with their customers HR systems which all points to having to notify to avoid the legal complications.

hericium 5y ago

David Bradbury, grow a backbone. You were breached and you did not notify your customers.

luciusdomitius 5y ago

I don't understand how OKTA is +4.34%/1M and 10.26%/5D. This is bat-shit crazy lol. Anyway I put a 10,000USD 5x sell at 166.19, let's see how it goes from here :D :D :D

luciusdomitius 5y ago
hahha. downvote at will. I just made $2k with literally twenty mouse clicks.
- __app_dev__ 5y ago
  Congrats! If you bought PUTS at market open today (or yesterday) you could have made 500% today. I'm excited about this and in the future will be checking stock price first thing when I see these smaller publicly listed tech companies getting hit with big hacks. For MSFT (and companies of that size) this type of news doesn't move the stock that much but with current volatility there will be plenty of great plays in the future for smaller companies.

Traubenfuchs 5y ago

tl;dr - Companies cheerfully handed over their golden skeleton and city (company) keys over to a third party service provider that offers SSO and now, after they got hacked by some kiddy that writes like a teenager, we found out that they also just cook with water (or less: api keys in slack). Now the public company communication is a reputation crushing web of lies, inaccuracies and whining.

luciusdomitius 5y ago
The stock hasn't crashed yet. I see opportunity.
- __app_dev__ 5y ago
  Could have made 500% on PUTS today alone though.

dang 5y ago

Recent and related: New Updated Okta Statement on Lapsus$ - https://news.ycombinator.com/item?id=30774193 https://news.ycombinator.com/item?id=30774193 - March 2022 (24 comments) Updated Okta Statement on Lapsus$ - https://news.ycombinator.com/item?id=30769537 https://news.ycombinator.com/item?id=30769537 - March 2022 (220 comments) Also: DEV-0537 (LAPSUS$) Criminal actor targeting organizations - https://news.ycombinator.com/item?id=30774406 https://news.ycombinator.com/item?id=30774406 - March 2022 (0 comments) Lapsus$ hackers leak 37GB of Microsoft's alleged source code - https://news.ycombinator.com/item?id=30763623 https://news.ycombinator.com/item?id=30763623 - March 2022 (117 comments)

deleted 5y ago

[deleted]

nijave 5y ago

I remember a couple years ago we were using Okta for AWS federated access. One day a bunch of role associations disappeared. Ended up badgering support for a couple weeks since the audit logs were empty before they finally admitted it was a "bad migration" Apparently they had some internal issue with orphaned records or something and deployed a code "fix" that ended up deleting legitimate records in a way that wasn't auditable from the customer side. I think it took about 4-6 weeks of trying to escalate through support and account reps until we actually got an answer. It was also very surprising to see role associations just disappear without a trace (we associated Okta groups with AWS IAM roles in the Okta integration)

dfsegoat 5y ago

> "I am greatly disappointed by the long period of time that transpired between our notification...once WE received the Sitel summary report WE should have moved more swiftly to understand its implications." This is an example of total non-ownership. He is the CSO. It should be "I" or "My" and not diffusing responsibility onto his team with "We". In my book you should celebrate your successes as a team ("we", "our") but failures are ALWAYS on leaders ("I", "my").

deepzn 5y ago

Again, sounds very defensive and not honest. Like rather than say the account had least priveleges and did not have "God" level access, you should state what authorities they did have, for eg: resetting pws, mfa's, etc...

nkotov 5y ago

What a weird and dumb response. The fact that they keep updating their response will result in a backfire.

wizwit999 5y ago

I'm curious about Okta's customer facing audit logs. It seems they do offer audit logs [1] [2] but I can't find the documentation on what all events are included and if actions by support are (which they should). Google for example includes transparency events in it's audit log [3], CloudTrail also shows actions taken by AWS support. I've talked about this before [4] but just having internal audit logs doesn't cut it nowadays, even in this case it took two months to check the audit logs, you should give your customers access to their audit logs, ideally in near realtime so they can do proactive monitoring. [1] https://help.okta.com/en/prod/Content/Topics/Reports/Reports_SysLog.htm https://help.okta.com/en/prod/Content/Topics/Reports/Reports... [2] https://developer.okta.com/docs/reference/api/system-log/ https://developer.okta.com/docs/reference/api/system-log/ [3] https://developers.google.com/admin-sdk/reports/v1/appendix/activity/access-transparency https://developers.google.com/admin-sdk/reports/v1/appendix/... [4] https://apptrail.com/blog/2022/03/07/internal-vs-customer-facing-audit-logs https://apptrail.com/blog/2022/03/07/internal-vs-customer-fa...

samcat116 5y ago
Apparently support actions can be found with "user.session.impersonation" from what I read.

dbenhur 5y ago

"Prior to Okta, David was the Senior Vice President and Chief Security Officer at Symantec" So, his prior experience is as CSO at a company who's principal business is selling fake security products. Ho, boy.