8 ms·

The screenshots show the account of someone who has a LinkedIn profile identifying themselves as a Tier 2 Technical Services Engineer [1] working for an outsour

by dhx 5y ago

The screenshots show the account of someone who has a LinkedIn profile identifying themselves as a Tier 2 Technical Services Engineer [1] working for an outsourced services provider in Costa Rica. They've been in the job for just over a year according to LinkedIn. This person states they provide troubleshooting assistance to Okta customers, and therefore may have needed elevated permissions over a number of Okta internal systems. This raises a number of architectural questions for Okta: - Can a support engineer for customers in one region access the systems/databases relied upon by customers in other regions? - Do employees get notified when their account is logged into or used (e-mail/SMS/otherwise)? - Do customers get notified when a support engineer accesses (with privileges) a system/database the customer relies upon? [1] https://work180.com/en-US/for-women/job/215617/tier-2-technical-support-engineer-remote-elig https://work180.com/en-US/for-women/job/215617/tier-2-techni...

toyg 5y ago
> Can a support engineer for customers in one region access the systems/databases relied upon by customers in other regions? My (admittedly outdated) experience is that this is always the case, regardless of any assurance vendors might have provided to the customer. In the end, people have to get stuff done, and offshore bodies are just too cheap to pass on. Things might have changed a bit since GDPR but, in practice, I expect there will always be "channels" for people to reach out cross-region.
- donalhunt 5y ago
  Also "timezones". Just because a customer is based in Europe doesn't mean they don't need to get a hold of support at 03:00 local time. Even small, growing, startups now have HQs in one part of the world and staff across the globe due to the ease of movement many enjoy.
draw_down 5y ago
“Do they have access” is one thing, but what happens when they use the access? Is there any logging or proactive monitoring? A single support account reading say 10x the normal number of accounts per day, with a different geographic distribution than the norm, should ring alarm bells. It’s not enough to say “well shucks they have access to the data, guess we’ll do nothing”
- cupofpython 5y ago
  I agree with you, put support agents in the panopticon. You could go as far as to give them access to each others logs and enable a bounty system for reporting (verified) abuse anonymously.
  - draw_down 5y ago
- Melatonic 5y ago
  I agree - this should be super important especially for a company like Okta
lupire 5y ago
Region is a red herring, except for government threats. Otherwise, the goal is to give each agent access to an arbitrary subset of users, so it's harder to recruit an agent who can access the "target" victim customer.
- cupofpython 5y ago
  It should be noted that this comes with significant (as in worth consideration, not necessarily huge) operations and logistical issues. Here's 3: * Balancing the assignment of users to appropriate agents * Less flexibility for agents to be specialized (instead of an agent responding to all X related tickets everywhere in the world, an agent must respond to X, Y, and Z from these specific users) * Handling user coverage when agents are busy, PTO, etc. The spirit of splitting agent access would mean agents would need to request access from an out-of-region authority in order to help with things last minute. An authority who will be expected to be responsive 24/7 without also being an attack vector
  - antoinealb 5y ago
    For the third point, couldn't you have some overlap between agents, so that each customer is served by N agents, each agent having a different set of M customers ? It is super expensive as an arrangement though.
    - cupofpython 5y ago
      Yeah that would work, but like you said it is expensive - or at least it is a cost that doesnt exist in the current model and would need specific attention. And we cannot assume customer requests / issues are uniformly distributed - it's not unlikely that 80% of issues come from 20% of customers. But if you give 80% of agents access to the same 20% of customers - you havent really solved the original problem. Albeit the risk factor is certainly lower and I would prefer this setup, who knows if that justifies the additional overhead and access control management. Technically, the current model is a full M customer x N agents mapping. All customers served by all Agents. Adding some 0's to that mapping should certainly be possible without sacrificing much from current system. No way to know how many is reasonable All that said, there is a natural logic that arises from this 'analysis' - Limit access to customers who are inactive, or who rarely have issues. A common sense addition would be to raise their default priority for when they do interact.
dhx 5y ago
== Information on mitigating security controls in place that could have limited the severity of the incident == The screenshots show that LAPSUS$ had access via the account of the outsourced Costa Rican based Tier 2 TSE worker to view details of and reset the password of a UK-based site reliability engineer of a well known Internet global infrastructure company. This level of access in itself doesn't sound hugely problematic on its own if it weren't for job applications for Okta Tier 2 TSE workers indicating server/network administration responsibilities. There is some publicly published information that provides more context on what access a Tier 2 TSE worker may have to customer data: - Okta's security whitepaper [1] states that multi-factor authentication is needed for administrative access to AWS instances (host operating systems) so this could have possibly prevented LAPSUS$ from using their access to do anything more sinister than perhaps causing customer organisation-wide outages by disabling/resetting all their user accounts. The response for control AC-17(06) at [3] indicates that remote access requires valid multi-factor authentication to occur so if LAPSUS$ could gain remote access, perhaps they had already gained the ability to generate valid multi-factor authentication tokens, or otherwise bypass multi-factor authentication for remote access. - The response for controls IA-02(09), AC-06(07), PS-04(01) and SA-12 at [3] indicates that Okta employee access to customer data is deemed privileged access and that fewer than 30 Okta employees have this level of privileged access to this customer data. I doubt this is true, unless the SP 800-53r4 responses at [3] are only referring to the US government tenancies that are hosted separately from tenancies of global companies. - The response for controls SI-04(12), SI-04(19) and SI-04(20) at [3] indicates that changes made to a customer tenancy (for example resetting passwords for customer users as shown in the LAPSUS$ screenshots) should have triggered events being sent to both Okta's Splunk SIEM system and also to a customer SIEM system if they have set one up. Theoretically this should quickly raise a red flag to the customer if LAPSUS$ were to use this access to bulk deny the customer access to their ICT systems. It is however not clear that privileged access to the host operating system for the AWS instance would cause an event to be sent to the customer's SIEM system. == Level of segregation of customer tenancies and whether LAPSUS$ may have been using Okta access to attack customers == It's fairly straightforward to enumerate the Okta hosting arrangements from DNS records as listed below. The sample targeted company in the screenshots has their Okta tenancy hosted within AWS region us-west-2 (Okta OK7) and other screenshots showed that the compromised account had recently accessed privileged access tools for OK3, OK8 and OK12. Thus it would appear that a Costa Rican-based Tier 2 TSE worker would have access to customer information within perhaps any of the clusters (not geographically restricted). ok-crtr-tls1-nlb-36253386bec48ce6.elb.us-east-1.amazonaws.com - response when a tenancy is invalid/doesn't exist ok2-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-1.amazonaws.com - companies (global) ok3-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-1.amazonaws.com - companies (global) ok4-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-1.amazonaws.com - companies (global) ok5-crtr-tls12-fips-nlb-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com - US government agencies ok6-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-2.amazonaws.com - companies (global) ok7-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com - companies (global) ok8-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.ap-southeast-2.amazonaws.com - Australian government agencies and some other Australian companies/associations ok9-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.eu-west-1.amazonaws.com - unknown / no customers found ok10-crtr-tls12-fips-nlb-xxxxxxxxxxxxxxxx.elb.us-east-2.amazonaws.com - US government agencies ok11-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-2.amazonaws.com - unknown / no customers found ok12-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com - unknown / no customers found To attempt to answer a question in another comment, whether LAPSUS$ could have been compromising other companies via Okta: microsoft.okta.com - likely customer - ok3-crtr-tls12-nlb-dfef298ffc8f82ca.elb.us-east-1.amazonaws.com nvidia.okta.com - likely customer - ok2-crtr-tls12-nlb-acd62f33a2a6a463.elb.us-east-1.amazonaws.com vodafone.okta.com - confirmed customer [4] - ok4-crtr-tls12-nlb-29367a8e4bb80716.elb.us-east-1.amazonaws.com samsung.okta.com [5] - possibly not a customer - ok-crtr-tls1-nlb-36253386bec48ce6.elb.us-east-1.amazonaws.com ubisoft.okta.com - possbily not a customer - ok-crtr-tls1-nlb-36253386bec48ce6.elb.us-east-1.amazonaws.com lg.okta.com [5] - possibly not a customer - ok-crtr-tls1-nlb-36253386bec48ce6.elb.us-east-1.amazonaws.com ... so, probably not? [1] https://www.okta.com/resources/whitepaper/okta-security-technical-white-paper/ https://www.okta.com/resources/whitepaper/okta-security-tech... [2] https://trust.okta.com/security/ https://trust.okta.com/security/ [3] https://www.okta.com/resources/whitepaper/using-okta-to-protect-il4-fedramp/ https://www.okta.com/resources/whitepaper/using-okta-to-prot... [4] https://miniorange.com/atlassian/atlassian-single-sign-on-sso-for-vodafone/ https://miniorange.com/atlassian/atlassian-single-sign-on-ss... [5] Note: also tried similar names known to be used by company divisions.
WhyNotHugo 5y ago
> Do employees get notified when their account is logged into or used (e-mail/SMS/otherwise)? No. It's a tricky situation too, since even if one did receive such email, I won't look at work email until working hours. So even with such a notification, if you logged in on friday 20:00, you've over 50 hours to do your work. I might also add that Okta DOES NOT support any form of physical 2FA (e.g.: yubikeys), only software-based ones.
- judge2020 5y ago
  https://help.okta.com/en/prod/Content/Topics/Security/mfa-webauthn.htm https://help.okta.com/en/prod/Content/Topics/Security/mfa-we... ?
- mkdirp 5y ago
  > I might also add that Okta DOES NOT support any form of physical 2FA (e.g.: yubikeys), only software-based ones. Not sure where you got this from, but we've been using yubikeys for a few months now. They even allow policies to only allow physical 2FA methods.
- Sohcahtoa82 5y ago
  > I might also add that Okta DOES NOT support any form of physical 2FA (e.g.: yubikeys), only software-based ones. You might be operating under outdated information. I absolutely use a YubiKey with my Okta account.
- sleepybrett 5y ago
  They, in fact, support many forms of 2fa, your org just doesn't have them enabled.

by dhx 5y ago

toyg 5y ago

> Can a support engineer for customers in one region access the systems/databases relied upon by customers in other regions? My (admittedly outdated) experience is that this is always the case, regardless of any assurance vendors might have provided to the customer. In the end, people have to get stuff done, and offshore bodies are just too cheap to pass on. Things might have changed a bit since GDPR but, in practice, I expect there will always be "channels" for people to reach out cross-region.

donalhunt 5y ago
Also "timezones". Just because a customer is based in Europe doesn't mean they don't need to get a hold of support at 03:00 local time. Even small, growing, startups now have HQs in one part of the world and staff across the globe due to the ease of movement many enjoy.

draw_down 5y ago

“Do they have access” is one thing, but what happens when they use the access? Is there any logging or proactive monitoring? A single support account reading say 10x the normal number of accounts per day, with a different geographic distribution than the norm, should ring alarm bells. It’s not enough to say “well shucks they have access to the data, guess we’ll do nothing”

cupofpython 5y ago
I agree with you, put support agents in the panopticon. You could go as far as to give them access to each others logs and enable a bounty system for reporting (verified) abuse anonymously.
- draw_down 5y ago
Melatonic 5y ago
I agree - this should be super important especially for a company like Okta

lupire 5y ago

Region is a red herring, except for government threats. Otherwise, the goal is to give each agent access to an arbitrary subset of users, so it's harder to recruit an agent who can access the "target" victim customer.

cupofpython 5y ago
It should be noted that this comes with significant (as in worth consideration, not necessarily huge) operations and logistical issues. Here's 3: * Balancing the assignment of users to appropriate agents * Less flexibility for agents to be specialized (instead of an agent responding to all X related tickets everywhere in the world, an agent must respond to X, Y, and Z from these specific users) * Handling user coverage when agents are busy, PTO, etc. The spirit of splitting agent access would mean agents would need to request access from an out-of-region authority in order to help with things last minute. An authority who will be expected to be responsive 24/7 without also being an attack vector
- antoinealb 5y ago
  For the third point, couldn't you have some overlap between agents, so that each customer is served by N agents, each agent having a different set of M customers ? It is super expensive as an arrangement though.
  - cupofpython 5y ago
    Yeah that would work, but like you said it is expensive - or at least it is a cost that doesnt exist in the current model and would need specific attention. And we cannot assume customer requests / issues are uniformly distributed - it's not unlikely that 80% of issues come from 20% of customers. But if you give 80% of agents access to the same 20% of customers - you havent really solved the original problem. Albeit the risk factor is certainly lower and I would prefer this setup, who knows if that justifies the additional overhead and access control management. Technically, the current model is a full M customer x N agents mapping. All customers served by all Agents. Adding some 0's to that mapping should certainly be possible without sacrificing much from current system. No way to know how many is reasonable All that said, there is a natural logic that arises from this 'analysis' - Limit access to customers who are inactive, or who rarely have issues. A common sense addition would be to raise their default priority for when they do interact.

dhx 5y ago

== Information on mitigating security controls in place that could have limited the severity of the incident == The screenshots show that LAPSUS$ had access via the account of the outsourced Costa Rican based Tier 2 TSE worker to view details of and reset the password of a UK-based site reliability engineer of a well known Internet global infrastructure company. This level of access in itself doesn't sound hugely problematic on its own if it weren't for job applications for Okta Tier 2 TSE workers indicating server/network administration responsibilities. There is some publicly published information that provides more context on what access a Tier 2 TSE worker may have to customer data: - Okta's security whitepaper [1] states that multi-factor authentication is needed for administrative access to AWS instances (host operating systems) so this could have possibly prevented LAPSUS$ from using their access to do anything more sinister than perhaps causing customer organisation-wide outages by disabling/resetting all their user accounts. The response for control AC-17(06) at [3] indicates that remote access requires valid multi-factor authentication to occur so if LAPSUS$ could gain remote access, perhaps they had already gained the ability to generate valid multi-factor authentication tokens, or otherwise bypass multi-factor authentication for remote access. - The response for controls IA-02(09), AC-06(07), PS-04(01) and SA-12 at [3] indicates that Okta employee access to customer data is deemed privileged access and that fewer than 30 Okta employees have this level of privileged access to this customer data. I doubt this is true, unless the SP 800-53r4 responses at [3] are only referring to the US government tenancies that are hosted separately from tenancies of global companies. - The response for controls SI-04(12), SI-04(19) and SI-04(20) at [3] indicates that changes made to a customer tenancy (for example resetting passwords for customer users as shown in the LAPSUS$ screenshots) should have triggered events being sent to both Okta's Splunk SIEM system and also to a customer SIEM system if they have set one up. Theoretically this should quickly raise a red flag to the customer if LAPSUS$ were to use this access to bulk deny the customer access to their ICT systems. It is however not clear that privileged access to the host operating system for the AWS instance would cause an event to be sent to the customer's SIEM system. == Level of segregation of customer tenancies and whether LAPSUS$ may have been using Okta access to attack customers == It's fairly straightforward to enumerate the Okta hosting arrangements from DNS records as listed below. The sample targeted company in the screenshots has their Okta tenancy hosted within AWS region us-west-2 (Okta OK7) and other screenshots showed that the compromised account had recently accessed privileged access tools for OK3, OK8 and OK12. Thus it would appear that a Costa Rican-based Tier 2 TSE worker would have access to customer information within perhaps any of the clusters (not geographically restricted). ok-crtr-tls1-nlb-36253386bec48ce6.elb.us-east-1.amazonaws.com - response when a tenancy is invalid/doesn't exist ok2-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-1.amazonaws.com - companies (global) ok3-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-1.amazonaws.com - companies (global) ok4-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-1.amazonaws.com - companies (global) ok5-crtr-tls12-fips-nlb-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com - US government agencies ok6-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-2.amazonaws.com - companies (global) ok7-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com - companies (global) ok8-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.ap-southeast-2.amazonaws.com - Australian government agencies and some other Australian companies/associations ok9-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.eu-west-1.amazonaws.com - unknown / no customers found ok10-crtr-tls12-fips-nlb-xxxxxxxxxxxxxxxx.elb.us-east-2.amazonaws.com - US government agencies ok11-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-2.amazonaws.com - unknown / no customers found ok12-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com - unknown / no customers found To attempt to answer a question in another comment, whether LAPSUS$ could have been compromising other companies via Okta: microsoft.okta.com - likely customer - ok3-crtr-tls12-nlb-dfef298ffc8f82ca.elb.us-east-1.amazonaws.com nvidia.okta.com - likely customer - ok2-crtr-tls12-nlb-acd62f33a2a6a463.elb.us-east-1.amazonaws.com vodafone.okta.com - confirmed customer [4] - ok4-crtr-tls12-nlb-29367a8e4bb80716.elb.us-east-1.amazonaws.com samsung.okta.com [5] - possibly not a customer - ok-crtr-tls1-nlb-36253386bec48ce6.elb.us-east-1.amazonaws.com ubisoft.okta.com - possbily not a customer - ok-crtr-tls1-nlb-36253386bec48ce6.elb.us-east-1.amazonaws.com lg.okta.com [5] - possibly not a customer - ok-crtr-tls1-nlb-36253386bec48ce6.elb.us-east-1.amazonaws.com ... so, probably not? [1] https://www.okta.com/resources/whitepaper/okta-security-technical-white-paper/ https://www.okta.com/resources/whitepaper/okta-security-tech... [2] https://trust.okta.com/security/ https://trust.okta.com/security/ [3] https://www.okta.com/resources/whitepaper/using-okta-to-protect-il4-fedramp/ https://www.okta.com/resources/whitepaper/using-okta-to-prot... [4] https://miniorange.com/atlassian/atlassian-single-sign-on-sso-for-vodafone/ https://miniorange.com/atlassian/atlassian-single-sign-on-ss... [5] Note: also tried similar names known to be used by company divisions.

WhyNotHugo 5y ago

> Do employees get notified when their account is logged into or used (e-mail/SMS/otherwise)? No. It's a tricky situation too, since even if one did receive such email, I won't look at work email until working hours. So even with such a notification, if you logged in on friday 20:00, you've over 50 hours to do your work. I might also add that Okta DOES NOT support any form of physical 2FA (e.g.: yubikeys), only software-based ones.

judge2020 5y ago
https://help.okta.com/en/prod/Content/Topics/Security/mfa-webauthn.htm https://help.okta.com/en/prod/Content/Topics/Security/mfa-we... ?
mkdirp 5y ago
> I might also add that Okta DOES NOT support any form of physical 2FA (e.g.: yubikeys), only software-based ones. Not sure where you got this from, but we've been using yubikeys for a few months now. They even allow policies to only allow physical 2FA methods.
Sohcahtoa82 5y ago
> I might also add that Okta DOES NOT support any form of physical 2FA (e.g.: yubikeys), only software-based ones. You might be operating under outdated information. I absolutely use a YubiKey with my Okta account.
sleepybrett 5y ago
They, in fact, support many forms of 2fa, your org just doesn't have them enabled.