7 ms·

Red flags for me: - Github alerted Okta about the access, they were not able to detect this themselves (https://docs.github.com/en/organizations/keeping-your-o

Red flags for me: - Github alerted Okta about the access, they were not able to detect this themselves (https://docs.github.com/en/organizations/keeping-your-organization-secure/managing-security-settings-for-your-organization/reviewing-the-audit-log-for-your-organization https://docs.github.com/en/organizations/keeping-your-organi...) - It only says "access to code repositories" (it does not say anything about the level of that access, it might as well mean write access, capability to trigger actions etc.) - Not relying on the confidentiality of source code is great, but malicious CD workflow actions would still be a risk if attackers had that level of access. - No information about the entry point for compromise. I doubt their 'commitment to transparency'.

axsharma 4y ago
Add to it that they reviewed "all recent commits to Okta software repositories." Due diligence or indicative of the threat actor having write access? Many unanswered questions.
- twistedpair 4y ago
  A good reason to give engineers PGP keys and turn on the "required code signing" feature on your org. Alas, security and productivity are perpetual odds.
  - chrisshroba 4y ago
    Just wondering - would this meaningfully impact productivity beyond causing engineers to have to learn how to sign a commit (which would presumable take less than an hour, once)?
    - ghostpepper 4y ago
      Actually generating a key and signing commits is pretty easy. I think the harder part would be ensuring all devs safely store the keys, rotate them regularly, etc.
  - mdaniel 4y ago
    Just a friendly reminder that both GH and GL now support using SSH keys for signing commits, and 1Password (and KeePassXC, FWIW) will safely store those SSH creds off-disk: https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification#ssh-commit-signature-verification https://docs.github.com/en/authentication/managing-commit-si... https://docs.gitlab.com/ee/user/project/repository/ssh_signed_commits/ https://docs.gitlab.com/ee/user/project/repository/ssh_signe... https://developer.1password.com/docs/ssh/agent/ https://developer.1password.com/docs/ssh/agent/ https://keepassxc.org/docs/KeePassXC_UserGuide.html#_ssh_agent https://keepassxc.org/docs/KeePassXC_UserGuide.html#_ssh_age... Although in full transparency, I still use GPG for my needs, since I better understand its workflow
  - jlokier 4y ago
    Note that your GPG key is discarded, and GitHub signs your commit itself with GitHub.com's own GPG key when anyone uses the GitHub UI to merge your PR. All those "verified" buttons you see on a typical repo history tend to actually be for the GitHub.com signing key, which is shared by everyone. Your GPG signature is only used to convince GitHub to sign the final commit with its key. It is possible to put your GPG signature on the merged commits, so that people can trust the commits came from you. That may be especially appropriate for security software. But you have to do the merges (or rebases as you prefer) outside GitHub for that, and push those merges directly to the main branch. That's what I do when I can, but it's not common practice. Many orgs require all merges to be done via GitHub, so end up with GitHub.com's shared signature on everything instead of their own.
IanCal 4y ago
> - It only says "access to code repositories" (it does not say anything about the level of that access, it might as well mean write access, capability to trigger actions etc.) Given the exceptionally broad granularity of github permissions with oauth at least I'd be concerned. I've repeatedly had to avoid using something because while I wanted to grant permission to, say, read one particular repo I'd have to allow write access to all private repos. Githubs solution is to give broad access to a user that has limited access, although then they shout at you for creating more users if you're not a paying member yourself (but may be in an organisation). https://docs.github.com/en/developers/apps/building-oauth-apps/scopes-for-oauth-apps https://docs.github.com/en/developers/apps/building-oauth-ap...
- e1g 4y ago
  GitHub recently introduced fine-grained tokens that can be scoped to a single repository that might work for your use-case https://github.blog/2022-10-18-introducing-fine-grained-personal-access-tokens-for-github/ https://github.blog/2022-10-18-introducing-fine-grained-pers...
  - ChymeraXYZ 4y ago
    Unfortunately this only seems to be available for repos you own yourself and not if an org owns the repo, making it useless in a company context until that is expanded. Great for personal stuff tho.
    - drothlis 4y ago
      Fine-grained access tokens are available on org-owned repos too, but the org has to opt in (for some reason).
    - e1g 4y ago
      They work, but the organization needs to approve the token (and its scope). As an org admin, I prefer it this way because I can audit what access developers give out to what repositories. The new tokens are still in Beta, so there are some other limitations: for example, GitHub Packages do not support them yet, so you cannot use them in NPM/yarn to get your private packages hosted on GitHub.
noselasd 4y ago
So genuine question, how do I detect access to my github repository before github alerts me about something nefarious ?
- leftcenterright 4y ago
  > You can stream audit and Git events data from GitHub to an external data management system. For a company of Okta's scale and importance, this should be part of SIEM. - https://docs.github.com/en/enterprise-cloud@latest/admin/monitoring-activity-in-your-enterprise/reviewing-audit-logs-for-your-enterprise/streaming-the-audit-log-for-your-enterprise https://docs.github.com/en/enterprise-cloud@latest/admin/mon...
  - captn3m0 4y ago
    GitHub's Audit logging is quite lacking, it doesn't include API requests.
kerng 4y ago
Another red flag is that they should mention what they will do with clear-text credentials that are in the source code that was stolen. There are always creds in private/company repos! Lots.
- chunk_waffle 4y ago
  > There are always creds in private/company repos! Lots. I disagree with the "always" in this statement. Sloppy, lazy private repos sure. It is possible to have them completely absent in any and all repos though I've only seen and been part of such an effort once, it takes a lot of work to make sure it happens and I have little faith in most companies following through with that.
  - TechBro8615 4y ago
    In my experience it's a pretty low bar to keep private credentials outside of source code. If a "security" company like Okta has secrets in their source code, that's embarrassing and unexpected. Any competent team of 2+ developers should be able to avoid this. However, what's more common is secrets in CI variables. If their GitHub was breached, they should be more concerned with whether the attackers had access to GitHub Actions logs or secrets.
- nobleach 4y ago
  In this day and age, I really do not understand why one of the first steps when spinning up a new repo (for this type of app) is not leveraging a tool like dotenv, and then a config system that uses environment variables for things like db credentials/etc. Yes it takes another hour of time to get that all going but, in the long run, you'll thank yourself! I've worked in places where all the code was open to all teams, with the exception of DevOps because they had too many hardcoded secrets and never had time to clean up their mess. I get it, they really were spread thin... but it should have never happened in the first place.
- madcadmium 4y ago
  Okta employee here. I can assure you that there are no clear-text credentials in our source code.
twistedpair 4y ago
IAM on GitHub needs so much <3. So broad, much ow. For example, I trialed major security vendor's enterprise product. They required their app be granted Admin on the GitHub org. All they needed to do was create issues, PRs, and read source code for analysis. There are scopes for that. I was eventually on a call with a principle engineer in this company, who kept saying they needed this permissions, and I kept showing him the API docs that showed that wasn't so. Eventually he said, "well, we won't _use_ all those permissions, so just give them to us anyway, because it's easier this way." Sure, I'll give you the ability to change all my code, add/remove users, drop repos... etc, and trust that some day, when you're hacked, someone will not use those over granted permissions maliciously? Security is hard. Be careful what permissions you give your 3rd party GitHub integrations.
- maartenh 4y ago
  Ugh. Doesn't raise the trust in their competence of protecting admin access credentials to GitHub. The same mindset leads to "We use just one shared ssh cert, because it is easier. And our VPN solution is a 2nd factor in any case".
- phpisthebest 4y ago
  >>"well, we won't _use_ all those permissions, so just give them to us anyway, because it's easier this way." Devs have been doing that since the dawn on computing. Ohh your App needs to be able to write to a protected folder on windows. Dont document what folder just force the app the run as Admin. Early Android Apps asked for all the permissions, all the time because of lazy devs security is hard, and gets in the way of what the devs wnat to do so they just find ways to bypass it
  - elecush 4y ago
    "because of lazy devs" :thinking:
    - makestuff 4y ago
      Definitely not because of the pesky product managers and sales teams who want a new feature to sell yesterday to boost their EOY bonus...
avisser 4y ago
Since GitHub alerted Okta, I'm assuming they use the regular, hosted github.com. I'm kinda shocked a security company doesn't have a private GitHub Enterprise server behind a firewall.
- baq 4y ago
  if they had, it'd be worse - they'd probably never know they were hacked...
- int0x2e 4y ago
  Hosting something yourself does not make it magically more secure. Even if you hire a small team of really smart people, they'd have to work pretty hard to do as good of a job as the many 100s working on security at GitHub...
- AtNightWeCode 4y ago
  One would assume that their code is never accessible over the public Internet. It is pretty much over for Okta now.
- trallnag 4y ago
  So you expect "critical" companies to self-host everything?

axsharma 4y ago

Add to it that they reviewed "all recent commits to Okta software repositories." Due diligence or indicative of the threat actor having write access? Many unanswered questions.

twistedpair 4y ago
A good reason to give engineers PGP keys and turn on the "required code signing" feature on your org. Alas, security and productivity are perpetual odds.
- chrisshroba 4y ago
  Just wondering - would this meaningfully impact productivity beyond causing engineers to have to learn how to sign a commit (which would presumable take less than an hour, once)?
  - ghostpepper 4y ago
    Actually generating a key and signing commits is pretty easy. I think the harder part would be ensuring all devs safely store the keys, rotate them regularly, etc.
- mdaniel 4y ago
  Just a friendly reminder that both GH and GL now support using SSH keys for signing commits, and 1Password (and KeePassXC, FWIW) will safely store those SSH creds off-disk: https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification#ssh-commit-signature-verification https://docs.github.com/en/authentication/managing-commit-si... https://docs.gitlab.com/ee/user/project/repository/ssh_signed_commits/ https://docs.gitlab.com/ee/user/project/repository/ssh_signe... https://developer.1password.com/docs/ssh/agent/ https://developer.1password.com/docs/ssh/agent/ https://keepassxc.org/docs/KeePassXC_UserGuide.html#_ssh_agent https://keepassxc.org/docs/KeePassXC_UserGuide.html#_ssh_age... Although in full transparency, I still use GPG for my needs, since I better understand its workflow
- jlokier 4y ago
  Note that your GPG key is discarded, and GitHub signs your commit itself with GitHub.com's own GPG key when anyone uses the GitHub UI to merge your PR. All those "verified" buttons you see on a typical repo history tend to actually be for the GitHub.com signing key, which is shared by everyone. Your GPG signature is only used to convince GitHub to sign the final commit with its key. It is possible to put your GPG signature on the merged commits, so that people can trust the commits came from you. That may be especially appropriate for security software. But you have to do the merges (or rebases as you prefer) outside GitHub for that, and push those merges directly to the main branch. That's what I do when I can, but it's not common practice. Many orgs require all merges to be done via GitHub, so end up with GitHub.com's shared signature on everything instead of their own.

IanCal 4y ago

> - It only says "access to code repositories" (it does not say anything about the level of that access, it might as well mean write access, capability to trigger actions etc.) Given the exceptionally broad granularity of github permissions with oauth at least I'd be concerned. I've repeatedly had to avoid using something because while I wanted to grant permission to, say, read one particular repo I'd have to allow write access to all private repos. Githubs solution is to give broad access to a user that has limited access, although then they shout at you for creating more users if you're not a paying member yourself (but may be in an organisation). https://docs.github.com/en/developers/apps/building-oauth-apps/scopes-for-oauth-apps https://docs.github.com/en/developers/apps/building-oauth-ap...

e1g 4y ago
GitHub recently introduced fine-grained tokens that can be scoped to a single repository that might work for your use-case https://github.blog/2022-10-18-introducing-fine-grained-personal-access-tokens-for-github/ https://github.blog/2022-10-18-introducing-fine-grained-pers...
- ChymeraXYZ 4y ago
  Unfortunately this only seems to be available for repos you own yourself and not if an org owns the repo, making it useless in a company context until that is expanded. Great for personal stuff tho.
  - drothlis 4y ago
    Fine-grained access tokens are available on org-owned repos too, but the org has to opt in (for some reason).
  - e1g 4y ago
    They work, but the organization needs to approve the token (and its scope). As an org admin, I prefer it this way because I can audit what access developers give out to what repositories. The new tokens are still in Beta, so there are some other limitations: for example, GitHub Packages do not support them yet, so you cannot use them in NPM/yarn to get your private packages hosted on GitHub.

noselasd 4y ago

So genuine question, how do I detect access to my github repository before github alerts me about something nefarious ?

leftcenterright 4y ago
> You can stream audit and Git events data from GitHub to an external data management system. For a company of Okta's scale and importance, this should be part of SIEM. - https://docs.github.com/en/enterprise-cloud@latest/admin/monitoring-activity-in-your-enterprise/reviewing-audit-logs-for-your-enterprise/streaming-the-audit-log-for-your-enterprise https://docs.github.com/en/enterprise-cloud@latest/admin/mon...
- captn3m0 4y ago
  GitHub's Audit logging is quite lacking, it doesn't include API requests.

kerng 4y ago

Another red flag is that they should mention what they will do with clear-text credentials that are in the source code that was stolen. There are always creds in private/company repos! Lots.

chunk_waffle 4y ago
> There are always creds in private/company repos! Lots. I disagree with the "always" in this statement. Sloppy, lazy private repos sure. It is possible to have them completely absent in any and all repos though I've only seen and been part of such an effort once, it takes a lot of work to make sure it happens and I have little faith in most companies following through with that.
- TechBro8615 4y ago
  In my experience it's a pretty low bar to keep private credentials outside of source code. If a "security" company like Okta has secrets in their source code, that's embarrassing and unexpected. Any competent team of 2+ developers should be able to avoid this. However, what's more common is secrets in CI variables. If their GitHub was breached, they should be more concerned with whether the attackers had access to GitHub Actions logs or secrets.
nobleach 4y ago
In this day and age, I really do not understand why one of the first steps when spinning up a new repo (for this type of app) is not leveraging a tool like dotenv, and then a config system that uses environment variables for things like db credentials/etc. Yes it takes another hour of time to get that all going but, in the long run, you'll thank yourself! I've worked in places where all the code was open to all teams, with the exception of DevOps because they had too many hardcoded secrets and never had time to clean up their mess. I get it, they really were spread thin... but it should have never happened in the first place.
madcadmium 4y ago
Okta employee here. I can assure you that there are no clear-text credentials in our source code.

twistedpair 4y ago

IAM on GitHub needs so much <3. So broad, much ow. For example, I trialed major security vendor's enterprise product. They required their app be granted Admin on the GitHub org. All they needed to do was create issues, PRs, and read source code for analysis. There are scopes for that. I was eventually on a call with a principle engineer in this company, who kept saying they needed this permissions, and I kept showing him the API docs that showed that wasn't so. Eventually he said, "well, we won't _use_ all those permissions, so just give them to us anyway, because it's easier this way." Sure, I'll give you the ability to change all my code, add/remove users, drop repos... etc, and trust that some day, when you're hacked, someone will not use those over granted permissions maliciously? Security is hard. Be careful what permissions you give your 3rd party GitHub integrations.

maartenh 4y ago
Ugh. Doesn't raise the trust in their competence of protecting admin access credentials to GitHub. The same mindset leads to "We use just one shared ssh cert, because it is easier. And our VPN solution is a 2nd factor in any case".
phpisthebest 4y ago
>>"well, we won't _use_ all those permissions, so just give them to us anyway, because it's easier this way." Devs have been doing that since the dawn on computing. Ohh your App needs to be able to write to a protected folder on windows. Dont document what folder just force the app the run as Admin. Early Android Apps asked for all the permissions, all the time because of lazy devs security is hard, and gets in the way of what the devs wnat to do so they just find ways to bypass it
- elecush 4y ago
  "because of lazy devs" :thinking:
  - makestuff 4y ago
    Definitely not because of the pesky product managers and sales teams who want a new feature to sell yesterday to boost their EOY bonus...

avisser 4y ago

Since GitHub alerted Okta, I'm assuming they use the regular, hosted github.com. I'm kinda shocked a security company doesn't have a private GitHub Enterprise server behind a firewall.

baq 4y ago
if they had, it'd be worse - they'd probably never know they were hacked...
int0x2e 4y ago
Hosting something yourself does not make it magically more secure. Even if you hire a small team of really smart people, they'd have to work pretty hard to do as good of a job as the many 100s working on security at GitHub...
AtNightWeCode 4y ago
One would assume that their code is never accessible over the public Internet. It is pretty much over for Okta now.
trallnag 4y ago
So you expect "critical" companies to self-host everything?