18 ms·

Ask HN: How does your company manage its encryption keys?

488 points by 2mol 6y ago

We just had an interesting data loss at work, that was due to data being encrypted at rest. We somehow managed to delete the encryption keys (still figuring out how), which became an obvious problem once our main database instance was rebooted. Luckily we were able to restore the data, but now I (we) really want to learn what a proper setup would look like. If you have any clear overview reading on the topic I'd be very interested to to know about it. In particular I'm wondering: how do you back up your encryption keys, or even put them in escrow somewhere? Assuming we don't rotate the keys constantly I would love to just save them in somewhing like a passsword manager that's secured with 2FA/FIDO. Would love to hear your thoughts!

dudus 6y ago
https://aws.amazon.com/kms/ https://aws.amazon.com/kms/
kissgyorgy 6y ago
Put it in a password manager. Nothing can be any simpler than that.
- KMnO4 6y ago
  I like this approach because 1Password provides a nice CLI that I can use in scripts: op get document UNIQUE_DOCUMENT_ID > pem Anyone in my department (or who can access the shared vault) can run this script by simply logging into 1Password cli (`op login`) before running.
toomuchtodo 6y ago
https://www.vaultproject.io/ https://www.vaultproject.io/ We use Hashicorp's Vault product to manage SSH credentials, TLS certificates, as well as application secrets across thousands of users, tens of thousands of virtual machines, and hundreds of applications. We pay for the enterprise version, but the free version is more than capable for most needs. Avoid a password manager if you can, it leads to poor security practices and availability issues depending on how and where the data is stored (IMHO, YMMV). Use single sign on where ever possible. Disclaimer: No affiliation other than a satisfied customer and frequent end user of the product.
- XCSme 6y ago
  Looks cool, one small nitpick is regarding their cookies banner. You can click "preferences" and it says "To opt out of a category of data collection, set the toggle to “Off” and save your preferences". Not only it is opt-out, but the mentioned "off" toggle doesn't even exist.
  - XCSme 6y ago
    They also have a link to their Privacy policy, which is a 404: https://www.vaultproject.io/privacy https://www.vaultproject.io/privacy
    - rnotaro 6y ago
      I bet it's because it's a relative link. I don't get the cookie banner because I'm from canada but the links at the bottom redirect to Hashicorp. The privacy policy is here: https://www.hashicorp.com/privacy https://www.hashicorp.com/privacy
  - XCSme 6y ago
    A bit weird to have a company working with such sensitive data not care about privacy.
    - ryanisnan 6y ago
      I think it's a bit disingenuous to claim that they do "not care about privacy". Hashicorp has demonstrated they do, on many occasions.
      - XCSme 6y ago
        Well, this was my first contact with this company and this was my experience. Probably they pay a lot more attention to their products than they do to their websites.
- wadkar 6y ago
  +1 for vault. You can setup your own instance if you like. IMHO, initial ramp up time will be worth the long term benefits.
Ididntdothis 6y ago
Very badly and inconsistent in my company ! We have a lot of people raising concerns about whatever you want to do. But not many actually have any contributions for how to do it right so every team rolls their own little solution. I will be very interested in hearing how others do it.
dijit 6y ago
Not sure the "right" way to do it. But this is what we did: For context: We run a centralised salt-master, salt master unencrypts content using gpg filters as part of variable generation (salt "pillars"). So it's encrypted at rest and encrypted in our git repositories. What we do/did, is: * grab a pair of differently branded USB sticks. * LUKS encrypt the USB sticks; we used a keyfile which is encrypted on our machines with our GPG key. * encrypt salt's GPG private key with all of our keys. * encrypt some of the "irreplacable" private keys (IE: CA roots) with all of our GPG keys. * store it all on the pair of USB drives * put the USB drives in a real-life vault, give the keys to the office team. We haven't needed to recover, but it's clearly documented how to recover if anything went wrong.
- openfuture 6y ago
  Bit rot is a thing. It'd be better to take a page out of the cryptocurrency handbook and print the keys out as QR codes (maybe with a simple password so it can only be restored if you know the pw).
- jeremygaither 6y ago
  How do you handle the LUKS key file? Do you encrypt it to the whole team? Just yourself? How do you circulate that LUKS key file?
- frabbit 6y ago
  Why differently branded?
openfinch 6y ago
Can't speak to my current employer as it's above my pay-grade to know, but at Job-1 we did the following: - All "hot" keys were stored in an offline credential manager in specific vaults depending on who needed access to them. Only staff with actual clearance could request temporary access to a vault (fully background checked, 1 year employment, etc). - Copies of each vaulth and our master CA cert were written to 4 encrypted USB sticks. Two stored on-site in the fire-safe and two off-site at our safety deposit box that only c-level staff could access. (We had the same process with our tokens and master logins for AWS). - Any work using those keys was on a pair-up basis, so at least two people, one doing the work and the other observing. - We had a detailed policy around this that covered each step in the process and who needs to approve them; everyone who could feasibly need to access the keys was briefed annually as part of our security awareness training. We handled a LOT of sensitive financial data, so this was the most appropriate way that we could find that maintained both sensible availability and key control. So in order to get to the keys you needed: - Access to the fire safe (Senior Ops, Senior Security and C-Level only). - The LUKS passphrase for the USB sticks (Senior Security and some C-Level only). - The passphrase for the specific vault (Senior Security and some C-Level only). I don't know how the passphrases were managed by our sec team, but I know that the C-Level staff had physical envelopes in their home safes.
- brutus1213 6y ago
  An expanded version of this would make a valuable book (or blog post at least).
  - deleted 6y ago
    [deleted]
- closeparen 6y ago
  How did you handle new secrets / rotations? Seems like a lot to keep in sync. Seems like we hear more frequently about the actual secrets being stored encrypted (potentially with hardware protection) in a central place, and only the keys to unlock them being distributed like this.
cj 6y ago
AWS KMS for most thing. AWS ACM for SSL certs. AWS SSM for ssh, eliminates the needs for ssh keys. Not everyone loves AWS (me included), but this stack works nicely in removing the need to ever touch raw encryption key files locally.
thesimon 6y ago
Vault for big stuff, git-crypt for file-level encryption of secrets (in Terraform files etc.)
- dijit 6y ago
  What happens if vault gets destroyed or corrupted?
  - NovemberWhiskey 6y ago
    This is why you have geographically separated disaster recovery replicas and backups.
deleted 6y ago
[deleted]
thieving_magpie 6y ago
Someone at my company generated the keys. They then put them on a network share without any security restrictions. They've been there for 5 years with no rotation. At least 2 are checked into source control.
- dijit 6y ago
  If those are the keys used in production, then I'm horrified. If they're dev-keys, I think this is pretty common.
  - thieving_magpie 6y ago
    We make no distinction between dev keys and production. Consider them production. Since it's of interest to HN, I am working on educating our very small team on how keys should be protected and used. I am the youngest developer by about 15 years. It's a very rural company and it often feels like all learning and passion for development stalled around 2005. It's a company that gave me a chance to grow into a development role with no previous experience so I feel indebted to try my best to keep the lights on.
    - dijit 6y ago
      aha, sounds fair. I don't judge too harshly- anyone who has black and white principles on these matters has never worked in any other industry most likely... all you can do is your best to steer the ship and convey the downsides. I think it's important too because it helps us understand how much friction people will tolerate. In many cases, even a small amount of friction will cause people to stop functioning completely; I recently tried setting up vault and it was a nightmare, I understand why people avoid picking it up. That doesn't mean we should not try; we have to become the advocates, arbiters and helpers for those systems. Good luck, you're not alone.
      - thieving_magpie 6y ago
        Thanks for the kind words. Good luck to you as well.
ReptileMan 6y ago
> How does your company manage its encryption keys? Poorly seems to be the answer industry wide. Both encryption and disaster recovery are hard tasks. Combine them and you have recipe for a total mess.
virgilp 6y ago
Nice try, NSA.
drcongo 6y ago
Depending on what the keys are for, we use either 1Password or EnvKey for storing them.
mbreese 6y ago
At one job I was afraid of losing a set of keys, so as an extra backup, we had a physical copy of the the key printed out as a QR code. This was then physically locked away (safe deposit or similar). But I like this as an option for a “failure resistant” offline backup. The main benefit is that doesn’t assume a stored away USB stick will still be readable while still having a somewhat machine readable format. Yes, the QR code was very large, but could be split into multiple files if needed.
r1ch 6y ago
I put them into a password manager which is backed up both on Dropbox as part of sync and Backblaze for long term.
rraval 6y ago
We use StackExchange's blackbox to check in GPG encrypted secrets into our monorepo: https://github.com/StackExchange/blackbox https://github.com/StackExchange/blackbox
MattGaiser 6y ago
At a prior company they were printed off and stored in a typical file folder.
AgalmicVentures 6y ago
YubiHSM 2 has worked fantastically well for us as a root of trust in a variety of applications, at a very reasonable price ($650) (I am unaffiliated with Yubico other than as a very satisfied customer). Accessible over USB or HTTP, it supports every major crypto algorithm [1], and keys can be backed up onto another HSM via a wrap key (if they are marked as exportable -- you can also control what can and cannot be exported -- in fact, every operation may be allowed or disallowed per key). Every operation is logged for audit, of course, and the device may be setup to require logs to be read before they are overwritten. In combination with configuring a special authentication key to access the logs, you can ensure that every operation on the HSM is logged to a remote store before additional operations may be completed. It does depend on your existing physical security, so that has to be taken into account when designing architectures including it. The micro form factor at least makes it trivial to put into an internal USB port. And of course, if you require a more enterprise grade tool, you may want to use an HSM in combination with a tool like Hashicorp Vault to manage your keys throughout your orgnaization. [1] https://developers.yubico.com/YubiHSM2/Product_Overview/ https://developers.yubico.com/YubiHSM2/Product_Overview/
MangoCoffee 6y ago
We use Azure KeyVault
- greener_fields 6y ago
  +1
- maxbaines 6y ago
  +1 Worth noting accessible using api via REST https://docs.microsoft.com/en-us/rest/api/keyvault/ https://docs.microsoft.com/en-us/rest/api/keyvault/ and powershell https://docs.microsoft.com/en-us/azure/key-vault/secrets/quick-create-powershell https://docs.microsoft.com/en-us/azure/key-vault/secrets/qui...
  - maxbaines 6y ago
    I also use this for my passwords https://news.ycombinator.com/item?id=22316520 https://news.ycombinator.com/item?id=22316520
  - idatum 6y ago
    +1 for Az KeyVault. I use it in my Docker deployment scripts using Azure CLI. Example here is a secret, but similar concept for certs using CLI: STORAGE_ACCOUNT_KEY=`az keyvault secret show --vault-name=<your keyvault> --name=<secret name> --query=value | tr -d "\""` This populates a .env file, referenced in Docker-compose.yml.
- Shicholas 6y ago
  +1, the integration to supplant K8 secrets is very nice.
munchbunny 6y ago
How paranoid do you want your security to be? In general I would suggest using a key vault. AWS, GCP, and Azure all have cloud versions that are backed by virtual HSM's built on top of actual physical HSM's. For the vast majority of usages they're good enough. Use admin account management to enforce 2FA/FIDO for all AWS/Azure/GCP logins. (You should be enforcing 2FA with phone/FIDO auth anyway.) If you need truly paranoid backups, you can back the key up onto a portable hard drive that you lock in a safe in the closet, with a few key people who know the code. I recommend against using a (cloud-synced) password manager. Cloud key vaults do the same thing but offer specific features relevant to server stuff. And if you want more paranoia, a physical safe is probably safer than extending your attack surface to a cloud-synced password manager. Also: make sure that you set up a ~quarterly ritual of opening and verifying the backup. For crucial backup fallback systems, you want to make sure you actually use the system so that you know if it fails.
eqvinox 6y ago
We have a main config/setup git repository which contains keys (and passwords) in git-crypt. 4 people have GPG keys able to decrypt it, and this is used in normal operation for deployment (i.e. for ansible to feed the keys into systems.) The repo is cloned in quite a few places. None of our secret material is particularly hard to revoke or replace; we don't run an internal CA or anything like that.
okso 6y ago
We store all keys that do not required automated access on Yubikey with the option that requires a physical touch per use. Usage includes SSH authentication, file encryption (backups and exchanges), git commit signatures and password/secret storage using `pass`. Copies of the offline master keys keys are stored on flash in safes onsite and offsite in bank vaults, and sub-keys are valid for one year. We use Hashicorp's Vault for secrets that require automated access.
uxp100 6y ago
I think it really matters how sensitive these keys are. It's been quite a few years since I interacted with them, but for some keys there is a server somewhere with an HSM installed, and two people have credentials for it. If you need something signed you send it to them, with a justification for why it needs to be signed with the real keys, and they will send you a path to get the signed file, and remind you to delete the signed file when you no longer need it. This is overkill for some things, and probably would be considered sloppy for others.
Spooky23 6y ago
I try to push as much to certificate auth as possible. Internal CA keys are stored on an offline USB HSM, which is locked in a cabinet. Access to the key requires 2/10 individuals to be physically present. There are different CAs for different purposes. There's an intermediate for device management, and another for user or service auth purposes.
- ahnick 6y ago
  So are your secrets served from a central node that authenticates the certificates? What does your process look like for changing secrets when someone leaves the company?
pianoraptor 6y ago
Piecing together a key management solution from open source is possible, but if you're trying to encrypt data and are open to commercial products, we do a lot of support of Vormetric. All of the key management is "built-in" and managed, so there really isn't much overhead. All software-based, with FIPS certified key management. It's very easy to encrypt data this way. It is expensive though. Disclaimer: while I used to work for Vormetric / Thales, I no longer do.
hn_throwaway_99 6y ago
I highly second the people saying KMS (AWS KMS, Google KMS, or KeyVault). * The pricing for just storing keys is incredibly cheap. * At least with Google KMS you can't delete the keys without a 24 hour waiting period (and you can alert on the deletion attempt), so that's a huge safeguard. * You get key access auditing out of the box.
- 2mol 6y ago
  Can you easily set up policies that prevent deletion of keys?
  - CiPHPerCoder 6y ago
    Yep: https://asecure.cloud/a/scp_kms_delete_keys/ https://asecure.cloud/a/scp_kms_delete_keys/
  - deleted 6y ago
    [deleted]
- nick-garfield 6y ago
  Definitely! And it depends on what key you're storing, but if you use AWS Secrets Manager, you can setup automatic key rotation to run periodically.
- opticalfiber 6y ago
  AWS KMS also enforces a waiting period of between 7 and 30 days before it will let you delete a key. There’s also a feature you can enable that automatically rotates your key once a year. KMS is great!
  - jacobsenscott 6y ago
    Given the keys never leave the KMS hardware encryption module, are you at all concerned that all your data will be destroyed if you lose access to KMS for any reason? That's what has always given me pause when I consider KMS. Or do KMS users just take on faith that AWS will always be there? Why do you like their auto-rotation? The keys that are rotated out are not never disabled, so I don't really understand the benefit. In what scenario would their auto-rotation improve security?
caipre 6y ago
Perhaps a dumb question, but how did you restore the data without the keys? Was a prior backup unencrypted?
gingerlime 6y ago
We use Bitwarden[0] for our secrets. It's open-source with a hosted option. Makes sharing passwords and keys across the team pretty straightforward. In addition to it, we use envwarden[1], which is a simple open-source wrapper around the Bitwarden CLI to manage our server secrets. It's super simple, but does the job for us well. We can then manage both passwords and keys in one place. Disclaimer: I created envwarden. I'm not affiliated with Bitwarden in any way however. Just a happy customer. [0] https://bitwarden.com/ https://bitwarden.com/ [1] https://github.com/envwarden/envwarden https://github.com/envwarden/envwarden
- shekhardesigner 6y ago
  I didn't know about envwarden till you mentioned it here. We migrated from bitwarden to offline client machine for the reason that we didn't wanted a copy of our encryption key be available anywhere else online.
- lux 6y ago
  envwarden looks awesome! Is there any potential for the Bitwarden project to incorporate it/make it officially supported?
shekhardesigner 6y ago
We keep the encryption key stored in plain text files on client offline machine. Catch: Client machine is encypted with VeraCrypt. Veracript hidden drives, one password is kept by product owner and another password is kept by head of security. Offline client machine is key for us. We rotate encryption keys after quarterly security audit.
N_A_T_E 6y ago
We have so many secret values it made sense to build our own internal product, an audited system which holds secrets in a backed up, locked down database. Apps pull from that system at runtime (or deploy time) but they can only access their own secrets using access control. We also use AWS KMS for AWS related resources.
- ahnick 6y ago
  Before building your own, did your company consider using Vault? If so, what factors led you to go down the build your own path?
jwr 6y ago
Here's what works for small and medium organizations for data which needs to be encrypted at rest, but is not often accessed (so, backups): 1. Buy a bunch of Yubikeys, minimum of 2. 2. Create GPG keys and store them on YubiKeys. Follow this guide: https://github.com/drduh/YubiKey-Guide https://github.com/drduh/YubiKey-Guide (if you want to, keep the secret keys, but in case of multiple YubiKeys I would not keep them anywhere). Remember to set the keys to always require touch. 3. Use GPG to encrypt your backups to multiple recipients (all of the YubiKeys). 4. Take care of the physical keys with proper storage and procedures. Do not store the keys together, have at least one in a really secure location, check if you have all the keys regularly, etc. 5. Test restores at least once per quarter, with a randomly selected key. The advantages of this solution is that it is simple, works pretty well, and gets you a lot of mileage with relatively little inconvenience. You don't have the risk of keys being copied, and guarding physical keys is easier than digital ones. You still have the problem of guarding the passphrases to the Yubikeys (if you use them), but that is much less of a problem than guarding the encryption keys. A passphrase without the physical key is useless. This setup works for organization from size 1 up to fairly large ones. Note that some recently fashionable security consultants crap on GPG from great height, but do not provide an alternative. It's a tool that while having multiple flaws, does many jobs better than anything else out there.
- rrdharan 6y ago
  This is all generally good advice, but I think there's huge potential complexity lurking here: > 4. Take care of the physical keys with proper storage and procedures. Do not store the keys together, have at least one in a really secure location, check if you have all the keys regularly, etc. Would be great to see what folks think this concretely looks like for joe random startup in Capital City, Somewhere. e.g. Does "really secure" mean "find a bank that still offers safety deposit boxes"? Does it mean paying for something like Iron Mountain (http://ironmountain.com/ http://ironmountain.com/) or one of its competitors?
  - techsupporter 6y ago
    > Does "really secure" mean "find a bank that still offers safety deposit boxes"? Realistically? Yes. This is what several of the companies I've done contract work for have done. You can still find at least one bank or self-storage place (look for the ones that don't have a nationally-advertised brand and don't look like they're made entirely out of corrugated metal) that do regular safety deposit boxes in pretty much any city. They may only be offered at a couple of locations and I've noticed credit unions bailing the hell out of this market as fast as they can decommission the vaults but boxes still exist. Let's assume all variables work in the other way, though. If you can't find a safety deposit box and don't have somewhere that's not your office you can drill into a floor or wall and you're storing a small device like Yubikeys or USB sticks, buy the heftiest portable gun safe you can find, one with a steel cable that loops back into the device, and stick it under your bathroom sink with the cable wrapped firmly around the water supply or drain pipe.
deleted 6y ago
[deleted]
nihil75 6y ago
We use cloud-provider managed encryption because we're not paranoid and don't have legal requirements to manage our own keys. We don't have SSH keys because it's not the 90's and we don't have servers.
- nocman 6y ago
  > We don't have SSH keys because it's not the 90's and we don't have servers. This seems unnecessarily snarky. There are lots of businesses in 2020 that still maintain their own servers, use ssh keys, and are staffed by admins and developers who very much know what they are doing (and are not at all "behind the times", as this comment seems to imply such businesses are). If that's not what you meant, well, OK, but I find it difficult to interpret it any other way.
  - pnutjam 6y ago
    He'll learn... probably the hard way.
    - NicoJuicy 6y ago
      Or the expensive way
      - nihil75 6y ago
        My serverless infra costs me 0$ per month.
z3ugma 6y ago
We use Bitwarden https://bitwarden.com/#organizations https://bitwarden.com/#organizations to store passwords as well as encryption secrets as a backup. Application repos have the encrypted secrets (meta) stored in their repos using Ansible-vault and the .vault_keys are stored in Bitwarden.
- pqdbr 6y ago
  We do exactly the same.
parasec 6y ago
The more complex your system grows, the more often it will fail and shoot you in the foot. I'd advise against systems like Hashicorp Vault - they just increase the complexity and while they have their merits in complex setups, you seem to be too small to be able to operate such a system. Have an offline backup printed along with the disaster recovery checklist and documentation and put them in a safe in your company - the checklist should be dumb enough that your drunk self can use it at four in the morning, because you were the nearest employee when everything went down. Ensure that you have stupid manual processes in place on rotation of the safe's PIN and encryption keys in general, including a sanity check if the newly generated keys actually work (e.g. if they are used for your backup storage, actually back something up and restore it). Ensure that the safe's PIN is available to at least another person and used regularly (e.g. because you store your backup tapes there). If you feel that you need to change from this very simple system to a more complex one, ask yourself why. What does your change actually add in terms of security and what risks does it add. In the end, you want your system available to customers and the security you add is to not only secure the data, but actually to know who can access it (the auditing part).
- 2mol 6y ago
  Great answer, thank you. I agree with your point about testing the restore process, right now I'm trying to think of a way to automate it. As a side note: for example we had some backups that are probably useless, because they are way too small. Catching this would mean more manual regular checks, or some automated rules, at which point it becomes quickly more complex again.
  - parasec 6y ago
    The backup part is quite easy - generate a well-known file with 1kb of data and include it in your backups. After the backup completed, validate that you can restore the well-known file and compare the content of the original and the restored. Easy to automate if you run a well-customizable backup system. Storytime: I did not check the restoration of my backups some time ago and had a faulty harddrive, so I needed to restore the backup. Backup was also as dumb as possible - essentially a tar and encrypting it with openssl. So I reinstalled the server, tried to decrypt it - got the error, that the key was wrong. Took me a good weekend to find out, that openssl changed the default hash algorithm between openssl 1.0 and 1.1. This would not have been catched with the proposed system, but now I really pin all default options in my scripts as well.
weitzj 6y ago
Online HSMs where you can exchange keys. Databases connect to the HSMs for crypto operations + offline HSMs with PCI cards in a safe for root keys. For other parts HashiCorp Vault/AWS kms
moxylush 6y ago
I've asked myself this question many times. Or "Who holds the keys to the keys?". Here is an air-gapped solution: https://www.arcanus55.com/?trusted55=A55PV2 https://www.arcanus55.com/?trusted55=A55PV2
lormayna 6y ago
In my previous experience, I was working for an HSM and Data Protection vendor. If you want to be resonable secure to don't lose keys and keep them safe, just use an HSM. If you need to encrypt filesystem, you can use a DP product (most of them are not so expensive) If you want to database content, you can use tokenization services.
lsllc 6y ago
Anyone using any KMIP based solutions? (looks like Hashicorp's Vault product implements a KMIP server). http://docs.oasis-open.org/kmip/spec/v1.4/kmip-spec-v1.4.html http://docs.oasis-open.org/kmip/spec/v1.4/kmip-spec-v1.4.htm...
gramakri 6y ago
I am curious how you managed to restore it if you lost the keys.
- 2mol 6y ago
  "accidental" backups
  - gramakri 6y ago
    ha ha, cheers mate. I asked because I had encrypted backups in the past for which I lost the key and wasn't so lucky :( I don't know your company size but we use stackoverflow's blackbox. It's very nice because you can check in the repo into version control. You can add and remove users on the fly as well.
itpragmatik 6y ago
AWS KMS
bloopernova 6y ago
When managing/deploying code we use SOPS (Secrets OPerationS) https://github.com/mozilla/sops https://github.com/mozilla/sops For standard password style secrets used by Ops, we use Team Password Manager. Which we chose about 5 years ago because it was self-hosted, the database was encrypted, and it had fantastic audit capabilities.
megavolcano 6y ago
AWS KMS. Access is added invidually using IAM roles. That's it.
up_and_up 6y ago
At my prev company we generated keys and split them using `ssss-split` and handed out shards to specific individuals via a keybase exploding message. Our system required at least 3 shards (combined via `ssss-combine`) to reboot. FYI: Hashicorp vault just uses Shamir's Secret Sharing scheme under the hood: https://github.com/hashicorp/vault/blob/45b9f7c1a53506dc97221f0915daeaeb0a6fe894/website/pages/guides/operations/rekeying-and-rotating.mdx#L20 https://github.com/hashicorp/vault/blob/45b9f7c1a53506dc9722...
haidrali 6y ago
We use keybase
- ahnick 6y ago
  We also use Keybase. We wrote an extension to our existing lightweight management script, encpass.sh(https://github.com/plyint/encpass.sh https://github.com/plyint/encpass.sh), to be able to access/manage our secrets from the CLI and in our infrastructure scripts. The extension is encpass-keybase.sh (https://github.com/plyint/encpass.sh/blob/master/extensions/keybase/KEYBASE.md https://github.com/plyint/encpass.sh/blob/master/extensions/...) kept in the same repo. The extension makes use of the per-user keys of Keybase. This allows you to manage access permissions using the Keybase GUI client. Once a user is added to the appropriate Keybase team, they will immediately have access to that teams secrets that are stored in the encrypted Git repos. To use it you just need to download the 2 shell scripts to a directory in your path (e.g. /usr/local/bin/) or you can clone the git repo.
ppafin 6y ago
On clean hardware under four eyes policy: https://imgur.com/a/FE8sslv https://imgur.com/a/FE8sslv
mavdi 6y ago
AWS secrets manager or SSM or KMS for any kind of secrets, keys etc. Works well because our entire stack is on AWS. Otherwise hashicorps Vault would do I guess but that’s yet another service on life support.
hellcow 6y ago
We use shh for secrets (https://egt.run/shh https://egt.run/shh). It's designed to integrate really well with your existing CLI tools like vim, xargs, and diff. It offers user-based permissions, and secrets are encrypted into a single file that's safe to commit into your git repo. We can stream secrets out of it directly to our remote servers during deploys. Unlike Vault you don't need to manage infra to run it -- it's just a file. Unlike cloud secret managers, there's no lock-in.
- ahnick 6y ago
  A couple of questions.. 1. Does every client have a copy of shh to interact with the secrets? Or are the secrets in the file served from a single centralized node? 2. What is your process of exchanging user keys with shh? 3. If someone leaves the company, what is the process you go through to change the secrets and rotate keys?
  - hellcow 6y ago
    1. Every client (i.e. developer) has their own copy of shh but interacts with a shared .shh file in the project root. Secrets can be decrypted locally in memory then streamed to the remote servers over ssh. You could in theory run this on a server to help address your third point. 2. Exchanging public user keys is done via email when an employee starts, then added to the .shh file and committed in the repo. 3. You'd need to write a script for this, which probably involves `shh rm-user {email}`. There is no silver bullet here for changing secrets; since developers have access to secrets, any secret they had would need to be regenerated. `shh` makes no assumptions about how you deploy, what secrets you keep, or their formats.
qubex 6y ago
I worked in a place that had a very nice in-house blade server setup with an attached NAS, which was encrypted (at rest, on storage). This was early in the “cloud” epoch and at any rate they preferred in-house iron for entirely understandable reasons. Also, they needed to do some pretty interesting things, so they had a NAS populated with enterprise-grade SSDs and a native AES-based encryption scheme. This kept the key on what was basically a glorified USB key. (For those of you who wish to know these details, the network between the blades and between the blades and the NAS was a nicely spec’d fibre channel network, and there was some iSCSI involved. The blades featured Itanium processors, which kind of gives the manufacturer away, and the firm had invested quite heavily in producing very high-performance code for those ill-fated microprocessors, but I digress.) So... it happened that somebody lost the USB key. Well, not quite. Somebody took it home during a weekend (whilst the system was shut down) and their kid used it for school work. This proved to be a “significant problem”. There was a backup, and it was encrypted and stored on an adjacent SAN. It wasn’t exactly stale, but it wasn’t entirely pristine either. There was much woe and gnashing of teeth. Nobody was fired because the dolt who maintained custody of the only USB key was the founder/CEO, so he couldn’t exactly blame himself. But, yeah. That happened, sadly.
tlarkworthy 6y ago
The new Google secret manager is a pretty nice way of having an easy to use web interface protected by IAM, with a REST API for applications to pull. You could easily prevent users from deleting keys. It's not as hardcore as Vault but it's a much simpler way of getting keys out of source control IMHO. You can have 2FA and audit logs easily too. Simpler than Google KMS too. https://cloud.google.com/secret-manager https://cloud.google.com/secret-manager
Edmond 6y ago
Lots of decent advise here already...be wary of a complicated (fancy product) approach and security theater....frankly when it comes to secret management the KISS principle should apply, with the caveat that it should be secure of course.
shifty1 6y ago
Generated by an automated process via service now and stored in cyberark
speedgoose 6y ago
We save the keys in a cloud vault. For the most important keys I also print them on paper, in text and in a QR-Code (that I generate with an offline tool). It is then placed into a physical safe. This is in case we lose access to our cloud vault, or if the keys are deleted from the cloud vault.
asldfhasdf2 6y ago
wrote our own kerberos-aaS clone with less features and vulnerable to more internal attacks than plain kerberos and more reliant in a central cert (not cert authority, cert), that is only used sporadically for cross services, not users (there's something else from major vendor there) and that team now keeps growing and the feature never improves :)
elwell 6y ago
Honeypot post, don't get doxxed
- Zezima 6y ago
  Seriously. Who would be dumb enough to answer this question when they know their key security is non-existent. Most people have their employer or work email on their HN profile. Really people?
- 2mol 6y ago
  Lol, I guess this should be the first answer of the thread. But also, at that point you're fully committed to security through obscurity, so folks should get at least the obscurity part right :)
sly010 6y ago
If you run on kubernetes it has first class support for secrets [0]. You can reference secrets in environment variables, or mount them as files in your containers. [0] https://kubernetes.io/docs/concepts/configuration/secret/ https://kubernetes.io/docs/concepts/configuration/secret/
- pricechild 6y ago
  Check out https://kubernetes.io/docs/concepts/configuration/secret/#risks https://kubernetes.io/docs/concepts/configuration/secret/#ri... - this isn't free.
ecesena 6y ago
We use Knox: http://github.com/pinterest/knox http://github.com/pinterest/knox Think of a lighter vault, with ACLs for people and/or machines to access keys and versioning to rotate keys. In our case, Knox depends upon AWS KMS to "lock/unlock" its storage.
- lux 6y ago
  This looks way easier than getting Vault setup! Unfortunately, googling for it returns a ton of info about Apache Knox which is an unfortunate name clash. I would love to see this catch on. Thanks for the share!
rogerkirkness 6y ago
We use Doppler. They're a YC Co. Like an easier to use Hashicorp. Has been great so far.
- bvallelunga 6y ago
  https://doppler.com https://doppler.com
mancini0 6y ago
We email them around lol. Wish we used istio.
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
BillinghamJ 6y ago
They are kept on Keybase >_< Sadly now forced to figure out what to move to. We're considering 1Password with it's CLI as a short term option, but will be wanting to move to Hashicorp Vault or similar on the mid-term
Dansvidania 6y ago
My company is too big for me to know about how everyone manages them (my guess would be sharepoint shared folders, with variable degrees of accesses). My team uses lastpass
vbhakta 6y ago
We use lastpass for sharing keys internally and we use AWS SSM ParameterStore, this section of AWS is only accessible by a few engineers.
deleted 6y ago
[deleted]
Faaak 6y ago
We use vault, but sometimes I just `openssl` gpg encrypt the secrets with the keys of all the members of my team and commit the .gpg to git. We all use yubikeys and use them to SSH. Not ideal, but it works... At least until one of us resign (but turnover is quite low here, so crossed fingers).
- nickbauman 6y ago
  You should be able to revoke the leaver's shared key from the keyring and reencrypt the secrets?
- ahnick 6y ago
  What is your process for handling key exchange with team members?
  - Faaak 6y ago
    when a new guy arrives to the company, we generate the keys on an air-gapped computer (with cahoskey et al) and upload the to the yubikey (they keep a separate encrypted usb key with the private keys). Then, some employees verify the new employee and sign their keys. There are then uploaded to teh keyservers and an internal mail is sent. Quite old school but it works quite well, alas we're small though (120).
nickbauman 6y ago
I have used EJSON and putting everyone's public key in a single keyring. Works but the secrets and keys end up taking up a lot of space.
nitwit005 6y ago
I've had these variations at work: Checked into the code encrypted, a custom secret vault, stored encrypted in a S3 bucket and downloaded/decrypted at startup. The main observation I'd make that if you put your keys somewhere only accessible in production, you've made it impossible to test anywhere except production. If you do that, you need to create a process where people can ship some small bit of code to test if the production key setup genuinely works (hint: it won't).
d0ne 6y ago
Disclosure: Founder https://dev.ionic.com https://dev.ionic.com Utilized globally by individual developers, large enterprises such as JP Morgan & Chase[1], and integrated into the KMS services such as Google Cloud[2]. 1. https://venturebeat.com/2019/02/27/ionic-security-raises-40-million-to-encrypt-files-accessed-on-the-cloud/ https://venturebeat.com/2019/02/27/ionic-security-raises-40-... 2. https://cloud.google.com/blog/products/identity-security/cloud-external-key-manager-now-in-beta https://cloud.google.com/blog/products/identity-security/clo...
- Predrag3141 6y ago
  dev.ionic.com is in particular an answer to, "how do you back up your encryption keys, or even put them in escrow somewhere?"
  - mfloyd5218 6y ago
    In particular, this solution implements a key vault that you can safely store keys using the security model provided by your OS (Windows, MacOS or Linux). The tool provides a CLI that lets you easily create and store keys from the command line.
uditj 6y ago
Used Doppler to store encryption keys and other secrets, and it worked well. Was pretty easy to setup and use their secrets store across our dev machines, CI, and production environments.
rgmvisser 6y ago
We use Doppler for our encryption keys, great way to easily store dev / staging / production keys in a secure and obfuscated way
bvallelunga 6y ago
A couple of our customers shared this thread with me so I thought I’d chime in. For transparency I am the CEO of Doppler (YC W19), a hosted secrets manager service. I know secrets management isn’t directly related to key management, but it’s a cool security topic we think about often. A one-liner about Doppler - lovable secrets manager built for the everyday developer, that works across all stages, from local development to production, on all stacks, and infras. For anyone who needs a super simple place to store their encryption keys that works with Heroku and has versioning, I think Doppler could help. It doesn’t have all the fancy (and really cool) features of KMS as it’s designed to be a kv store for secrets, but it could be helpful. We have a free tier for anyone who wants to try it out. https://doppler.com https://doppler.com
clintfred 6y ago
https://github.com/IronCoreLabs/ironhide https://github.com/IronCoreLabs/ironhide is a tool we built for managing our own developer secrets. It allows encrypted files to be checked in to git or stored elsewhere. You can think of it similar to gpg, with the upside that ironhide has the ability to change who can decrypt the secret without re-encrypting the data. Check it out and if you have any questions, feel free to ask here or open an issue in github. We also have a Rust version in the works for those interested in something native.
ComputerGuru 6y ago
We commit encryption keys, themselves encrypted, to git alongside the code and everything else. They’re fully versioned and therefore protected against data loss, and we don’t treat dev keys as any different from production keys (just stored in a separate file). I first wrote about it back in 2017 (1) and we released an open framework for multiple languages/frameworks (2). 1: https://neosmart.net/blog/2017/securestore-a-net-secrets-manager/ https://neosmart.net/blog/2017/securestore-a-net-secrets-man... 2: https://neosmart.net/blog/2020/securestore-open-secrets-format/ https://neosmart.net/blog/2020/securestore-open-secrets-form...
z3t4 6y ago
You need to have a threat model. Then work out a solution from that. Ask yourself, why are you encrypting the hard drivers ? One threat model might be: Burglars sneaking in during the night and stealing the hard drivers. Then you would store the keys on a different location then the disks. Then you make it a routine to reboot parts of the fleet, like scheduled simulation/training so that everyone knows what to do when you actually need them.
greyhair 6y ago
Hardware key control (HSM) with smart card controlled access. Locked in a vault. Keys are an actual secret key (within the HSM), unknown to any human. Two people with two different access cards have to be present to enable key operations. So, you don't have to worry about changing keys as employees come and go, because no one knows the actual keys. There is a whole structure of spare cards stored in offsite secure storage in case a card is damaged, lost, or stolen. Card set one is stored separately from card set two.
Shicholas 6y ago
GitHub Actions, Terraform Cloud, and strict RBAC for everyone on our team so they can't see/change these values on GitHub/Terraform/our cloud environments. This doesn't need to be hard.
bsder 6y ago
Lots of answers, and I still don't see anything valid for a 5 person startup--2 of whom are executive types.
matheusmoreira 6y ago
> I (we) really want to learn what a proper setup would look like Signing and authentication keys are expendable but encryption keys are worth keeping even after they've been rotated since decryption of existing data may be necessary. The key can be printed on paper and stored in a physical safe. Paper isn't a high density storage medium but it is remarkably durable and perfect for small amounts of data such as encryption keys. It also counts as an offline backup. Keys can also be printed as QR codes. They support error correction and enable automatic data restoration. Even 4096 bit RSA keys fit in a binary mode QR code and the smaller ECC keys allow use of high error correction modes, making the data even more durable. I wrote a binary decoding feature for ZBar in order to support this exact use case: zbarcam --raw --oneshot -Sbinary > key It's available on version 0.23.1 and above.
recursive 6y ago
On a USB drive in a locked cabinet. It's not a hardened cabinet. A hard yank would probably open it.
certera 6y ago
Plugging my project Certera as a means to manage keys used for Let's Encrypt certificates: https://docs.certera.io https://docs.certera.io You can rotate keys and facilitate key pinning scenarios. Cheers!
ykevinator 6y ago
We have a shared Dropbox account with 2fa hardware and no otp
paloswag 6y ago
Notepad
person_of_color 6y ago
Related question: How do big companies (FAANG) store their root private key?
zmmmmm 6y ago
We check many of our passwords into a git repository ... but they are all encrypted with unix pass with the public key of each individual in the team authorised to access. Our services require the gpg agent running to launch, allowing them to read the secrets.
tzs 6y ago
Shamir's Secret Sharing algorithm [1]. 4 people have shares, with 2 shares required to reconstruct the key. How the 4 shareholders store their shares is up to them. Mine is in a secure note in 1Password. [1] https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing
bulletsvshumans 6y ago
Wouldn't YOU like to know?
fortran77 6y ago
We check them into github! (sadly...)
ransom1538 6y ago
My humble security advice: 1) If you don't need it - don't store it. If you need it - but it needs to be encrypted - probably! don't store it. 2) Think one way hashes with salts, think deletion policies, rotations, small disk drives. 3) You will get owned!!
- dfc 6y ago
  I agree with only store what you need. But I can't think of many things in a professional setting that a company should store but not encrypt.
snow_mac 6y ago
We use Azure Key Vault
winrid 6y ago
Vault.
bashinator 6y ago
AWS KMS, and equivalent for other cloud providers. They are in every sense better qualified to handle encryption secrets than my lone ass is.
XorNot 6y ago
The dream for me has always been to get LDAP/Kerberos tied into container/VM orchestration, and do everything that way with per-instance accounts. LDAP is ubiquitous enough as an auth method (how do you auth to Vault? You auth to LDAP with it...) that any service you run or use is likely to speak it. Why this isn't done more often is a mystery to me and probably the number 1 source of credentials being baked into things accidentally: oh we need a service account into the <enterprise system> which uses Active Directory. Might have answered my own question there though.
pengaru 6y ago
There's a reason gpg supports --armor; print a backup copy of any important keys and stick it in a safe deposit box.
rataata_jr 6y ago
Lastpass?
sudhirj 6y ago
Some on a USB hardware dongle under lock and key. We also run HSMs and the key cards to those require quorum. So the CTO, Founder and head of devops each have multiple cards at home and work such that any two of them can make a quorum with all their cards, or all three can make a quorum with some of their cards.
delduca 6y ago
What about https://www.envkey.com/ https://www.envkey.com/?
based2 6y ago
anyone use https://www.rcdevs.com/products/spankey/ https://www.rcdevs.com/products/spankey/ ?
ahmetcetin 6y ago
we use lastpass
gerritjvv 6y ago
the simplest option is to store your keys in aws secrets manager (if you use aws), and then write some tooling around it. self promotion * You did ask how people do it :), this is my way, Ive written my own service which has been in production for more than 3 years, http://pkhub.io http://pkhub.io (if you would like to try it send me an email to admin@pkhub.io). This was before aws secrets manager, the tooling is usefull cause I wrote: running your app with its needed secrests dev/stage/prod, accessing dbs, downloading and installing ssh keys to ssh agent, utilities.. end of course you could write all these yourself with aws secrets manager. there is hashicorp's vault but tbh it always seemed like way to complicated to setup. my advice in general would be: to get something secure but simple enough that your engineers can do their work and access the resources they need, without the oh only bob has the keys on his laptop situation.
awayfromme 6y ago
Hi, this issue brought me back in the days when i was just a IT handyman in a small company. The priority of that company was to don't share keys or everything related to them with no one. For no one i mean, third-parties software of wherever a password or an encryption would be watched to someone. At the time, i thought this "obsession" was clearly a sign of mental illness, cause the company was very small and we were in the nowhere of nothing.(maybe nowdays i still think it). Our method was based on a selfmade MD5 encryption script using ruby on rails. Put your password into it, it print it on a datacoin blockchain that generates a univocal MD5 hash. This hash goes around 5 (later 6) local server those collect the encrypted key. Obviously these servers were without an internet connection, running only for internal company purposes.(such as this). For sure a weakness of this procedure was the slowness for obtain a new password or to change it. I think that the most secure place is where there'snt an internet connection. Thanks for bring back memories :D

varikin 6y ago

This just pushed the problem further down the stack. You should have keys to unlock vault when it is restarted. How do you secure those keys?

mmm_grayons 6y ago
I can't seem to understand how a "secrets manager" helps things. Could someone who does ELI5 why it's better than a config file with permissions locked down?
- bvandewalle 6y ago
  It makes sense at scale. If you are a company of two there are probably better solutions. At scale, you can very granularly define policies for each secret. When a secret is accessed, it is done so through a user or application identity. Each access is also logged.
  - scott00 6y ago
    So then how do you manage the secret that authenticates an application's identity? And what good is the logging if after an application has the secret it can do whatever it wants with it?
    - bvandewalle 6y ago
      if it is an instance on the cloud, GCP and AWS let you define ServiceAccounts that get populated on the Instance at boot time. you should only let the instance access the secret it requires.
      - rdslw 6y ago
        and how do you manage secrets that let you define that ServiceAccounts? As OP wrote, you did not solve it, just moved it to a different level.
- closeparen 6y ago
  In a monolithic application they're pretty close. In a microservices architecture, suppose you have 100 services and schedule 6 service containers per host. That is 94 services worth of secrets that don't need to be there, expanding the blast radius of single host compromise. There may be a credential on that host that can be exchanged for more secrets, but audit logs can tell you whether it actually was, and you can revoke it. Secrets may still need to go to config files for third party stuff, but you can write your in-house applications to fetch secrets at runtime and hold them only in memory. That's less opportunity for compromise vs. both memory and disk. Also have heard that Linux's process address space isolation is less prone to vulnerabilities than user account separation or filesystem permissions. Not really sure how true that is.
- LASR 6y ago
  The real difference over a file is that you some intelligence(the manager) running that can offer all kinds of additional security functionality. A typical deployment might involve placing the manager on a secure host that has access to generate and rotate keys, for example. The manager can then configured to re-generate keys and vend them on-demand to instances that need require them. You can configure these keys with very limited access, and also make them expiring. The manager then becomes effectively a keystore that can never export master keys, but only vends out some limited-scope keys to other instances. Other instances would have to authenticate using some pre-configured host keys or even be authenticated directly though the cloud provider. If your instances are compromised, the worst someone can do is to get access to a limited-scope key that will expire. Hopefully you have other measures in place that would prevent and detect someone from just sitting on an instance and sucking up all your data and exporting them.
  - mmm_grayons 6y ago
    Thanks, this makes a lot more sense and seems much more useful than the justifications I've heard others give. I appreciate it.
bvandewalle 6y ago
If you use Vault, you should use it as an RBAC system as well. That means that each application got a ServiceAccount (SA) and each user got a username/password. Based on your identity, you get access to specific secrets from Vault.
goodoldneon 6y ago
Store them in a separate instance of Vault.
closeparen 6y ago
You could do a lot worse than to have the operators store their shares in their separate password managers or on paper in safe places. It does admit the possibility that an operator's share could be copied. To work around that you can get a proper HSM that needs a quorum of smart cards presented to unlock. (The offline, low QPS, root of trust-oriented ones are not exactly cheap, but much cheaper than the network-attached ones targeting high QPS transactions). Vault Enterprise has PKCS#11 integration. With the Thales nShield stuff, you can replicate key material from one to another for redundancy while allegedly still preserving the "can't ever get keys out in plaintext" property. Not sure about others.
ruuda 6y ago
We put them in a password manager (1Password) to which multiple accounts have access. Each account is secured with a key, passphrase, and 2FA.
jmarcher 6y ago
You can use GCPKMS (probably AWS KMS too) to unseal Vault automatically. https://github.com/sethvargo/vault-on-gke/blob/master/terraform/k8s.tf#L203-L208 https://github.com/sethvargo/vault-on-gke/blob/master/terraf... The KMS ring itself is only accessible by Vault. People with high enough privileges for our Vault GCP project could technically grant themselves access to it, but on day-to-day business, nobody can view the project. At some other place, where we were using AWS, I wrote a script that would store encrypt unseal keys (need multiple due to shamir) via pgp using their keybase public key. IIRC you can store encrypted keys in Vault that can be accessed only for unsealing purposes (please correct me on this). When needing an unseal, the script on the user side would then decrypt the key and submit it to Vault for unsealing. It worked well enough, and it felt like being in the movie/game GoldenEye, but nowhere as slick as auto unseal. Regardless of the setup, yes, at the end of the day any solution is really just pushing the problem further down the stack.
perlgeek 6y ago
There are actually two answers to that: * You can split the key between different persons, and you can even implement "n of k" schemes, like you specify (at key creation time) that you need any 4 out of 9 shards to unseal the vault. You can then keep those shards on separate operator's laptops, in separate backup systems etc. * You can use a hardware security module to unseal the vault (support for that is not included in the free version, IIRC). But even if the vault wasn't stored encrypted, it'd still be a huge improvement over "keys on NFS", because only machine administrators get access to the whole DB, and you can limit and audit the access of everybody else in a sane manner.
greyhair 6y ago
We also use an HSM. Smart cards are held by two separate people, each card set is different. Both have to be present to restart the HSM. Our HSM can require up to five different cards be required for certain operations. For us, we only require two for normal operations. For HSM management (key generation, card authentication, etc..) we require a third card set member to be present. This protects against accidental key erasure, or fraud.
- deleted 6y ago
  [deleted]

panpanna 6y ago

If this is a public repo then you are already hacked. There are multiple automated systems that scan public repos for credentials. 5 minutes later you are mining bitcoins for them.

thieving_magpie 6y ago
Luckily I was in charge of setting up our remote repositories and made sure that by default they are all private repos, otherwise this would have happened without a doubt.
toomuchtodo 6y ago
Github monitors for public commits of service secrets. Not an excuse to commit secrets, but there is a bit of a safety net. > When you push to a public repository, GitHub scans the content of the commits for secrets. If you switch a private repository to public, GitHub scans the entire repository for secrets. > When secret scanning detects a set of credentials, we notify the service provider who issued the secret. The service provider validates the credential and then decides whether they should revoke the secret, issue a new secret, or reach out to you directly, which will depend on the associated risks to you or the service provider. https://help.github.com/en/github/administering-a-repository/about-secret-scanning https://help.github.com/en/github/administering-a-repository...
ganstyles 6y ago
In a boneheaded movei I accidentally committed my SendGrid creds to GitHub. Pretty quickly after, GitHub alerted me. However by then my SG account was sending thousands of automated spam messages. Those automated scammer systems are FAST. Not particularly germane to the discussion, but really disappointed in how SendGrid handled things. I notified them immediately, rotated all API tokens, and tey could not turn it off, so the spammer sent messages for days and eventually my SG account got suspended.
- jabart 6y ago
  My SG API keys, for an account that we terminated still send out emails if I happen to use an old config for a service. IP Rules are super helpful in this case, still need to rotate when exposed but can limit the exposure.
- bkandel 6y ago
  So this has happened at more than one company that I've been at, only difference is that these were AWS keys and used to mine bitcoin. AWS was actually pretty good about it, we rotated the keys as quickly as we could and they dropped all the charges.
  - meddlepal 6y ago
    I once worked for a co-founder who despite all the warnings I gave about not committing infra credentials to source control still went ahead and explicitly committed credentials to public source control because "the developer experience was better". The CEO was not pleased with the 30K (or maybe it was 60K) bill... and I just pointed at the CTO and was like "I fought this battle and was overruled"
    - mb7733 6y ago
      The most frustrating part of this is that the developer experience isn't better when you check credentials into source control. It seems convenient until the credentials change (which they ought to now and then). Then when you check out an old revision of the project, it is broken. You end up having to copy and paste the new creds back in time and it's finicky as hell.

AlfeG 6y ago

We have very simmiliar issue. All our databases have password Qwerty1234 Android keystore is checked in repository with access key in scripts. Security keys for external services are also checked in into repository. Some external services for production are managed by devs that are long time ago not working in our company

eitland 6y ago
Hehe. Less than 8 years ago I asked for help to add a column in a database at a company I helped. This was a few days after they met me for the first time. The company solved this by giving me a root username and password that worked on every single important database in the company, at least every customer database. I had to beg them to create a somewhat restricted account. The same company was however deeply sceptical to all kinds of remote work. The security equivalent of penny wise pound foolish I guess :-]
- AlfeG 6y ago
  On one my past job there were fingerprint reader system on enter to office. Almost 6 years later, I were still able to enter office with my fingerprint.
  - shoo 6y ago
    At one of my past jobs there was a fingerprint reader system to enter the office. It didn't work reliably to recognise fingerprints of employees, so after a while people settled on the solution of having a large brick next to the door which was used to wedge the door open during the daytime after the first person managed to get the door open in the morning.
    - acruns 6y ago
      I get this same thing with being invited to an Azure instance. years later I still have full access
- BrandoElFollito 6y ago
  Skepticism regarding remote work often comes from the fact that a company is not sure whether the employees work like they should (especially for larger companies). If they slack off, at least they do it in the office and not freely at home (imagine the possibilities!)
- tempestn 6y ago
  With root access I suppose you could have created a restricted account yourself!
  - prepend 6y ago
    I end up doing something like that whenever I’m granted root. I create a limited account, then delete the root or, if it’s shared, ask that the password be changed.
  - Fr33maan 6y ago
    I laughed so much at this
    - UweSchmidt 6y ago
      It's more than an absurdity, you have to do that to protect yourself from embarrassing mistakes. It prevents you from accidentally deleting or modifying the system in a difficult-to-restore way. If that happens, any security issue that existed becomes purely abstract and academic.
- needcaffeine 6y ago
  At a previous job they refused to give me database access and instead insisted I ask them whenever I needed any columns added/altered, however I did have access to the code to do my job and...mysql root credentials were committed to the repo. To keep the charade up I sent one of every 20 requests to them to do for me.
- vedant_shety 6y ago
  This feels awfully familiar. My current company uses the same generic username-password combination for every server. But we aren't allowed internet access on our workstations because "security"