4 ms·

My developers use CVS. Sometimes, they are really stupid, and they will checkin passwords. With the RCS archives, I can use vi (or nano, or any other reasonab

by chasil 4y ago

My developers use CVS. Sometimes, they are really stupid, and they will checkin passwords. With the RCS archives, I can use vi (or nano, or any other reasonable editor), and remove this foolishness. When I run "git cvsconvert" any foolishness is ENGRAVED IN STONE. Removal is possible in git, but not easy. This is my problem. THERE ARE SO MANY IDIOTS. What can I do? EDIT: For Windows-centric users of git, you need to run this in every and all repositories RIGHT NOW. git grep -i password $(git rev-list --all) [actually, everybody should try it]

AnonCoward42 4y ago
I might be an idiot myself, but would `git rebase -i <commitBeforeSnafu>` do the trick? edit: This is interactive rebase, where you can rewrite history. Instructions are in the file you are editing once the command is executed. As a rule of thumb, don't rebase on production, however with such a SNAFU you probably need to.
- gizmo686 4y ago
  No. The old commits still exist, a rebase just makes it so that they are not part of the history of the current branch. If you know the commit hash of a commit containing the offending data, you can still check it out. You can also find such commits in the reflog, or if they are part of the history of any other branch or tag. The data is also in git's internal data structures if you know enough to interpret those directly. Still, a rebase is a critical part of fixing this. If you rebase every branch that has the offending data in their history, and recreate any tags that have the data, then the offending data would be present only in unreachable objects (which can still be found through the reflog or knowing a commit hash). Eventually, git's garbage collection will clean up such unreachable data. You can force this behaviour earlier by using the `git gc` command. By default git will not remove recent objects (which is why you can reflog your way out of a botched rebase), but you can override this behaviour as well. Of course, all of the above assumes you are working on your own repository. Given git's decentralized design, you need to clean up and garbage collect on every clone that did a fetch since the offending data was checked it. Worse, each clone also keeps track of remote branches, and considers those to be reachable as well, so a garbage collection will not work correctly until a given clone fetches from all of its remotes after those remotes removed the offending data [0]. Further, there is not good tooling to check that you actually did this correctly, so when you are done, you need to hope you fully purged the data. The plus side of all of this is that once something is checked in, it is extremely difficult to accidently delete it. The downside is that it is also difficult to deliberately delete it. Also, it is fairly easy to make it difficult enough to find to negate the day to day benifets of it still existing, which does very little to protect against a motivated attacker. [0] Fourtuantly, you do not need all of the remote to have garbage collected, so you could have every make the data unreachable, fetch on all remotes, then garbage collect.
- stu2b50 4y ago
  Rebasing doesn't rewrite history, it creates an entirely new history, but all of the old commits still exist unless they are reaped by the GC at some point.
ramadis 4y ago
Chill?
- chasil 4y ago
  What is really far more critical to understand is that my vi changes what everybody sees in CVS. If there is some foolishness, git propagates it EVERYWHERE. Every git repository is equal. I have no control of proliferation. Sigh.
  - alvarlagerlof 4y ago
    It's more useful to fix the problem before it happens. Try to set up systems or workflows that make it easy to do the right thing, rather than trying to fix it afterwards.
    - chasil 4y ago
      I really had to converse with the lead in this wise: If I stapled your HR record, including your bank account and routing number, to our shared refrigerator so I could remember it, would you think that appropriate? Really, this was the context of our meeting. Now I am left with the aftermath of his departure, and knives at my sides.
pluijzer 4y ago
You can try and talk to your developers with more respect. People make mistakes and don't have to be idiots or stupid for it. edit: mistakes happen always it is better to change the workflow in such a way that a the consequences can be contained.
- chasil 4y ago
  - paraknight 4y ago
    "If everywhere you go smells like shit, maybe it's time to check your own shoes"
  - electrondood 4y ago
    > They quit far too much for any rapport. The developers you publicly describe as stupid and idiots seem to leave every three months. Huh.
    - Ancapistani 4y ago
      Forcing them to use CVS is probably a big part of it, too.
orf 4y ago
Fix whatever processes, tooling, culture or training you have that leads your developers to make these mistakes
sxg 4y ago
Multiple people committing the same mistake is much more likely to be a systems problem rather than an individual developer's problem. The other reply comments are saying the same thing, but your (parent) comment needs to be called out.
- deleted 4y ago
  [deleted]
lstamour 4y ago
Stop using any passwords that get checked in. Set expiry dates for access tokens. Set up git commit hooks that scan for password or API key prefixes and block the commit. Set up appropriate .gitignore files, you can place them in subfolders to keep them simple. If possible, switch from using passwords or tokens without expiry to using ephemeral or time-limited tokens such as machine or pod identities, JWT tokens, IAM service accounts, public/private key pairs (if you can get by with only a public key in the repo) or two-factor authentication. Consider distributing time-limited passwords with Hashicorp Vault. Some teams might use a cloud-hosted secrets manager or password manager like 1password to distribute passwords and then have code load the password as it runs on a developer machine. GitHub has a secrets scanner, if you pay them enough money for “Advanced Security” such that secrets can never be pushed upstream if recognized as such. Also, converting from any repo format to another repo format requires care and multiple repeated attempts. A lot can go wrong. Reposurgeon (from a sibling comment) is highly recommended but also not easy to use. It takes a lot of attention to detail to really get the details right.
- User23 4y ago
  Git filter-branch will erase your shame. But as others have said, those credentials are burnt and need to be treated accordingly.
  - CrazyPyroLinux 4y ago
    These days https://github.com/newren/git-filter-repo https://github.com/newren/git-filter-repo is superior.
- chasil 4y ago
  Just kill me now. We had the same problem in TFS. I found them, I warned them, my riot got us kicked off our shared server. We migrated to our corporate TFS cloud, and they kicked us off for precisely the same reason. This just won't/can't stop.
tigerlily 4y ago
Don’t use cvsconvert. Use reposurgeon, you will get much more fine grained control. Here’s a brief précis on getting started: https://gist.github.com/ryfactor/f70529438f254d44c00771765084d5fb https://gist.github.com/ryfactor/f70529438f254d44c0077176508...
- chasil 4y ago
  I don't think that we will move, and I likely will not assume TFS, but I will look.
CrazyPyroLinux 4y ago
Upvoted. Why in the world was this flagged? I don't think it's off-topic, since the topic is particularly relevant to git newbies, and the committing of stuff that shouldn't be in the repo is a huge problem I've observed in that group. (Not just credentials, but all sorts of junk like binaries and logs and tempfiles, and individual settings.) It's such a common problem that there's been many tools and workflows that have been set up to mitigate it. This situation often results from not having those tools and workflows in place to give even the "idiots" a "pit of success" to fall into. e.g., pre-commit hooks installed on dev machines by team policy, and pre-receieve hooks on central repos, both along with CI jobs running truffleHog etc. And even that is downstream of proper code design of minimizing credential use, and focusing/localizing it into files which are already .gitignore'd

by chasil 4y ago

AnonCoward42 4y ago

I might be an idiot myself, but would `git rebase -i <commitBeforeSnafu>` do the trick? edit: This is interactive rebase, where you can rewrite history. Instructions are in the file you are editing once the command is executed. As a rule of thumb, don't rebase on production, however with such a SNAFU you probably need to.

gizmo686 4y ago
No. The old commits still exist, a rebase just makes it so that they are not part of the history of the current branch. If you know the commit hash of a commit containing the offending data, you can still check it out. You can also find such commits in the reflog, or if they are part of the history of any other branch or tag. The data is also in git's internal data structures if you know enough to interpret those directly. Still, a rebase is a critical part of fixing this. If you rebase every branch that has the offending data in their history, and recreate any tags that have the data, then the offending data would be present only in unreachable objects (which can still be found through the reflog or knowing a commit hash). Eventually, git's garbage collection will clean up such unreachable data. You can force this behaviour earlier by using the `git gc` command. By default git will not remove recent objects (which is why you can reflog your way out of a botched rebase), but you can override this behaviour as well. Of course, all of the above assumes you are working on your own repository. Given git's decentralized design, you need to clean up and garbage collect on every clone that did a fetch since the offending data was checked it. Worse, each clone also keeps track of remote branches, and considers those to be reachable as well, so a garbage collection will not work correctly until a given clone fetches from all of its remotes after those remotes removed the offending data [0]. Further, there is not good tooling to check that you actually did this correctly, so when you are done, you need to hope you fully purged the data. The plus side of all of this is that once something is checked in, it is extremely difficult to accidently delete it. The downside is that it is also difficult to deliberately delete it. Also, it is fairly easy to make it difficult enough to find to negate the day to day benifets of it still existing, which does very little to protect against a motivated attacker. [0] Fourtuantly, you do not need all of the remote to have garbage collected, so you could have every make the data unreachable, fetch on all remotes, then garbage collect.
stu2b50 4y ago
Rebasing doesn't rewrite history, it creates an entirely new history, but all of the old commits still exist unless they are reaped by the GC at some point.

ramadis 4y ago

Chill?

chasil 4y ago
What is really far more critical to understand is that my vi changes what everybody sees in CVS. If there is some foolishness, git propagates it EVERYWHERE. Every git repository is equal. I have no control of proliferation. Sigh.
- alvarlagerlof 4y ago
  It's more useful to fix the problem before it happens. Try to set up systems or workflows that make it easy to do the right thing, rather than trying to fix it afterwards.
  - chasil 4y ago
    I really had to converse with the lead in this wise: If I stapled your HR record, including your bank account and routing number, to our shared refrigerator so I could remember it, would you think that appropriate? Really, this was the context of our meeting. Now I am left with the aftermath of his departure, and knives at my sides.

pluijzer 4y ago

You can try and talk to your developers with more respect. People make mistakes and don't have to be idiots or stupid for it. edit: mistakes happen always it is better to change the workflow in such a way that a the consequences can be contained.

chasil 4y ago
- paraknight 4y ago
  "If everywhere you go smells like shit, maybe it's time to check your own shoes"
- electrondood 4y ago
  > They quit far too much for any rapport. The developers you publicly describe as stupid and idiots seem to leave every three months. Huh.
  - Ancapistani 4y ago
    Forcing them to use CVS is probably a big part of it, too.

orf 4y ago

Fix whatever processes, tooling, culture or training you have that leads your developers to make these mistakes

sxg 4y ago

Multiple people committing the same mistake is much more likely to be a systems problem rather than an individual developer's problem. The other reply comments are saying the same thing, but your (parent) comment needs to be called out.

deleted 4y ago
[deleted]

lstamour 4y ago

Stop using any passwords that get checked in. Set expiry dates for access tokens. Set up git commit hooks that scan for password or API key prefixes and block the commit. Set up appropriate .gitignore files, you can place them in subfolders to keep them simple. If possible, switch from using passwords or tokens without expiry to using ephemeral or time-limited tokens such as machine or pod identities, JWT tokens, IAM service accounts, public/private key pairs (if you can get by with only a public key in the repo) or two-factor authentication. Consider distributing time-limited passwords with Hashicorp Vault. Some teams might use a cloud-hosted secrets manager or password manager like 1password to distribute passwords and then have code load the password as it runs on a developer machine. GitHub has a secrets scanner, if you pay them enough money for “Advanced Security” such that secrets can never be pushed upstream if recognized as such. Also, converting from any repo format to another repo format requires care and multiple repeated attempts. A lot can go wrong. Reposurgeon (from a sibling comment) is highly recommended but also not easy to use. It takes a lot of attention to detail to really get the details right.

User23 4y ago
Git filter-branch will erase your shame. But as others have said, those credentials are burnt and need to be treated accordingly.
- CrazyPyroLinux 4y ago
  These days https://github.com/newren/git-filter-repo https://github.com/newren/git-filter-repo is superior.
chasil 4y ago
Just kill me now. We had the same problem in TFS. I found them, I warned them, my riot got us kicked off our shared server. We migrated to our corporate TFS cloud, and they kicked us off for precisely the same reason. This just won't/can't stop.

tigerlily 4y ago

Don’t use cvsconvert. Use reposurgeon, you will get much more fine grained control. Here’s a brief précis on getting started: https://gist.github.com/ryfactor/f70529438f254d44c00771765084d5fb https://gist.github.com/ryfactor/f70529438f254d44c0077176508...

chasil 4y ago
I don't think that we will move, and I likely will not assume TFS, but I will look.

CrazyPyroLinux 4y ago

Upvoted. Why in the world was this flagged? I don't think it's off-topic, since the topic is particularly relevant to git newbies, and the committing of stuff that shouldn't be in the repo is a huge problem I've observed in that group. (Not just credentials, but all sorts of junk like binaries and logs and tempfiles, and individual settings.) It's such a common problem that there's been many tools and workflows that have been set up to mitigate it. This situation often results from not having those tools and workflows in place to give even the "idiots" a "pit of success" to fall into. e.g., pre-commit hooks installed on dev machines by team policy, and pre-receieve hooks on central repos, both along with CI jobs running truffleHog etc. And even that is downstream of proper code design of minimizing credential use, and focusing/localizing it into files which are already .gitignore'd