6 ms·
Ask HN: Best way to prepare a codebase for open-source?
Hi HN,
How would you approach the process to opensource a proprietary codebase, especially regarding things such as ensuring that no secrets are sitting somewhere in the history.
I'd be tempted to publish without history but I feel like a lot of important context will be lost.
- edderkopp 5y agoBFG Repo-Cleaner (https://rtyley.github.io/bfg-repo-cleaner/ https://rtyley.github.io/bfg-repo-cleaner/) scans the history for secrets and removes them.
- Raed667 5y ago99.9% of people will only care about the current version. So don't worry too much about history, people mostly just want stuff that is accessible and works. Edit for clarification: You can delete the history for the open-source version and publish it with a fresh history. And keep the original history internally in case it is ever needed.
- dschuessler 5y agoThis is breathtakingly bad advice from a security perspective. If secrets might sit around in a repository you don't just publish it and bet on people not looking for them, at least not if data security is of any worth to you.
- Raed667 5y agoThat is not what i said. I said he should delete history and publish a new version with a new commit "v1"
- dschuessler 5y agoYou literally edited that in after my answer.
- winter_squirrel 5y agoHe added clarification, but his original comment still implied the same thing
- madeofpalk 5y agoGosh I wish there was a history available so I could understand all this context!
- OJFord 5y agoRead it in the context of OP's question - GP is suggesting squashing all history and not worrying about it, as OP alluded to.
- dschuessler 5y agoYes, I see how it can be understood that way. GP has edited it anyways to clarify this point.
- kissgyorgy 5y agoHistory is way more important than the current version of the code.
- thrower123 5y agoI've had managers tell me this when we've had to cut over from SVN to git. Maybe it's true for them... For me out on the pointy end of the spear, it's not hard to bring that history across, you just have to know how to do it, and it's invaluable when you're trying to figure out bugs.
- blagie 5y agoIt's not a numbers game. The remaining 0.1% can be very important too. There are times to kill history, and there are times to bring it along. History does a lot of things. - Giving people credit. - If, in 20 years, you want to pull up an old (no-longer-compatible) document - Public patent prior art - ... and a swarm of others "Internally" is likely to no longer exist a decade down the line. Whenever I've open sourced, I've brought along history. In one case, we did need to rewrite history to remove a few sensitive files, but git makes that easy enough. If keeping history is a heroic effort, by all means kill it. However, that's not a good generic best pracitce.
- Jugurtha 5y agoI recently put something on GitHub that we wrote for our product. The repo was on GitLab. You can add a git remote and push and it will keep your git history. I wrote the library because the issue it solves in MinIO's Python client was marked as "won't fix" and it has been useful for many people (we put it on PyPI before adding it to GitHub), and I was glad a few days ago to see that MinIO added something very similar to their Python client (they added a Python wrapper just like bmc). https://github.com/ikodotai/bmc https://github.com/ikodotai/bmc
- hvgk 5y agoClone it then push it to a clean repo to destroy the history. Seriously. I understand security concerns but the history could expose previous bad practices which can be applied to other products you run. Bad patterns are endemic. Also it gives you the chance to secret scan it and remove any swearing and embarrassing comments (I haven’t seen a codebase without any yet)
- hobo_mark 5y agoKeep the proprietary history to yourself, publish just a squashed snapshot of the state you want to make public, use git-replace [1] to link the public history with the (hidden) private one when you want full history, develop only on the public one from now on. [1] https://git-scm.com/book/en/v2/Git-Tools-Replace https://git-scm.com/book/en/v2/Git-Tools-Replace
- infogulch 5y agoI've never seen git replace before, very interesting! Can you use git-replace to edit an old commit message without rebuilding the whole commit graph and instead by just adding a new commit? The article seems to imply that you can use it to "truncate" history but still be able to browse it if you happen to have the full history, or something. I'll have to look more closely into exactly what's going on.
- throw_away 5y agoI think this just changes your own local view of the history. So you can fix the message, but in order for anybody to see it on push, you'd need to reify the changes with git-filter-branch & rebuild. https://www.oreilly.com/library/view/git-pocket-guide/9781449327507/ch10.html#_keeping_it_real https://www.oreilly.com/library/view/git-pocket-guide/978144... https://blog.plover.com/prog/git-replace.html https://blog.plover.com/prog/git-replace.html
- blagie 5y agoI haven't tried, but my read was "yes" when I looked at the intro, and "no" once I got an illustration of the data structure. There is something ambiguous in one or the other. I'm leaning towards "no." The text doesn't make sense, and the data structure does. So I think you still rebase; you can just keep both versions around.
- exikyut 5y agoHere's a possible practical idea: 1) run the full contents of HEAD into a space-separated list of tokens considered "okay". 2) dump out the full history into a space-separated list of tokens, filter out everything in the "okay" list, and list what's left. You might want to set up some sort of incremental regex filter thing to chew through the list efficiently. But if you implicitly trust HEAD as "incontrovertibly okay" this might filter out a lot of tokens for you.
- marto1 5y agoOne really important part people forget about is providing a development workflow. How should people reach out to you ? How can they make patches ? Report security flaws ? Which design principles are you following and are you open to change these ? How will this discussion be facilitated ? Making something available publicly is a far shot from actually having an open source project in my opinion. Good luck in your endeavors, it's a lot of work!
- protomyth 5y agoIf it is an old codebase that you are not sure of the origin of all the code, then you need to do a code audit and maybe even consult a lawyer. Do not publish the history. Fresh start for open source.
- trebligdivad 5y agoAs well as cleaning up/squashing history; consider: a) Make sure it's actually all your code - not code you copied from some other closed source project, or a contractor gave you 3 years ago. b) Remove the questionable comments about other employees and their managers c) If you're removing history you are removing some of the rational about why things are like they are - that does make it harder for people in the future to change things
- andrew_ 5y agoTangential, but still relevant; Once you decide on the cleanup for open sourcing, choose a path that's comfortable for YOU, the maintainer, and clearly define contributing rules/guidelines. Take breaks, and feel free to let the community know. And remember, it's only a hobby, nothing to get stressed or burnt out by.
- PaulAJ 5y agoOn secrets: you should be cycling all your secrets on a regular basis anyway. If you aren't, now might be a good time to start. In addition to the good advice here, you might want to check for anything potentially embarrassing, such as offensive language in comments, identifiers or commit comments. Some "tech bros" can be remarkably dumb about that stuff. Of course if you developed all this yourself, no problem.
- mrslave 5y agoAnother reason to squash into a single commit: not all of your developers want their name & work email address published without their consent.
- ufmace 5y agoI'd worry about the licensing too. What open-source license would you publish it under, and do you actually have permission from the owners and everyone who participated in writing it to publish under that license?
- johannes1234321 5y agoAside from the code: Know your intention. What do you want to achieve by open sourcing? Are you trying to grow your username, gain trust or is it just throwing code over the fence so users can go on while you focus elsewhere. Do you hope for contributions or do you want to continue driving the project? From there you can derive the community management you have to do. The more involvement you want from externals the more you have to invest in community management. (The more you want the quicker and more thorough you have to respond) Then be aware of all the legal things. When using libraries: anything lgpl or gplnlicensed stuff with something incompatible etc.? And then for secrets best is indeed squashing the history. Makes it in the beginning a bit annoying, but you might have code comments or commit messages referring to customers or have experiments with libraries of unacceptable license or whatever in there. Limiting review to recent state is a lot simpler.
- bumblebritches5 5y agoTo remove secrets you can use git filter-repo to rewrite commits
- joshxyz 5y agoCI tests are nice, unlimited use of GitHub Actions for public repos