15 ms·
Git Internals, Techniques, and Rewriting History
- acemarke 7y agoOh hey, that's my post! Glad someone found it interesting enough to submit. Don't expect this to hit critical mass by now, but happy to answer questions on the rewriting thing if anyone has any.
- funkattack 7y agoSlide #39 seems to use a graph from https://marklodato.github.io/visual-git-guide/index-en.html#reset https://marklodato.github.io/visual-git-guide/index-en.html#... , that is published under "Attribution-NonCommercial-ShareAlike 3.0 United States (CC BY-NC-SA 3.0 US)". This should be credited appropriately and your content should be shared alike.
- acemarke 7y agoYeah, I grabbed a bunch of images from various sources for the slides. Unfortunately, I was in a hurry and didn't record all the places I got them originally. I'll try to find time to go through and attribute things. Thanks for the heads-up!
- funkattack 7y agoIn my book "credited appropriately" reads like, put the site down till you have done your homework!
- acemarke 7y agoLook, I'm at work. I agree that it needs to be updated, and will do it as soon as I have time.
- jeanmichelx 7y agoCool post, i wish I could do something similar at work. I'm wondering why you went through the pain of making an HTTP server to process JS files, instead of say putting the temporary file over a ramdisk and running the formatters in parallel over multiple files. Plus, even if it's slow, you only have to run it a few times right?
- acemarke 7y agoA RAM disk never even crossed my mind :) I know they exist and have used one in the past, but it didn't occur to me as a possible tool. Also, I was doing this on Windows, and I don't even know if you _can_ set up a RAM disk on Windows. (edit and of course as I say that, someone literally just linked a Windows RAM disk FS driver upthread.) And no, I was re-running this _dozens_ of times to get it right, especially when I was in the "run -> transform fails on broken JS code -> debug -> write regex -> rerun" portion.
- jeanmichelx 7y agoAh makes sense you want to be fast then! Thanks for your answer
- dahart 7y agoReading the approach you took for reformatting code, I was curious whether after you’d done it, it still seemed worth using the python+libgit2 approach? Do you know how long the shell approach would have taken without using libgit2? Did it save time overall to use the lower level approach, including the presumably extra coding time? The code formatter was probably the limiting run-time factor, no? When I’ve used filter-branch and BFG in the past, I had to run the whole batch multiple times before I got it correct, and before I was sure the result was safe... did you have to do that, run the filter multiple times? Would you take the same approach if you had to do it again from scratch? I’m curious if I should invest in this approach next time it comes up for me.
- acemarke 7y agoYes, it was definitely worth it, for multiple reasons: - I'm a lot more comfortable with Python than I am with shell scripting - The Python+libgit2 approach _absolutely_ saved time. Being able to iterate through commits in 4-ish hours vs 24 hours for a complete run was huge, and yes, I did a lot of partial runs to test how the process was going. - Yes, the code rewriting portion ended up as the bulk of the time. I could do a no-op iteration of the commit history in about 30 minutes, and the final processing was about 4.5 hours. If I ever needed to do this again, yes, I'd use basically the same approach, just with the advantage of applying the knowledge I learned right away.
- deleted 7y ago[deleted]
- dang 7y agoWe changed the URL from https://blog.isquaredsoftware.com/2019/10/presentation-hooks-hocs-tradeoffs/ https://blog.isquaredsoftware.com/2019/10/presentation-hooks... to the slides which have the content.
- acemarke 7y agoNote that the original post links to an earlier article I wrote that goes into a lot further detail on the actual rewrite process: https://blog.isquaredsoftware.com/2018/11/git-js-history-rewriting/ https://blog.isquaredsoftware.com/2018/11/git-js-history-rew...
- samwestdev 7y agoany video archive of the presentation?
- acemarke 7y agoNo, I only gave this as an internal brown bag talk at work.
- ridiculous_fish 7y agoWhat are the best practices for extending git? Every example seems to be a shells script that calls out to git; is there a better approach?
- saagarjha 7y agoWhat kind of functionality are you trying to add? For something simply, shelling out to git is probably OK; if you're writing something more complicated many projects rely on libgit2. For really weird stuff you can probably just directly go and edit objects inside of .git, as long as you're careful.
- ridiculous_fish 7y agoI would like to extend git such that you can check out a commit, amend it, rebase all descendants onto the amended commit, and update any branches accordingly. Effectively like `rebase -i` but not modal. This requires keeping some state around. libgit2 is a nice pointer, I'll check that out. (hg users will recognize this as "restacking.")
- pvinis 7y agoExtending how? Are you looking for automation or new commands or something else?
- bloopernova 7y agoYou can add a section called alias to your ~/.gitconfig [alias] lol = !git --no-pager log --graph --decorate --abbrev-commit --all --date=local --pretty=format:\"%C(auto)%h%d %C(blue)%an %C(green)%cd %C(red)%GG %C(reset)%s\" Then you can run "git lol" and get that cool output. That's one way to "extend" git.
- hnrodey 7y agogit aliases custom shell scripts use a Git library in your language of choice and write something
- emmelaich 7y ago
- mederel 7y agoThe slides don't display well on mobile device
- rudiv 7y agoSeconded; the text does not wrap in portrait orientation on small breakpoints and trying to scroll trigger the progression of slides.
- acemarke 7y agoSorry, mobile formatting was not something I was concerned about when I was putting these together. I presented them from my laptop, so that was all that mattered at the time.
- Phrenzy 7y agoDo I sense an upcoming fork? ;)
- pikzel 7y agoWould have been nice if we could scroll through the pages and not having to use the arrow keys.
- mettamage 7y agoI find the diagram on page 15 to get an overall feel/flow for basic git usage a very good diagram. If I'd still be teaching at a coding school I'd hand it out to my students.
- rinchik 7y agoIsn't a bit off? It shows "git diff" as a diff between staging and workspace when it should be between workspace and local repo.
- mettamage 7y agoWhoops, I think you're right. I skimmed over git diff, as when I use git diff I constantly re-figure out for what it precisely applies again. I just care to remember "it diffs".
- pjc50 7y agoNot quite: if you stage something, it no longer appears in "git diff" and you have to use "git diff --cached" to show it. One of those things which I trip over occasionally. "git diff --cached" = staging vs. HEAD "git diff" = workspace vs. staging (but when nothing is staged, staging == HEAD) (I think?)
- rinchik 7y agoYep. you are right, "but when nothing is staged, staging == HEAD" is exactly what i had in mind
- ethomson 7y agoNo, this slide is correct. git diff (without any arguments) shows you what you haven't staged. In other words, the different between the working folder (which here is called "workspace") and the index.
- darekkay 7y agoI always found "rewriting history" misleading. Git is immutable by default, so you cannot "rewrite" the history. In fact, you are creating an "alternate history" (which the article mentions). While the difference appears subtle, it takes away the fear of users who never use any "altering" commands because they think they might lose their changes or mess up something.
- chrismorgan 7y agoThe most common scenario is that the original history is immediately discarded, by virtue of no longer having any refs pointing to it—though it lingers in the reflog for a while before being garbage collected. In that situation, I think it’s quite reasonable to call it rewriting history, because the original history no longer exists; for the term “alternate history” to have much meaning, the original must still exist, to be compared to. I’m looking at the long-term picture here, rather than how you interact with it when in the process of rewriting history. Clarifying that you’re replacing history rather than modifying it is useful, but I certainly have no beef with the expression “rewriting history”.
- dahart 7y agoYou’re right in the case of rebase, and I like the point about making alternate histories, but I don’t mind the word ‘rewriting’ because the net effect is to modify the history that ‘master’ points to. I assume the downvotes are due to the article’s main point being about true history rewriting; using filter-branch and BFG isn’t creating an alternate, it’s truly re—writing. Anyway, personally, I’d pick on the other word sometimes, and suggest we don’t call it rewriting ‘history’ when rebasing local changes before push. Maybe it’d be better to define “history” to mean things that are pushed and shared with others, and call commits that haven’t been shared something like “local changes”. Might be moot in this case since the article was mainly about rewriting the entire repo, so “history” is totally appropriate, but still perhaps nice to use terminology to distinguish between commits that have been shared from commits that haven’t, since that is the line across which one should be more careful with any kind of rewrite.
- pjc50 7y agoI love how the first real slide goes straight into the fake manpage generator: https://git-man-page-generator.lokaltog.net/ https://git-man-page-generator.lokaltog.net/ "App Repo Size Issues:" yes, this is the Achilles heel. By being a full distributed system where every client has to carry a full copy of the history of everything, some common practices become unsustainable. I've had two employers that checked build artefacts into SVN, for example: do that with git and the repo becomes unusably large very quickly. Vendoring dependencies, a useful practice if your project is slow-moving, will also bloat the repo. They should be using "shallow clone" for Jenkins, but even that can be surprisingly large. I've also been through the "apply BFG to repo" phase (very time-consuming, and blocks all commits while you're doing it!) > New idea: run the formatter against every commit in the history, so that it looked as if the code was "always formatted the right way". This is actually brilliant, for the reason they give - keeping "credit" assigned correctly to original commits. > Determined it was okay if older commits were potentially "broken", as long as the latest commit runs and has all of our changes as of late 2018 I'm less OK with this, as the chances of an automatically introduced horrible bug which you can't trace seem rather high and you've wrecked any chance of using git-bisect! But if the "tip" is the only supported released version, I suppose it's less critical. It's also interesting that most of the speed issues are addressed by re-architecting to avoid syncing to disk. If there was an easy Windows RAMdisk this might have made almost as much of a difference.
- acemarke 7y agoAgreed on all points. We happen to be in a situation where we have very long development cycles, and also rarely need to go back and rebuild older versions. So, in our case we could get away with this. And yes, the speed improvements came from a combination of iterating over Git commits in a single Python process vs running Git as an external command, and avoiding touching disk for everything except reading and writing blobs and commits.
- jacobush 7y agoBut there is an easy RAMdisk for Windows! http://www.ltr-data.se/opencode.html/#ImDisk http://www.ltr-data.se/opencode.html/#ImDisk
- aequitas 7y agoMy collegae told me I'm the guy with his phone number in the git.txt file, don't know if that's a compliment? But whenever I try to explain Git to someone I try to step away from a computer and just work out the problem on a whiteboard. Using post-it's for branches/HEAD/tags and marker to write down the commits and the commit-tree. Preferrably permanent marker to reflect the permanent nature of commits with regards to branches (eg: rebasing keeps the original commits around). Also I ditch files in favor of a picture of a cat there changes are attributes like added body parts or toys since most people like cats better than files. Taking it away from a computer really helps to reduce the complexity involved and thinking about what manipulations you make to the Git 'database' visually really helps understanding the concepts imho. Just going through it step by step as if you where the Git binary making the changes, and after a while the Git commands turn from hard to remember trivia to tools in your toolbox.
- pjc50 7y agoConversely, this is a massive indictment of the git user interface. People used to talk about "CASE tools": Computer Aided Software Engineering, by analogy with the CAD tools that replaced physical drawing tables once the technology got good enough. I joke that certain tools are "computer impaired software engineering", and git can certainly belong in that category at times. What we have built is a tool with a good internal model but a user interface that it's easier for you to not use a computer, to step away to the whiteboard, in order to work out what to do. We don't tolerate this for, say, word processors; people don't go to a piece of paper to lay out their headings and only then work out the incantations necessary to achieve that result. (Has anyone attempted to make a "transparent porcelain" GUI which represents the "intuitive" internals in an actually explanatory way? The success of which could be measured by the reduced number of mistakes and amount of apprehension experienced by non-expert users?)
- aequitas 7y agoVery true. But the word processor example is a little scewed. It depends on the audience and the value they want to get from the tool, take Latex for example which is widely used writing scripts, being far away from WYSIWYG. Have a look at Gitless[0]. It's a layer on top op Git which simplifies/rearranges some of the concepts and commands of Git while still manipulating the Git DB in the same way (you can switch back and forth between Git and Gitless seamlessly). [0] https://gitless.com/ https://gitless.com/
- rsp1984 7y agoSafari user here. For some reason the navigation arrows are missing and I can't get past the first page...
- acemarke 7y agoSorry! I built it using the Spectacle web slides toolkit, and have only looked at it in Chrome and Firefox because that's what I use myself.
- dreamcompiler 7y agoImpossible to resize and read on my phone. It amazes me when a developer writes a presentation about a developer tool using a goopy JS-dependent infrastructure that apparently has never been tested on mobile.
- acemarke 7y agoAs I said down-thread, my only immediate concern when I put the presentation together was presenting it from my own laptop. I appreciate that folks are wanting to read it on mobile, but I've got a lot of stuff on my plate (primarily work around maintaining Redux), and modifying my slides to work better on mobile has not been a priority.
- dreamcompiler 7y agoFair enough. Sorry about not reading before commenting.
- acemarke 7y agoNo worries. The complaints about it not being readable on mobile are legit. If I could wave a magic wand and make it all magically responsive, I would. But, my original goal was simply to actually make the slides for the presentation, and show them during my talk. I do specifically use the Spectacle React/JS toolkit, as I _want_ to publish the slides online later for viewing on the web (one of several reasons why I don't make them in PowerPoint). But, part of the reason I can get away with that is that publishing them on my blog is just a matter of uploading the built assets and moving on, vs having to convert them from Powerpoint by hand or something. I've got a ton of other priorities and tasks to deal with, and figuring out what's needed to make the slides well-formatted on mobile realistically isn't anywhere on that list. Honestly, this thread is the first time anyone's actually complained about that. I'm not sure how much of the formatting issue is due to Spectacle's own styles, vs the typical slide layout that I have in there (which is mostly flexboxes with two items side-by-side). If anyone has some specific suggestions on how to alter the styles to make them work better, I could try to apply those and rebuild it.
- emilfihlman 7y agoNote: the presentation has ui flaws and does _not_ fit on a 3:2 screen, so you'll have to zoom to about 80% for it to be visible. This is apparent on slide 8.