5 ms·
Going down the rabbit hole of Git's new bundle-URI
- jedimastert 2y agoThis actually might solve a massive CI problem we've been having...will report back tomorrow
- jwpapi 2y ago[flagged]
- autarch 2y ago> This has resulted in a contender for the world's smallest open source patch: Hah, got you beat: https://github.com/eki3z/mise.el/pull/12/files https://github.com/eki3z/mise.el/pull/12/files It's one ASCII character, so a one-byte patch. I don't think you can get smaller than that.
- ZeWaka 2y agoThat's a line modification, so presumably you'd count just an insertion or just a deletion as 'smaller'.
- autarch 2y agoYes, but so is the PR shown in the article. You're not going to get a diff that's less than one line unless you are using something besides the typical diff and patch tools.
- san1t1 2y agoMy smallest PR was adding a missing executable file permission.
- Izkata 2y agoIn terms of file contents, I have a bugfix commit somewhere that just adds a single newline to a config file (no line deleted in the diff). We were using something that had its own own custom parser for what looked like an INI file, and we managed to find an edge case where the wrong number of blank lines would break it.
- timdorr 2y agoSure you can: https://github.com/timdorr/-/commit/9e5a571abd3fc4f8714e8c401076e10338e0bdce https://github.com/timdorr/-/commit/9e5a571abd3fc4f8714e8c40...
- retroflexzy 2y agoThere is a cursor rendering fix in xf86-video-radeonhd (or perhaps -radeon) that flips a single bit. It took the group several years to narrow in on.
- deleted 2y ago[deleted]
- immibis 2y agoThe Dolphin patch that fixed the heat haze in Dragon Roost Island was also a single bit, changing a 3 to a 7 in source code, and also took several years. https://dolphin-emu.org/blog/2014/01/06/old-problem-meets-its-timely-demise/ https://dolphin-emu.org/blog/2014/01/06/old-problem-meets-it... This comment has been delayed by about 12 hours due to the Hacker News rate limit.
- wavemode 2y agoA commit which does nothing more than change permissions of a file would probably beat that, from an information theory perspective. You might say, "nay! the octal triple of a file's unix permissions requires 3+3+3 bits, which is 9, which is greater than the 8 bits of a single ascii character!" But, actually, Git does not support any file permissions other than 644 and 755. So a change from one to the other could theoretically be represented in just one bit of information.
- ithkuil 2y agoOk. But you need to specify the name of the file. So it also needs to be a one character filename. I wonder if file deletion is theoretically even smaller than permission change.
- arghwhat 2y ago> So it also needs to be a one character filename. No one counts the path being changed, just like they also don't count the commit message, committer and author name/email, etc.
- chungy 2y ago> But, actually, Git does not support any file permissions other than 644 and 755. So a change from one to the other could theoretically be represented in just one bit of information. Git also supports mode 120000 (symbolic links), so you could add another bit of information there.
- masklinn 2y ago> Git does not support any file permissions other than 644 and 755. It does not but kinda does, git stores the entire unix mode in the tree object (in ascii encoded octal to boot), it has a legacy 775 mode (which gets signaled by fsck —strict, maybe, if that’s been fixed), and it will internally canonicalise all sorts of absolute nonsense.
- schacon 2y agoDamn you! I did find it a little funny that my patch was so small but my commit message was so long. Also, I haven't successfully landed it yet, I keep being too lazy to roll more versions.
- pR0Ps 2y agoSame: https://github.com/ianare/exif-py/pull/75/files https://github.com/ianare/exif-py/pull/75/files I feel like a bit of a fraud because this was the PR that got me the "Mars 2020 Contributor" badge...
- andrewshadura 2y agoInterestingly, Mercurial had solved the bundles more than ten years ago and back then they already worked better than Git's today
- nine_k 2y agoBut branches were more problematic.
- capitainenemo 2y agoMercurial has had git-like "lightweight branches"/bookmarks without the revision record of mercurial named branches for over 15 years. There are good reasons to use the traditional branches though. https://mercurial.aragost.com/kick-start/en/bookmarks/ https://mercurial.aragost.com/kick-start/en/bookmarks/
- DrinkyBird 2y agoThe topics[0] feature in the evolution extension is probably even closer to Git branches, since they are completely mutable and needn't be a permanent part of your repo. Bookmarks are just pointers to changesets, and although that's technically how Git branches work, it's not how they work in practice in Mercurial because of its focus on immutability (and because hg and git work differently). [0]: https://www.mercurial-scm.org/doc/evolution/tutorials/topic-tutorial.html https://www.mercurial-scm.org/doc/evolution/tutorials/topic-...
- capitainenemo 2y agoNot the only mercurial feature where that's the case.. sad, I keep rooting for the project to implement mercurial frontend over a git db, but they seem to be limited by missing git features.
- kps 2y agoJujutsu (jj) is heavily inspired by Mercurial (though with some significant differences) and can operate with git as a storage backend. https://github.com/jj-vcs/jj https://github.com/jj-vcs/jj
- jakub_g 2y agoThis is super interesting, as I maintain a 1M commits / 10GB size repo at work, and I'm researching ways to have it cloned by the users faster. Basically for now I do a very similar thing manually, storing a "seed" repo in S3 and having a custom script to fetch from S3 instead of doing `git clone`. (It's faster than cloning from GitHub, as apart from not having to enumerate millions of objects, S3 doesn't throttle the download, while GH seem to throttle at 16MiB/s.) Semi-related: I always wondered but never got time to dig into what exactly are the contents of the exchange between server and client; I sometimes notice that when creating a new branch off main (still talking the 1M commits repo), with just one new tiny commit, the amount of data the client sends is way bigger than I expected (tens of MBs). I always assumed the client somehow established with the server that it has a certain sha, and only uploads missing commit, but it seems it's not exactly the case when creating a new branch.
- maccard 2y agoFunny you say this. At my last job I managed a 1.5TB perforce depot with hundreds of thousands of files and had the problem of “how can we speed up CI”. We were on AWS, so I synced the repo, created an ebs snapshot and used that to make a volume, with the intention of reusing it (as we could shove build intermediates in there too. It was faster to just sync the workspace over the internet than it was to create the volume from the snapshot, and a clean build was quicker from the just sync’ed workspace than the snapshotted one, presumably to do with however EBS volumes work internally. We just moved our build machines to the same VPC as the server and our download speeds were no longer an issue.
- ks2048 2y agoHow much bandwidth and time is wasted cloning the entire history of large projects when people only need single snapshot in a single branch? According to SO, newer versions of git can do, git init git remote add origin <url> git fetch --depth 1 origin <sha1> git checkout FETCH_HEAD
- jes5199 2y agoI have a vague recollection that GitHub is optimized for whole repo cloning and they were asking projects not to do shallow fetching automatically, for performance reasons
- sureIy 2y agoI don't know if that applies anymore or if it doesn't apply on GitHub Actions, but shallow clones is the default there. See `actions/checkout`
- jakub_g 2y agoGH Actions generally need a throwaway clone. The issue with shallow clones is that subsequent fetches can be expensive. But in CI most of the time you don't need to fetch after clone.
- nyanpasu64 2y agoAs I understand this issue affected Homebrew and CocoaPods: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomment-193772935 https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm... > Apparently, most of the initial clones are shallow, meaning that not the whole history is fetched, but just the top commit. But then subsequent fetches don't use the --depth=1 option. Ironically, this practice can be much more expensive than full fetches/clones, especially over the long term. It is usually preferable to pay the price of a full clone once, then incrementally fetch into the repository, because then Git is better able to negotiate the minimum set of changes that have to be transferred to bring the clone up to date.
- masklinn 2y ago
- geenat 2y agogit needs built in handling of large binary files without a ton of hassle, it's all I ask. It'd make git universally applicable to all projects. mercurial had it for ages. svn had it for ages. perforce had it for ages. just keep the latest binary, or last x versions. Let us purge the rest easily.
- GrantMoyer 2y agoGetting better: https://git-scm.com/docs/partial-clone https://git-scm.com/docs/partial-clone
- Cthulhu_ 2y agoIf git lfs fulfills this role, it could become pre-installed with new git installations.
- WorldMaker 2y agoIt is preinstalled on Windows and macOS today by the official installers. Ask your Linux distribution if out-of-the-box git lfs is right for you.
- pjc50 2y agoAs someone who used and administered p4 for ages, I regard git as a regression in this regard. Making git a fully distributed system is really expensive for certain use cases. My current employer still uses p4 for large integrated-circuit workflow assets. A previous workplace was trying to migrate from svn to git, when we realized that every previous official build had checked in the resulting binaries. A sane thing to do in svn, where the cost is only on the server, but would have resulted in a naive conversion costing 50Gb on every client.
- mbac32768 2y agoOne consequence of git clone is that if you have mega repos, it kind of ejects everything else from your cache for no win. You'd actually rather special case full clones and instruct the storage layer to avoid adding to the cache for the clone. But this isn't always possible to do. Git bundles seem like a good way to improve the performance of other requests, since they punt off to a CDN and protect the cache.
- robertlagrant 2y agoNothing to do with the article, but I appreciate the slightly idiosyncratic GitButler YouTube videos that explain how bits of Git work.
- schacon 2y agoIf you want us to cover something on Bits and Booze, just let me know! :)