9 ms·
Are tarballs obsolete?
- discreditable 11y agoCorrect me if I'm wrong but doesn't a git clone involve downloading the project's entire git history as well as $CURRENT_VERSION? That seems a little excessive.
- nilved 11y agoYou can use `--depth=1` to only download one revision, and `--branch` to specify an alternate branch or tag.
- XorNot 11y agoAs far as I know though you can't grab a specific revision this way (I'd love to be shown otherwise) - which is an issue when setting up dockerfiles since you want to bake the sha hash which works into the file.
- akshaykarthik 11y agogit clone --depth=1 will get you just $CURRENT_VERSION
- hueving 11y agoBy default, yes. But you can use the --depth option to limit it to a clone of the most recent state.
- deleted 11y ago[deleted]
- tjbiddle 11y agoWhile I disagree with the authors sentiment as a whole; this issue could be solved by using a shallow clone.
- abritinthebay 11y agoIronically git actually supports TAR for this exact thing: git archive exports tar files. You can do a low depth to minimize that if you want (--depth=1 or something like that if I remember correctly). You could just rm -rf the root .git directory after but that is a download bloat issue. That said this should work: git clone --depth=1 --branch=master git://myrepo && rm -rf myrepo/.git` but.. yeah. Not really great.
- dajohnson89 11y agoAnd what if you find a bug, and want to regress to a previous version? What is so cumbersome about downloading a project's history?
- deleted 11y ago[deleted]
- ubernostrum 11y agoWell, some projects have been around for a while. "All the history" can be a LOT of history.
- eru 11y agoActually, even for the Linux kernel, it's only a few hundred megabyte.
- ufo 11y agoThe article suggests that this is not a big deal anymore now that computers have more storage and networks have more bandwidth.
- lukeschlather 11y agoThere are still a lot of networks that don't have enough bandwidth to make git trivial. Most home networks, it's going to turn a 5-second download into 60 seconds or more. Some networks, it's going to turn a 1-minute download into a half an hour.
- MBlume 11y agoSure, but on the other hand, downloading a tarball for every minor release means you're redownloading a lot of code. Depending on how much churn there is, you may save bandwidth in the long run by keeping a clone and updating as you go.
- joslin01 11y agoThe problem tarballs solve is bundling files together. That use-case will probably be with us for awhile even if it has fallen out of favor for source code.
- raimue 11y agoThe article is discussing whether it is worth to provide tarballs for software releases, not tarballs in general. The title might be misleading.
- tjbiddle 11y agoAbsolutely not; there are a myriad of reasons to prefer tarballs over Git: - Offline use - Permission restrictions (Internal git networks) - True-persistent versioning (Can always delete and update a tag), no external tools needed to download - Can restrict extra garbage that may not be need to be shipped - Etc.
- scrollaway 11y ago> Offline use Tarballs are just as offline as git repositories... > Permission restrictions (Internal git networks) What? > True-persistent versioning (Can always delete and update a tag) Why is that of concern regarding "tarballs vs git"? If anything it's a win for git. > Can restrict extra garbage that may not be need to be shipped Only valid point you presented, and it's better discussed elsewhere in this thread.
- gizmo686 11y ago>Tarballs are just as offline as git repositories... Tarballs are easier to shoenet into an offline machine. To do this with git repositories, you would either need to deal with transfering a directory structure or, more likely, pack the respository into some sort of tarball.
- voltagex_ 11y ago`git bundle create` will do what you want.
- vacri 11y ago> Why is that of concern regarding "tarballs vs git"? If anything it's a win for git. Git tags (and branches) are mutable. Tarballs technically are, but don't move around as much. Unless someone's being actively malicious, you can be confident that no-one's going to put new code in an old-versioned tarball.
- scrollaway 11y agoThe exact same can be said of git. You're reaching. Git has builtin verification of absolutely everything it holds (tags included) since it uses sha1 for objects and commit messages, and they include the previous state... so a point in a git tree in that way is a hell of a lot more immutable than a tarball.
- tuna 11y agodocker brought back the frenzy of delivering software in a tarball just like slackware in the past. Long live patrick volkerdig and tar -zxvf ... -C / !
- GPGPU 11y agoPro Tip: You don't need the dash. That's how old-school tar is!
- TheDong 11y agoEven though Docker's disk and transport format are technically tar, I think that this is still a wrong statement. tarballs are ubiquitous already. Docker actively hides that it uses tarballs (it abstracts away all interactions with them, to the point that it really is an implementation detail excluding docker export/import).
- vezzy-fnord 11y agoTarballs versus DVCS appears to be a false dilemma. They're independent mechanisms. The more proper question to ask here is "Are tarballs obsolete for distributing source code?".
- pjscott 11y agoThat's what the article is asking. Perhaps a misleading title.
- nuxi7 11y agoThe distributed form of a project (especially one based on GNU autotools) is often not merely a tarball of the upstream repository. Often the author has pre-run some stages of the build that she feels would impose unnecessary or esoteric dependencies. Typically this involves pre-generating Makefile.in from Makefile.am and configure from configure.ac. Another common one is for the maintainer to pre-generate the documentation. The user of the distributed tarball is fully capable of re-running these steps, but they have become optional. This is a fact often forgotten when projects switch to github and just start using the automated release tarballs github will make from the repo.
- davexunit 11y agoYes! Thank you for explaining this. I don't think a lot of people realize what makes up a good release tarball.
- hartror 11y agoExactly. Many web tool chains involve pulling from external repositories such as npm, bower or even github itself. If one of these are down you cannot deploy your application. Even better many packaging systems such as npm or setuptools allow you and library maintainers to specify a flexible version numbers for your dependencies. If during the course of your build, test and deploy chain one of these dependencies changes your application could break through no fault of your own. You cannot rely on maintainers to not release breaking changes in minor versions, it happens all the time, intentionally or not.
- Touche 11y agoAt least for npm there is a solution for that problem: https://docs.npmjs.com/cli/shrinkwrap https://docs.npmjs.com/cli/shrinkwrap
- Jasper_ 11y agoThis becomes more complex when distributions want to patch configure.ac or Makefile.am -- they need to patch the included Makefile.in files as well. Trying to balance these two forces is the reasoning behind the AM_MAINTAINER_MODE flag. These days, I prefer to simply use git archive for a tarball and make users run autogen.sh themselves. See https://blogs.gnome.org/desrt/2011/09/08/am_maintainer_mode-is-not-cool/ https://blogs.gnome.org/desrt/2011/09/08/am_maintainer_mode-... and the associated comments.
- deathanatos 11y agoThe article seems to mean "tarballs are obsolete for source distribution", not obsolete on the whole. > Here’s an advantage of the clone/pull distribution system; every clone is implicitly validated by its SHA1 hash chain. Given that TLS does not consider SHA1 secure[1] anymore… I'm not sure that's an assumption I'd be making. [1] in the sense that it's being rapidly deprecated; even if it doesn't trigger validation failures today, the fact that it will tomorrow is telling.
- eru 11y agoSHA1 is still good as a checksum against random errors.
- JoachimSchipper 11y agoSHA-1 is clearly imperfect, but not that big a problem here. The writing is on the wall for SHA-1's collision resistance. A break in collision resistance would allow a malicious software maintainer to distribute different tarballs to people of interest than to the internet at large, but would not allow a third party to replace a honestly-generated tarball with some other tarball ("second pre-image"). Since you almost certainly need to trust the maintainer not to insert backdoors anyway (a well-hidden backdoor will not be detected by the normal packaging process), trusting the maintainer not to maliciously distribute different software to a small number of people of interest isn't that unreasonable.
- tw04 11y agoNo. Source: Veritas NetBackup.
- dalke 11y agoMy comment from yesterdays post on this topic: If you're willing to hand-wave away certain things as being outside of the "center of the open-source software-release ritual", then you can get a 'yes'. But in addition to source-distributions mentioned in the essay, which are "too tiny a minority", two other issues are: 1) If you define "git" as "pervasive" then you can declare that git is the solution. In my field, some of the tools are still only accessible via tarball/zip. Two examples are the IUPAC InChI distribution, from http://www.iupac.org/home/publications/e-resources/inchi/download.html http://www.iupac.org/home/publications/e-resources/inchi/dow... , and VMD from http://www.ks.uiuc.edu/Development/Download/download.cgi?PackageName=VMD http://www.ks.uiuc.edu/Development/Download/download.cgi?Pac... . PyMol, a more popular free software visualization package, is also not developed using git. 2) If you define "open-source software-release" to exclude fee-based free software distribution (perhaps it is also "too tiny a minority"?), then it preferentially excludes certain flavors of 'open-source software'. For example, I write free software, and distribute it for money. A tarball is an easy way to identify the delivery of the contractually obligated product. I can also send it via email, vs. setting up a private git server and getting accounts set up for my customers. How much free software development is part of the hidden world of non-publicly accessible development? I don't think anyone really knows. So if your free software baseline assumes "pervasive git", a public development repo, and no cost to access the code, then sure, the answer is a "yes". Until then, it's a "no." And my examples show that baseline is only a subset of free software development, with no clear idea of how large it is.
- spb 11y agoAnother place where tarball/zip is significantly more viable than Git: the browser (and Chromebooks as such). I know there are things like Tim Caswell's JSGit, but they're woefully unfinished and assembly-required compared to an interface like JSZip.
- tritium 11y agoNo! Things like package managers and git, however pervasive, are agents and not the testable, serialized representation of data. Why wouldn't you make the same file set available as a single download via plain, old HTTP(S)? What's so hard about that? Why place something on a shelf, behind an agent that demands a learning curve, when it could also be available to a browser? You CAN'T hash a tree of files quickly by hand. You need a program to help test the integrity of a tree with some recursive problem solving. Take tarballs away from people, and you're putting things on a shelf, out of reach, taking away a level of safety and reliability from a certain cross-section of your audience. Not only that, but EVERY platform has a web browser, even if it's curl or wget. With a single file, available by HTTP, any platform can get to it, so if your machine dies, and you have to call a friend for help, and they don't use the same OS, they won't have to install a special tool, or hope an intervening third-party web service provides coverage. If we're simply talking file formats, and possibly exchanging the tarball for zips, ISOs or what-have-you, as alternative serial representations of the same data, that's not even a discussion. If you're worried about practical utility, load, bandwidth and/or availability (or even, perhaps, culling the dull, dumb, toxic and obnoxious from the herd, terrible as that sounds) then changing the agent permitted to access the resource might be one solution, but not necessarily the right solution.
- simula67 11y ago> Why place something on a shelf, behind an agent that demands a learning curve, when it could also be available to a browser? wget <filename> git clone <repo url> What is the difference in learning curve ? > With a single file, available by HTTP, any platform can get to it, so if your machine dies, and you have to call a friend for help, and they don't use the same OS, they won't have to install a special tool, or hope an intervening third-party web service provides coverage. People have been making an argument for static compiling basic tools for a long time now ( I support it ). I think git can be considered a pretty basic tool now given its popularity.
- swegg 11y agoThe authenticity check with GPG or SHA sums requires a gentler learning curve for a 'flat' tarball.
- aayala 11y agono
- jokoon 11y agoSeen how any filesystem will slow down when dealing with many individual files, i'd say no.
- dvh 11y agoTarballs can have empty directory
- liw 11y agoI am strongly in favour of tarballs for releases. Git tags can be moved, for example, so that's an extra thing to be careful about, if one is worried about reproducibility. http://yakking.branchable.com/posts/releases/ http://yakking.branchable.com/posts/releases/ That's my write-up on the topic of making releases, from about a month ago.
- brianzelip 11y agoThanks for posting your blog post link. Having only published static websites, I've been curious about the release process and your notes provide some insight, especially the parts about signatures.
- fsiefken 11y agotarballs which are also usually compressed (gz, bz2, 7z, xz) may have the benefit of being more efficient to download over bandwidth constraint or size constrained links. Also being 1 file, it's only one request and it works over zmodem or modern derivatives.
- RexRollman 11y agoI like tarballs. Speaking as a non-programmer who compiles things like Vim and Scrypt, it is nice to be able to download a release without having to install a VCS first.
- err4nt 11y agoTarballs are also easier to attach to an email or to share with other computers :)
- olgeni 11y agoStep 1: gems, npm, pip, etc. Step 2: "containers". Step 3: this.
- ZeWaren 11y agoConsider using core OS tools (wget/fetch, tar, etc.) versus the dependency tree required to run git on your system. Do you need curl? Do you need perl? Do you need a XML parser and a regex runtime engine?
- Johnny_Brahms 11y agoI love tarballs. They got a whole lot more enjoyable with all the paralell compression utilities out there. My favourite is for a file format that has fallen out of grace (bzip2) [1]. It is fast enough to make IO the bottleneck on my HDD (although not on my ssd). 1:http://compression.ca/pbzip2/ http://compression.ca/pbzip2/