23 ms·
XARs: An efficient system for self-contained executables
- rwmj 8y agoI'm not really a fan of containers, but I read this and thought "why not containers"? The page mentions cryptically "They could almost be thought of as a self-executing container without the virtualization". The "self-executing" bit makes sense - you don't have to remember to type "docker". "without the virtualization" doesn't make sense unless they mean without cgroups or are talking about Kata Containers.
- terrelln 8y agoXARs are just self mounting compressed readonly filesystems with an executable inside. We get hermitic dependencies by setting the PYTHONPATH, LD_LIBRARY_PATH and such in the bootstrapping script. One big benefit is that the filesystem only decompresses the pages as needed, which greatly improves the start up time over existing solutions.
- ctur 8y agoGenerally XARs are lighter weight than a container. While you can (and sometimes we do) use XARs to deploy, say, a self-contained service like a website, often they are used to replace command line tooling. Container isolation (cgroups, namespaces, etc) would make it difficult to do some of the system level tasks we use such tools for, such as configuration changes or monitoring. Likewise, we often are replacing a PAR or C++ tool with a new XAR version, and it is nice to simply replace the executable and not have to change how it is invoked. In this regard, invoking a XAR is identical to running any normal executable or shell script.
- e12e 8y agoI can see why you'd sometimes want less isolation (although things like docker-compose runs fine from a docker container). But how is it "lighter" than a container? Aren't you striving for self-contained executables? What do you leave out of a XAR that you'd want to put into a container? [ed: I now saw this question and answer: https://news.ycombinator.com/item?id=17526178 https://news.ycombinator.com/item?id=17526178 Frankly using system/external python (or other VM) seems a bit risky... But whatever works, I guess..]
- lozenge 8y ago"virtualization refers to the act of creating a virtual (rather than actual) version of something, including virtual computer hardware platforms, storage devices, and computer network resources." Containers have virtual file systems they access instead of the host one. XARs don't.
- fenesiistvan 8y agoI am a windows developer and the single thing that stops me porting my apps to linux is an easy to use deploy method. Is there some good way to handle this task without to spend months learning about linux administration like shell scripts, finding the best place for configs, logs on different linux distros, daemons setup, etc. Something simple and distro independent would be fine...
- rwmj 8y agoI am a Linux developer and the single thing that stops me porting my apps to Windows is an easy to use deploy method. Is there some good way to handle this task without to spend months learning about Windows administration like installers, MSI, the registry, logging, services? The non-flippant answer is to just provide the source and let the distros package it for you. It's a different model. Linux users want to get their software through an integrated package manager, and volunteers will take your software and do all the work needed to make that happen.
- RcouF1uZ4gsC 8y agoIt is very easy on Windows to create a statically linked executable that you can just copy to wherever you want.
- rwmj 8y agoAnd it's very easy on Linux to provide a source tarball and let the distro packagers do all the work for you. Plus, you know, static linking exists on Linux too, it's not something that only Windows can do.
- isostatic 8y agoEvery night we have a job (run by jenkins I think) download the latest ffmpeg code, add some non-approved patched, and cross compile to static binaries for x64 linux, osx, and windows (we've dropped the 32 bit linux build) It's not hard to make a static binary. On the other hand most of the "software" I write - mainly shell scripts - includes things like apache configuration, requires other applications (like apache, iperf, tcpdump, lldp, etc) I want to deploy these to multiple boxes, I want them versioned, I want them easy to update, I want to know what version is installed. All of this is handled by a 10 line config file in a .deb.
- aumerle 8y agoIt's somewhat counter-intuitive that start times with XAR are lower than start times without it. Is fuse faster than a kernel filesystem? Even with compression?
- terrelln 8y agoAdmittedly I haven’t profiled this yet, but my guess is it is a constant overhead of setting up pkg_resources that the native code uses to load the entry point. The test against native start speed was hot, so the pages required were already in the page cache, so the filesystem shouldn’t matter.
- ctur 8y agoFUSE isn't generally lighter weight than a filesystem but it can be relatively competitive for simple use cases like a read-only filesystem. Additionally, squashfs lets you pack metadata and data very tightly, and since it is a readonly filesystem, has some optimizations normal filesystems can't (how data is placed, overhead of managing metadata operations, etc). Also squashfs lets you choose how the files are laid out and compressed so that all files of a certain type, such as all .pyc files, are close together, which increases compression ratio and reduces overhead for subsequent file accesses (i.e., can reduce random disk or flash IO). In practice the timings of XAR vs filesystem are close enough to be "in the noise" -- it's when compared to PEX or PARs that the difference is quite large.
- aumerle 8y agoMakes sense, this is somewhat analogous to the import speedup you can get by putting all the python modules into a zip file. I tend to do that when distributing python applications on windows, where the speedup is more noticeable.
- ctur 8y agoYep, it's similar, but squashfs is more optimized than zip files for random access like a filesystem (rather than an archive). Also when using zstd-based squashfs files, there is much less overhead for the decompression itself which effectively becomes free.
- ctur 8y agoHi, I'm Chip, one of the authors of the blog post and XAR itself. Happy to answer any questions anyone may have about how XARs work, the way we use them, or the motivations that drove their development.
- jkingsbery 8y agoHow do you pronounce "XAR"? "Ex-AR"? "Sar"? "Shar"?
- ctur 8y agoOne syllable, rhymes with "car" but with a "z" sound ("zar").
- blattimwind 8y agoCzar?
- progval 8y ago> XAR is pronounced like "czar" (/t͡ʂar/). The 'X' in XAR is meant to be a placeholder for all other letters as at Facebook this format was originally designed to replace ZIP-based PAR (Python archives), JSAR (JavaScript archives), LAR (Lua archives), and so on. https://pypi.org/project/xar/ https://pypi.org/project/xar/
- shock 8y agoHi Chip, thanks for releasing this as open source. From a quick look XARs seem pretty similar to AppImages in the sense they both use an executable preamble and a squashfs so I'm wondering which would be a better choice for me to distribute my Python apps, and why. Thanks!
- ctur 8y agoI am not an expert in AppImage but I think the main thing is it is pretty easy to make a XAR of a Python script and that the overhead is probably less (especially for repeated invocations). I think XAR is simpler than the other alternatives; they have more complex specifications, being aimed at distributing full GUI applications (though I bet they work fine for these kinds of use cases, modulo perhaps quick and easy conversion of a Python program into a XAR). I suspect, but haven't measured, that XAR has lower execution overhead since it will re-use a mount point if a tool is recently invoked, rather than remounting every time. One thing XAR doesn't really try to do is work cross-platform. It will rely on the system Python, for instance, rather than embed the interpreter and all libraries inside (it will embed the libraries your tool depends on that aren't part of Python itself). This is a pro in some cases (lower overhead), but a con if you want something you can carry across wildly different systems.
- juliangoldsmith 8y agoThis all reminds me a bit of Tiny Core Linux. IIRC, it uses SquashFS images for all its packages, mounts them in a specific spot, then uses either symlinks or UnionFS to put everything together.
- mediocrejoker 8y agoI wonder how this compares to AppImages https://appimage.org/ https://appimage.org/
- thangngoc89 8y ago- AppImages: Linux apps that run anywhere - XARs: packages Python (node.js, lua scripts app) into executable files
- mediocrejoker 8y agoThanks for providing that summary. Does this mean the two could be used together?
- thangngoc89 8y agoFrom my understandings: - AppImage: it packs Linux apps into a tar file so you can unzip it later and run the executable of the app. The main selling point of AppImage is it's distro-independent - XAR: it packs dynamic languages programs (python, node.js, lua) into a executable file. The executable includes the language runtime, a fuse filesystem to mount the program's source code. In conclusion, I don't think you ever needs to make an AppImage for XAR executable file.
- RazZziel 8y agoNot exactly: An AppImage file packs a Linux app as a compressed ISO image with an ELF preamble that is able to access the contents of the ISO image without unpacking, so you can just double click the package and run the app without ever unpacking it.
- probonopd 8y agoIs there anything XAR can do that AppImage cannot? There are very complex Python applications (e.g., Ultimaker Cura) which are packaged in the AppImage format as self-standing single-file executables, including the Python interpreter, libraries, and other resources.
- secure 8y agoCool idea! Is there any particular reason to use SquashFS via FUSE instead of via the Linux kernel driver? Slightly related: we also recently switched to SquashFS for the gokrazy.org’s root file systems. If you’re curious about how SquashFS works under the hood, check out https://github.com/gokrazy/internal/blob/master/squashfs/writer.go https://github.com/gokrazy/internal/blob/master/squashfs/wri.... I also intend to publish a poster about it at some point.
- ctur 8y agoWe actually started with using "real" squashfs files. This had three main disadvantages: - We had to maintain our own setuid executable to perform the loopback setup and mount (rather than relying on the far more tested and secure open source fusermount setuid binary that all FUSE file systems rely on) - Getting loopback devices to behave inside of containers (generally cgroup and mount namespace containers) was a little tricky at times in some of our environments - We didn't want to have a huge number of extra loopback devices on every host in our fleet In fact, after implementing the loopback-based filesystem version, we almost abandoned XAR as the downside of the security considerations and in-container behavior wasn't ideal. The open source squashfuse FUSE filesystem really is what made it possible. Another side benefit is we could iterate far faster with squashfuse -- this let us fix some performance issues, add idle unmounting, and implement zstd-based squashfs files, and then deploy that to our fleet, faster than we could deploy a kernel to 100% of hosts.
- secure 8y agoThanks, makes sense!
- fooblitzky 8y agoSounds similar to TCL's StarKits
- rgovostes 8y agoProbably too late now, but xar already stands for "eXtensible ARchiver" and is a file format used on macOS in some package installers. It's notable for having an embedded XML "table of contents" that describes metadata of the archived files, so new fields can easily be added while maintaining backwards compatibility. (Compared to, say, the zip file format which does not even specify how to store Unix file modes.) https://en.wikipedia.org/wiki/Xar_(archiver) https://en.wikipedia.org/wiki/Xar_(archiver)
- JohnDotAwesome 8y agoNames are hard. My name is John. Programmers get really confused when I tell them that. "Did you know there's another programmer named John? You probably should have researched names before you decided to go with that one"
- mindslight 8y agoThe larger "person's name" namespace includes the first name and last name. And people do generally treat it as a surprise when they find someone with the same name.
- dsr_ 8y agoNamespaces can help with that. For example, I note that you aren't just John, you're John Fawcett. And you probably have a middle name (or two, or more). I bet you've worked somewhere with a collision on John. Did you adopt a handle, like John F, or maybe JohnDotAwesome?
- JohnDotAwesome 8y agoI really wish I did have more than one middle name, both of which starting with `R` (indeed, my middle name does start with R). Maybe then as a J.R.R. Fawcett I could write fantasy novels.
- 52-6F-62 8y agoOh I'm one of the lucky ones then. It's funny, I often wish I had fewer names. I swear— very few forms ever allow room for two middle names or two middle initials. R.R.A. Fairley But I'm a little more into science fiction
- jarvuschris 8y agoHave you looked into Habitat? It provides a similar result with a complete build workflow that works across technologies and platforms: https://www.habitat.sh/ https://www.habitat.sh/ There's a rapidly growing library of libraries and services packaged with it: https://bldr.habitat.sh https://bldr.habitat.sh Its build artifacts can be exported to a number of formats including container images and tarballs, maybe a XAR exporter could be built: https://github.com/habitat-sh/habitat/tree/master/components/pkg-export-docker https://github.com/habitat-sh/habitat/tree/master/components...
- JohnDotAwesome 8y agoSeems like Habitat (which looks awesome by the way) relies on Docker. Which, if you consider performance heuristics in the article (size, cold/hot start time), may be a non-starter for what they're trying to do.
- djb_hackernews 8y agoCan you expand on that? In my experience nothing about Docker implies a performance impact in terms of size or start time.
- collinf 8y agoWell, there is the overhead of creating and removing namespaces each time a container is ran, or communicating with the Docker daemon. I think to most people it would be negligible, but fb operates at a scale where these normally insignificant pieces matter. I would be interested to hear more about the _why_ of a system like this over containerization. edit: rwmj's comment has a good discussion over the benefits of this over containerization.
- nwmcsween 8y agoI promise you 100% the overhead is docker and nothing else.
- 8y ago
- saagarjha 8y agoKind of an unfortunate name, considering that xar (eXtensible ARchive) is already a thing: https://en.wikipedia.org/wiki/Xar_(archiver) https://en.wikipedia.org/wiki/Xar_(archiver)
- nine_k 8y agoI'd hazard to say that almost any 3-letter abbreviation has been already taken, many of the easy-to-pronounce ones, multiple times.
- 52-6F-62 8y agoOh there are some fabulously childish three and four letter abbreviations available yet. Probably not very appropriate, though..... https://fileinfo.com/browse/ https://fileinfo.com/browse/
- peterwwillis 8y agoI would use this if it didn't depend on OS-specific features. Squashfs is not portable to Windows, unless you extract it to disk. I actually prefer the jar/Tomcat model, where the read-only image gets distributed to servers, and when you run the app the image gets unpacked to disk as needed. You could also write I/O wrappers that would obviate the need to extract them to disk, and you could even make compression optional to reduce performance hits. It seems like all you really need is a virtual filesystem implemented as a userspace i/o wrapper. Basically FUSE but only for the one app. There's no need for the FUSE kernel shim because only the application is writing to its own virtual filesystem. So this would work on any operating system that supported applications that can overload system calls. For example, I would start with this project http://avf.sourceforge.net/ http://avf.sourceforge.net/ and modify it to run apps bundled with itself. With FUSE installed, other apps could interact with its virtual filesystem, but without FUSE, it could still access its own virtual filesystem in an archive. I would then extend it by shimming in a copy-on-write filesystem to stack modifications in a secondary archive.
- twsted 8y ago> I would use this if it didn't depend on OS-specific features. I always make the same mistake to assume that none would deploy something on windows for a server–side environment.
- ctur 8y agoI agree it is a bummer that FUSE doesn't directly work on Windows, but it should be doable -- we would love for someone to figure out the best way to do this on Windows. Happy to collaborate with anyone who'd like to make this a reality.
- PeCaN 8y agoWhile FUSE doesn't work directly, there are userspace filesystem implementations for Windows. Dokany¹ even implements most of the FUSE API. 1. https://github.com/dokan-dev/dokany https://github.com/dokan-dev/dokany
- TheAceOfHearts 8y agoI'm on mobile, but do you have an example of bundling a node app somewhere? I'm curious how it compares to using something like pkg [0]. [0] https://github.com/zeit/pkg https://github.com/zeit/pkg
- terrelln 8y agoWe currently don't have a nice open source API for building node apps, but would welcome PRs that get us in this direction! There are two ways to build a node app using the XAR builder tools. 1. Use the `make_xar` tool which will create a XAR from a directory and takes an optional script to run on execution. 2. Use the XAR builder library to make a XAR builder that is specialized for building node apps.
- dagenix 8y agoDoes the XAR file containing the Python executable itself, or, does running it rely on having Python installed on the host already?
- terrelln 8y agoThe current Python XARs rely on Python being on the system path. But it would be easy to build a custom Python XAR with the XAR builder library that includes the Python executable and makes sure to use the packaged executable.
- ASinclair 8y agoHow similar is this to Google PAR/SAR executables for Python and Bash scripts respectively?
- terrelln 8y agoFacebook's PAR is a self-extracting zip file, I assume Google's is similar. XARs are self-mounting SquashFS archives (a compressed read only filesystem). This means that XARs don't have to be extracted to a temporary directory to run, they can run in place. Zip files have to be completely extracted before running, but SquashFS decompresses pages on the fly, so startup times are much faster (especially with zstd compression).
- wesleyy 8y agoI don't think Google's implementation of their hermetic par files are open source.
- jeffrallen 8y ago<troll type="language">Just use Go.</troll>
- nikolay 8y agoIs Facebook an NIHS (Not Invented Here Syndrome) sufferer?
- lttlrck 8y agoIndirectly this proves to be a useful discovery mechanism for me - when tools crop up on HN I think ‘hey that’s interesting’, then oftentimes when I read the comments I find there are numerous existing solutions I had never heard of along with helpful links and insightful info :-) It’s brilliant. It’s one of the reasons I value this site so much.
- dcgudeman 8y agoRequirements Python >= 2.7.11 & >= 3.5 you need both?
- jillesvangurp 8y agoNot a bad idea. I wonder how this compares to ubuntu's snaps. Seems like a good idea to me but I've not really seen it used much yet. On OS-X apps have been distributed in a .app form for ages. It's very uncommon for OS X apps to have installers or a more complicated installation (and uninstallation) than drag and drop. So, good idea and it kind of fixes a big issue where most linux distributions seem to insist on dll hell with just about anything littering the file system with cruft and just about every interpreter out there reinventing ways to create virtual environments.
- FRidh 8y agoYet another format for self-contained executables, and one that looks pretty similar to the already existing AppImage. Note that Nix users can use `nix-bundle` to create AppImages of all the software in Nixpkgs, which is according to Repology one of the largest and freshest package sets: https://repology.org/statistics https://repology.org/statistics
- russellbeattie 8y agoFacebook spent a decade contributing virtually nothing to open source, now they're flooding the world with random projects of varying and questionable value - most developed as a result of Facebook's severe N.I.H. attitude. I'm honestly not sure which is worse.
- tony-allan 8y agoXAR is a simple way to package and deploy Python and similar apps. Dependencies are a real issue. Anything to improve the situation and to also deploy related files is a great idea. I don't like a lot about what Facebook does with user data and marketing but their support of open source is better than many companies. Give credit where it is due.