4 ms·

Hi, I'm Chip, one of the authors of the blog post and XAR itself. Happy to answer any questions anyone may have about how XARs work, the way we use them, or th

by ctur 8y ago

Hi, I'm Chip, one of the authors of the blog post and XAR itself. Happy to answer any questions anyone may have about how XARs work, the way we use them, or the motivations that drove their development.

jkingsbery 8y ago
How do you pronounce "XAR"? "Ex-AR"? "Sar"? "Shar"?
- ctur 8y ago
  One syllable, rhymes with "car" but with a "z" sound ("zar").
  - blattimwind 8y ago
    Czar?
- progval 8y ago
  > XAR is pronounced like "czar" (/t͡ʂar/). The 'X' in XAR is meant to be a placeholder for all other letters as at Facebook this format was originally designed to replace ZIP-based PAR (Python archives), JSAR (JavaScript archives), LAR (Lua archives), and so on. https://pypi.org/project/xar/ https://pypi.org/project/xar/
shock 8y ago
Hi Chip, thanks for releasing this as open source. From a quick look XARs seem pretty similar to AppImages in the sense they both use an executable preamble and a squashfs so I'm wondering which would be a better choice for me to distribute my Python apps, and why. Thanks!
- ctur 8y ago
  I am not an expert in AppImage but I think the main thing is it is pretty easy to make a XAR of a Python script and that the overhead is probably less (especially for repeated invocations). I think XAR is simpler than the other alternatives; they have more complex specifications, being aimed at distributing full GUI applications (though I bet they work fine for these kinds of use cases, modulo perhaps quick and easy conversion of a Python program into a XAR). I suspect, but haven't measured, that XAR has lower execution overhead since it will re-use a mount point if a tool is recently invoked, rather than remounting every time. One thing XAR doesn't really try to do is work cross-platform. It will rely on the system Python, for instance, rather than embed the interpreter and all libraries inside (it will embed the libraries your tool depends on that aren't part of Python itself). This is a pro in some cases (lower overhead), but a con if you want something you can carry across wildly different systems.
  - buchanae 8y ago
    Could you write the core in Go, compile cross-platform, bundle the core binary, and then remove the reliance on the host's Python?
oahayder 8y ago
Hi Chip. How does facebook deploy XARs? Is an advantage over docker containers not having to run the docker daemon on every machine?
- ctur 8y ago
  We generally deploy XARs like a normal executable (delivered via RPMs or other native packaging). We also have an internal packaging format we use to deploy them as well. The big thing for us is we treat them like normal executables, so any standard deployment model for an executable works for XARs.
ksec 8y ago
Is it related to / inspired by Ruby Packer[1]? Because I am skimming it and seems 90% similar. [1] https://github.com/pmq20/ruby-packer https://github.com/pmq20/ruby-packer
- ctur 8y ago
  Not related, but at first glance it looks similar. However, it appears that ruby-packer uses a library to access the squashfs image rather than actually mounting it. This makes it difficult (probably impossible) to load .so files from the squashfs file without having to copy them externally or to carry other helper executables along inside the image without also having to copy them externally to invoke. A big goal for XAR is to be utterly transparent and not require hooks into Python, Node, or other runtimes to function.
mmastrac 8y ago
Awesome work. Did your team evaluate creating a virtual filesystem that could process the SquashFS images without involving the kernel? Having completely independent executables that could run on _any_ system with zero additional install would be sweet. To clarify - a stub in each XAR would act as a filesystem driver and intercept calls to open/read/etc, redirecting them to the internal data blob. Edit: I see your comment below which answers this! https://news.ycombinator.com/item?id=17524910 https://news.ycombinator.com/item?id=17524910
- rkeene2 8y ago
  For what it's worth, this is how Tcl self-contained executables work [0], since Tcl already has a virtual filesystem mechanism. [0] https://www.tcl.tk/starkits/ https://www.tcl.tk/starkits/
jwatte 8y ago
Chip, thanks for sharing! Could you have solved the same problem ("deploy apps with dependencies in a single file") using snaps? (snapcraft.io)
- ctur 8y ago
  I think XARs are much simpler. snapcraft (and AppImage and other tools) seem to be much more invasive to the development flow and more opinionated about what the contents of the image look like (directory structures, yaml/xml/json files for metadata, etc). XAR is much simpler. All it needs is a shell script (or raw executable) to run inside the XAR. Everything else is up to the bootstrapping. Note this isn't always a good thing -- opinionated software has its place. For what we use XAR for, though, having it "just work" with standard Python tooling without expecting open source libraries or modules or build tools behave differently is a strong plus.
  - RazZziel 8y ago
    AppImage only needs a shell script, which you can use to run whatever you throw at it (native, python, perl, ruby, java, wine, qemu...), the only difference seems to be it will use a compressed ISO instead of squashfs to pack the directory and later mount it with fuse. Additionally and optionally you can setup a small .desktop file to provide some convenience metadata what will be useful to pack the app easily, and an icon file.
CBLT 8y ago
I recently found out about Singularity[0], which seems to be very similar (squashfs application bundle). What advantages does XAR have? [0] https://github.com/singularityware/singularity https://github.com/singularityware/singularity
- neuromantik8086 8y ago
  A budget? Singularity was basically born of not-invented-here syndrome in the neuroscience community and doesn't have nearly the amount of resources that Facebook is able to devote to this problem. If that were to change (i.e., Greg Kurtzer gets millions of dollars in VC or NIH funding out of nowhere) and Singularity weren't a lone developer project it might be worth considering as a viable alternative to XARs.
raghava 8y ago
Hi Chip! Did you evaluate flatpak/appimage/snap before developing XARs? If so, what were the shortcomings that you noticed in them? I have found reasonably good success with snap on Python and node apps, but am not an expert on them. Just want to know from other practitioners about any gotchas that others might have studied/stumbled upon. Thanks!
- ctur 8y ago
  XAR development began in 2014 and hit production use in an early form in 2015; this was before some other options were out there, or before, I believe, AppImage went the squashfuse route. XAR doesn't aim to, say, represent a sandboxed filesystem or provide some isolation or provide a specification for GUI apps to seamlessly integrate with your desktop. Instead, the main idea is to just get a single file that, when run, mounts a filesystem and runs a command inside it. In my mind, at least, it's a simpler, smaller primitive you could build other systems on top of (like we have with deploying Python, Lua, and Node).

by ctur 8y ago

jkingsbery 8y ago

How do you pronounce "XAR"? "Ex-AR"? "Sar"? "Shar"?

ctur 8y ago
One syllable, rhymes with "car" but with a "z" sound ("zar").
- blattimwind 8y ago
  Czar?
progval 8y ago
> XAR is pronounced like "czar" (/t͡ʂar/). The 'X' in XAR is meant to be a placeholder for all other letters as at Facebook this format was originally designed to replace ZIP-based PAR (Python archives), JSAR (JavaScript archives), LAR (Lua archives), and so on. https://pypi.org/project/xar/ https://pypi.org/project/xar/

shock 8y ago

Hi Chip, thanks for releasing this as open source. From a quick look XARs seem pretty similar to AppImages in the sense they both use an executable preamble and a squashfs so I'm wondering which would be a better choice for me to distribute my Python apps, and why. Thanks!

ctur 8y ago
I am not an expert in AppImage but I think the main thing is it is pretty easy to make a XAR of a Python script and that the overhead is probably less (especially for repeated invocations). I think XAR is simpler than the other alternatives; they have more complex specifications, being aimed at distributing full GUI applications (though I bet they work fine for these kinds of use cases, modulo perhaps quick and easy conversion of a Python program into a XAR). I suspect, but haven't measured, that XAR has lower execution overhead since it will re-use a mount point if a tool is recently invoked, rather than remounting every time. One thing XAR doesn't really try to do is work cross-platform. It will rely on the system Python, for instance, rather than embed the interpreter and all libraries inside (it will embed the libraries your tool depends on that aren't part of Python itself). This is a pro in some cases (lower overhead), but a con if you want something you can carry across wildly different systems.
- buchanae 8y ago
  Could you write the core in Go, compile cross-platform, bundle the core binary, and then remove the reliance on the host's Python?

oahayder 8y ago

Hi Chip. How does facebook deploy XARs? Is an advantage over docker containers not having to run the docker daemon on every machine?

ctur 8y ago
We generally deploy XARs like a normal executable (delivered via RPMs or other native packaging). We also have an internal packaging format we use to deploy them as well. The big thing for us is we treat them like normal executables, so any standard deployment model for an executable works for XARs.

ksec 8y ago

Is it related to / inspired by Ruby Packer[1]? Because I am skimming it and seems 90% similar. [1] https://github.com/pmq20/ruby-packer https://github.com/pmq20/ruby-packer

ctur 8y ago
Not related, but at first glance it looks similar. However, it appears that ruby-packer uses a library to access the squashfs image rather than actually mounting it. This makes it difficult (probably impossible) to load .so files from the squashfs file without having to copy them externally or to carry other helper executables along inside the image without also having to copy them externally to invoke. A big goal for XAR is to be utterly transparent and not require hooks into Python, Node, or other runtimes to function.

mmastrac 8y ago

Awesome work. Did your team evaluate creating a virtual filesystem that could process the SquashFS images without involving the kernel? Having completely independent executables that could run on _any_ system with zero additional install would be sweet. To clarify - a stub in each XAR would act as a filesystem driver and intercept calls to open/read/etc, redirecting them to the internal data blob. Edit: I see your comment below which answers this! https://news.ycombinator.com/item?id=17524910 https://news.ycombinator.com/item?id=17524910

rkeene2 8y ago
For what it's worth, this is how Tcl self-contained executables work [0], since Tcl already has a virtual filesystem mechanism. [0] https://www.tcl.tk/starkits/ https://www.tcl.tk/starkits/

jwatte 8y ago

Chip, thanks for sharing! Could you have solved the same problem ("deploy apps with dependencies in a single file") using snaps? (snapcraft.io)

ctur 8y ago
I think XARs are much simpler. snapcraft (and AppImage and other tools) seem to be much more invasive to the development flow and more opinionated about what the contents of the image look like (directory structures, yaml/xml/json files for metadata, etc). XAR is much simpler. All it needs is a shell script (or raw executable) to run inside the XAR. Everything else is up to the bootstrapping. Note this isn't always a good thing -- opinionated software has its place. For what we use XAR for, though, having it "just work" with standard Python tooling without expecting open source libraries or modules or build tools behave differently is a strong plus.
- RazZziel 8y ago
  AppImage only needs a shell script, which you can use to run whatever you throw at it (native, python, perl, ruby, java, wine, qemu...), the only difference seems to be it will use a compressed ISO instead of squashfs to pack the directory and later mount it with fuse. Additionally and optionally you can setup a small .desktop file to provide some convenience metadata what will be useful to pack the app easily, and an icon file.

CBLT 8y ago

I recently found out about Singularity[0], which seems to be very similar (squashfs application bundle). What advantages does XAR have? [0] https://github.com/singularityware/singularity https://github.com/singularityware/singularity

neuromantik8086 8y ago
A budget? Singularity was basically born of not-invented-here syndrome in the neuroscience community and doesn't have nearly the amount of resources that Facebook is able to devote to this problem. If that were to change (i.e., Greg Kurtzer gets millions of dollars in VC or NIH funding out of nowhere) and Singularity weren't a lone developer project it might be worth considering as a viable alternative to XARs.

raghava 8y ago

Hi Chip! Did you evaluate flatpak/appimage/snap before developing XARs? If so, what were the shortcomings that you noticed in them? I have found reasonably good success with snap on Python and node apps, but am not an expert on them. Just want to know from other practitioners about any gotchas that others might have studied/stumbled upon. Thanks!

ctur 8y ago
XAR development began in 2014 and hit production use in an early form in 2015; this was before some other options were out there, or before, I believe, AppImage went the squashfuse route. XAR doesn't aim to, say, represent a sandboxed filesystem or provide some isolation or provide a specification for GUI apps to seamlessly integrate with your desktop. Instead, the main idea is to just get a single file that, when run, mounts a filesystem and runs a command inside it. In my mind, at least, it's a simpler, smaller primitive you could build other systems on top of (like we have with deploying Python, Lua, and Node).