11 ms·
Warp – self-contained, single binary applications
- TheDong 8y agoThis sounds like a much less general binctr [0]. I don't see how warp handles the most trivial case of a dynamically linked executable; I see no references to patchelf or other such tricks to ensure it looks within the application's cache directory for dynamically linked libraries, nor do I see any mount namespaces. Does warp do that? Why would I use warp if it doesn't when there are more generic solutions that can handle such a case, namely binctr, docker, guix pack, nix.... [0]: https://github.com/genuinetools/binctr https://github.com/genuinetools/binctr
- sagichmal 8y ago> This sounds like a much less general binctr. binctr is Linux-only, and appears to need Docker as a runtime requirement? This has neither constraint, apparently?
- TheDong 8y ago> appears to need Docker as a runtime requirement binctr does not require any runtime component other than a modern linux kernel with unprivileged usernamespace support. Targetting OSX / Windows does make warp distinct from what I mentioned, though I don't use those OSs so I admit I didn't think about that possible issue. Anyway, I thought this was a solved problem on both Windows and OSX since they both have conventions for self-contained applications, while linux does not.
- sillysaurus3 8y agoAnyway, I thought this was a solved problem on both Windows and OSX since they both have conventions for self-contained applications, while linux does not. This is mistaken for Windows. Even if programs are run from Program Files, they will often source their DLLs from other locations.
- ealexhudson 8y agoI wrote stowage.org to do something similar, having a Docker runtime (which stowage _does_ have) is actually pretty neat in a couple of ways because you also get a kind of basic package management for free. It's pretty nice for sharing dev tools amongst a team. Maybe these things don't have broad use cases, but I think there are a lot of solvable pains here.
- hbt 8y agocheck out whalebrew it is similar to your project
- ianl 8y agoBeautiful
- w8rbt 8y agoVery neat. Would like to see more languages targeting single binaries. This is one reason (I believe) that has made Go so popular.
- jpic 8y agoI wish <my favorite language> also had this. However, containers work. Not sure why they do java though, i thought java has been happy with jars so far.
- regecks 8y agoIf anybody is thinking about Java, check Graal out. I was able to successfully create fast, completely static ELF binaries of real JVM applications from fat JARs (not in the fake self-executable archive way). It's still early days in terms of what's available in the runtime (e.g. the AWT subsystem is missing), but it's pretty damn impressive anyway.
- cjfont 8y agoJARs are okay, but they still require you to have the java runtime installed in order for them to execute.
- pjmlp 8y agoYou can bundle everything together, or use a commercial JDK with AOT compilation support.
- weberc2 8y agoWe're struggling using containers for our Python and Javascript applications. I think we're using venvs and node_modules and we oughtn't (or we shouldn't keep them in our project directory where they sometimes--but not always--get overwritten by source code volume mounts).
- dharmab 8y agoFor Python, this works well for my team: https://www.dharmab.com/programming/python/docker/2017/07/24/isolated-python.html https://www.dharmab.com/programming/python/docker/2017/07/24...
- dmitrygr 8y agoBased on the readme, this is basically an SFX archive, which runs a binary upon completion of extraction to temporary folder, much like winzip & the like have created for ages.
- munk-a 8y agoThat's a bit odd, with weird system configuration you'd still be able to inject a bad .dll or .so as a priority shared library. I don't know why you wouldn't just prefer to bundle all the function references into a single binary blob.
- GordonS 8y agoIt was me that posted this link, and I must confess I did so before reading the full readme. I had assumed this was some kind of clever AoT tool... but after reading more, I was rather disappointed :( I'm sure this still has some uses, but they are rather limited, and very dull.
- paavoova 8y agoHaven't read the link, but that's quite terrible, and the title is misleading. No idea why it's getting buzz in this thread. It means if tmpfs/equivalent isn't available, it will cause disk writes each application launch. Or if it caches the unpacking, it's susceptible to injection. And suppose the binary doesn't have write access to the filesystem?
- GordonS 8y agoIt's the latter - it caches and is susceptible to injection.
- marcoperaza 8y agoFrom the README, it looks like this unpacks the application and dependencies into a local cache in the user's home directory on first run. Is it possible to instead execute the compressed application code directly and present to the resulting process a virtualized file system that decompresses the dependencies on the fly too? Then you could run the single binary without giving it any filesystem access. One way to do it would be to have have the parent process provide some routine that the kernel can call for reading a particular "file". Or perhaps just have standard embedding and compression methods that the kernel supports, so you would just provide the offsets to the kernel. I guess what I mean is: do any platforms provide the necessary syscalls to pull that off?
- rkeene2 8y agoThis is in fact what Tcl Starkits/Starpacks do -- all access to the filesystem goes through the Tcl VFS, which can be redirected to the script level.
- the8472 8y agoYes, on linux you could do this with FUSE, in principle you would still need one empty temporary directory as mountpoint, but that can be placed in a tmpfs, e.g. under /run/user/ by convention.
- TheDong 8y agoWhat you describe can be accomplished on linux using mount namespaces. It gives a namespaced view of the filesystem which the parent process can control via picking its root location and what is bindmounted where. You've basically described a container where the filesystem is virtualized to a safe location on disk. Disallowing reading/writing data entirely is pretty easy with seccomp or ptrace, though that's not required with namespacing.
- jordwalke 8y agoI asked this question just yesterday. The challenge (the one I was interested in) was if this can be done in a general purpose way for arbitrary programming languages so long as they used libc to go through the file system. I actually think it might be theoretically possible using LD_PRELOAD and some equivalent on OSX. I don't know why I'm compelled to want the executable to leave no trace on disk, but it seems much cleaner. However, if you _write_ to the virtual file system, that will need to change the packed binary itself which might result in problems (the binary would grow/shrink as you pack more / less files into it at runtime by writing to the "disk"). https://twitter.com/jordwalke/status/1050277143085608960 https://twitter.com/jordwalke/status/1050277143085608960 Maybe it's just not worth it and a temp directory is not so bad. I just worry about little changes to the way `mktmp` works across OS updates etc. A totally self contained executable that leaves no trace, with virtual file system support is about as isolated and reliable as you could ever hope. I just don't think anyone has been determined enough to achieve it.
- open-source-ux 8y agoIt's fascinating how self-contained binaries are now coveted as such a desirable feature. Dozens of languages (if not more) were capable of this decades ago. In fact, some of them could generate self-contained binaries that allowed you to bundle app assets like icons and images inside the executable - no additional add-ons needed. And they were native executables, not complicated wrappers with layer upon layer of different technologies. This is not to take anything away from this project, just making an observation. The programming field often feels like it's in a giant loop constantly re-discovering what went before.
- weberc2 8y ago> The programming field often feels like it's in a giant loop constantly re-discovering what went before. It might feel that way, but this is driven by economics. Basically disk, memory, and bandwidth is far cheaper than it used to be so who cares if you waste a GB or so copying the same libraries all over if you don't have to solve for dependency hell?
- Koshkin 8y ago> disk is far cheaper Not in the cloud.
- weberc2 8y agoTo be clear, you're asserting that the price of disk in the cloud is comparable to what it was decades ago?
- Koshkin 8y agoThings are more complicated than that. (Depending on the pricing scheme, you might end up paying per gigabyte per hour, for example; which may or may not include transfer/storage of actual data; a faster "drive" could be much more expensive than a slower one, etc.)
- 8y ago
- matthewbauer 8y agoI think it’s funny to use java as an example when jar files have been around forever. Still there are some use cases for other things where statically linking is not possible. I wonder what benefits this has over other solutions? Originally when it said multi-platform i thought it meant that one binary could run on multiple platforms like a fat binary. But i think all of the examples require you to have a specific target in mind.
- monocasa 8y agoThere's something to be said for the fact that these are native binaries. Like under windows you'd be able just call CreateProcess with the file. Jars never did that and intrinsically needed an interpreter to run them.
- rictic 8y agoThis. A .jar file could be anything. Is it a library or an executable? When I run it, do I need to make sure its dependencies are on my classpath? What're the java command line flags I need? Do I have the right version of java on this machine? There's something to be said for an entirely self-contained native binary, at least as an easy on-ramp for users.
- GordonS 8y agoUnfortunately, it turns out this is not a completely self contained binary - it's more of a packer. So if you pack a Java app, you're still going to need the JRE, and if you pack a .NET Framework app, you're still going the .NET Framework, etc.
- haxton 8y agoSo I don't know what this is doing. But I know that microsoft is developing WPF to run in a fulled contained .NET Core binary. Meaning that there is no requirement of having any preinstalled software when running it. I would assume this technology to be translated into most any .NET core app.
- billconan 8y agoIt doesn’t seem to package my app correctly. I didn’t use dynamic loading, but it still say can’t open dependencies.
- fiatjaf 8y agoI don't understand a thing, is this for any kind of application or Rust? How will this thing know how to run my bizarre app in a bizarre language?
- treve 8y agoIt downloads the runtime for your bizarre language, assuming it's supported.
- mychael 8y agoWhat problem does this solve? How does it differ from Docker?
- tomc1985 8y agoThis solves some of the app packaging pain that goes into building desktop apps. Not sure how this works but it probably bundles dependencies into the binary a la static linking. Docker mocks the OS and virtualizes, its a very different approach I'm stoked for anything that eases the pain of desktop deployment that isn't Electron. Right now the cross-platform story is... complicated
- solarkraft 8y agoBut it seems like it continues one of the pain point of Electron: Shipping hundreds of MBs in the same dependencies for every single app.
- tomc1985 8y agoIndeed, but it's either that or you deal with the dependencies at deploy, or you write your software to not have any dependencies.
- tedivm 8y agoDocker isn't really used for desktop applications. Docker also requires a daemon (and on OSX a full virtual machine) to run. A better analogy for this system would be jar files that allow more languages than just java.
- tgtweak 8y agoFor starters you don't need docker installed, or any container at all. This is largely targeted to end-user deployments where you're providing a binary to users to run, and the current defacto is to make a self extracting exe which dumps all the dependencies along with your executable to a temp directory before executing. Rather messy. Edit: just looked at the warp implementation, it's a self extractor and executor
- tgtweak 8y agoLatest .net core 3 apparently supports single-bibary packaging. It has been a often-requested feature for at least 2 years. https://github.com/dotnet/corefx/issues/13329 https://github.com/dotnet/corefx/issues/13329 (The last comment referring to warp is indicative)
- wokwokwok 8y ago.Net Core 3 wont be out until next year. The current .Net Core 2.x runtime has had this feature since it came out though (last year?). (Hello world + runtime is like 100 MB and several hundred files)
- MarkSweep 8y agoFor what it's worth, the IlLinker can get Hello World down to 20MB and as many files: https://github.com/dotnet/core/blob/master/samples/linker-instructions.md https://github.com/dotnet/core/blob/master/samples/linker-in... If you use some extra options to disable crossgen and link System.Private.CoreLib, you can get the size down to 10MB, at the expense of startup time: https://github.com/AustinWise/IlLinkerExample https://github.com/AustinWise/IlLinkerExample
- jobigoud 8y agoYou can do this in regular .NET already using Costura Fody. It puts the DLLs (native or managed) as resources and load them at runtime. https://github.com/Fody/Costura https://github.com/Fody/Costura
- chriswarbo 8y agoReminds me of https://github.com/solidsnack/arx/ https://github.com/solidsnack/arx/ which I came across recently. I've not used it, but it sounds like it does a similar decompress-then-run approach, but using a shell script as the runner rather than a native binary.
- oscargrouch 8y agoIts funny. Im creating a multi-os application platform based on Chrome runtime, and thats how im packaging my multi-target binaries. Im using a SQLite btree file as backend, working basically as a key-value store. On the header i define what target triples are supported (the ones the dev built the binaries), and on the runtime open, see if the current host is supported and unpack the bynary payload always checking the hash before running it. (If its on the filesystem already see of the hash matches with the one saved before on the DB as a record) Giving its a db the owner of the public-key signature(the creator of the file), can modify and add others binaries latter. Im not disclosing it now, because its just part of a bigger platform, and you dont distribute only the apps, but a whole "container"(not in a Docker sense, but as a multiplatform container more focused on final users instead of cloud backends). I hope i can launch it soon here on HN.
- sandGorgon 8y agoThe best of breed here is zeroinstall - https://0install.net/ https://0install.net/. The software catalog is here - http://0install.de/catalog/ http://0install.de/catalog/ comes with its own SAT solver for dependency resolution. zeroinstall is what Canonical was considering before it created Snap Packages.
- pbhjpbhj 8y agoAny idea why they went with snap instead? There seems to have been a recent proliferation of new packaging systems -- which is fine. But, from an Ubuntu perspective I don't think they should be included if they're not integrated with apt (perhaps it is and I missed the memo?). Digikam moved to AppImage, which sounds cool but adds a further system one needs to update, and the automatic update never worked for me (I think their package didn't include the necessary metadata?). Having all application updates available in one place, and official packages be recent, is the 'killer feature' for me. I used to be a Slackware user, but when I migrated to slapt-get I realised I was done with rolling my own packages and checkinstall-ing them and wanted system where things were more managed - not having half the apps installed ad-hoc. When you have to install a new system to get just a few apps it feels like I might as well go back to ./configure;make;[check|make ]install. Seems we need an xdg install record with a unified app that calls the sub-system (pip, 0install, AppImage, shell script, dpkg, whatever) to abstract all that away.
- rkeene2 8y agoAppFS ( http://appfs.rkeene.org/ http://appfs.rkeene.org/ ) is an extension of the idea proposed originally by 0install (in their original LazyFS approach).
- EdSchouten 8y agoAnd for Python there is Subpar: https://github.com/google/subpar https://github.com/google/subpar
- metaphor 8y agotVQhhsFFlGGD3oWV4lEPST8I8FEPP54IM0q7daes4E1y3p2U2wlJRYmWmjPYfkhZ0PlT14Ls0j8fdDkoj33f2BlRJavLj3mWGibJsGt5uLAtrCDtvxikZ8UX2mQDCrgE Anyone know what this magic is all about? Honestly not feeling warp-packer downloading executable blobs during runtime either.
- dgiagio 8y agoIt's a placeholder in the final binary that gets replaced by the target application executable file name so Warp knows what to run. This is a known and established technique. I don't have the link right now (mobile) but .NET Core does something similar for their native deployment, for example.
- diggernet 8y agoGood point about the downloading. Warp-packer should have all the warp-runners installed locally with it. Or at the very least they should be packages you install depending on your target platforms.
- sick_of_web_dev 8y agoSo essentially this just puts all of your files in the resource section of a PE or ELF executable and extracts them when you run it. Not really very new or noteworty or am I missing something here?
- sigmonsays 8y agoso bash self extracting tarball of the 2000s?? https://www.linuxjournal.com/node/1005818 https://www.linuxjournal.com/node/1005818
- angst 8y agoI hope it can use downloaded executable for warp-runner. Downloading executable from windows console doesn't work too well in my corporate environment. Currently placing windows-x64.warp-runner.exe to same directory as warp-packer doesn't help. I really like the idea though. Simplest 3rd party solution I've encountered so far for the purpose.