5 ms·
What's the most portable way to include binary blobs in an executable?
- svnpenn 4y agoEasy with Go https://godocs.io/embed https://godocs.io/embed
- JonChesterfield 4y agoArticle is missing .incbin from file scope inline asm. Much more convenient than a separate .s file. Probably doesn't work on windows, no measurable compile time cost (unlike massive arrays of comma separated chars).
- mmastrac 4y agoOne missing approach is just appending the binary data to the end of the file, and then reading the resource from /proc/self/exe on Linux (or the equivalents on Mac and Windows). It's not "portable" per-se, but all modern platforms [1] have a way to interrogate the binary contents of the currently-running executable. [1] _NSGetExecutablePath, GetModuleFileName(), getexecname() etc EDIT: Apparently https://github.com/gpakosz/whereami https://github.com/gpakosz/whereami will manage a lot of this complexity for you
- naasking 4y agoAnd on microcontrollers where embedded binaries are essential?
- duskwuff 4y agoMost microcontrollers run code directly from flash memory -- there's no "executable file" (or, indeed, any files) involved at all.
- naasking 4y agoThe lack of files is exactly the point. Any kind of blobs need to be bundled with the compiled code in a -defined and preferably portable way, and this is the domain where it's used the most. The way the OP described just doesn't work in that context. We had a recent thread about the new #embed which does this properly and portably.
- anyfoo 4y agoDon't do this, it's ugly and relies on assumptions that aren't true. I haven't checked each spec, but it is very unlikely that your ELF/mach-O/PE/... is still valid with added junk at the end. You may try it out and it may work, but that is true for many things that may come back to bite you (or others) in spectacular ways.
- fabian2k 4y agoI'd be interested in any example where this approach would produce an invalid executable. I have used this without issues, but of course I have certainly not tried this in every possible environment.
- anyfoo 4y agoComputing history is chock full of examples where something "seems to work" but is actually invalid (and a mach-O treated that way would be invalid [EDIT: or just "not accepted" by some parts of the system, see below], whether it runs or not), and then Raymond Chen has to write a blog post about it decades later. Here's just one out of many as a random example: https://devblogs.microsoft.com/oldnewthing/20041026-00/?p=37473 https://devblogs.microsoft.com/oldnewthing/20041026-00/?p=37... Back to this particular case, the binary will fail strict code signing validation on macOS. It may still run because the kernel does not access the binary past the coverage of the code signature (and all the bits there are still intact), similar to how multiarch binaries work, but you will at least severely be hampered to distribute your binary, since Gatekeeper won't be happy either.
- dmitrygr 4y ago> it is very unlikely that your ELF/mach-O/PE/... is still valid with added junk at the end. I've written loaders for all of the executable formats you mentioned, and maybe a dozen more. I know of none where this would violate the strict interpretation of the word of the spec. That being said, valid file != happy OS
- anyfoo 4y agoAgreed. As above: It may for example run, but not be accepted by other parts of the OS (as evidenced).
- ncmncm 4y agoELF provides for any number of different kinds of "section" that you can have automatically mapped into your address space at startup. You just need a way for your program to know where it is. There are lots of different ways to get that.
- titzer 4y agoYes, but the article was mostly about what tools do you use to get that section into the ELF.
- kevin_thibedeau 4y agoThis is broken: .incbin "string_blob.txt" ... printf("%s\n", string_blob); Text files don't have to have a NUL termination. The proper way to embed data with the .incbin directive is to add a label after the file and use that directly for pointer arithmetic or compute the size with another assembly directive.
- kelnos 4y agoIn this case it works because the author explicitly put a NUL at the end of the string in the text file. I don't think the author was trying to suggest that you can do this with arbitrary data.
- jandrese 4y agoCouldn't you follow up the .incbin statement with a .const 0 or something similar?
- jviotti 4y agoMy team is working on this problem in the context of creating Node.js single-executable applications. While the naive approach of just appending data at the end of the binary works, it is not friendly with code-signature in macOS and Windows given that signing operates on PE and Mach-O sections. We have recently open-sourced a small tool called Postject (https://github.com/postmanlabs/postject https://github.com/postmanlabs/postject), which is able to inject arbitrary data as proper ELF/Mach-O/PE sections for all major operating systems (with AIX support coming). The tool also provides C/C++ cross-platform headers for easily traversing the final binary and introspect whether the segment is present or not. The tool is based on the LIEF (https://github.com/lief-project/LIEF https://github.com/lief-project/LIEF) project. At Postman, we are making use of this on our custom Node.js single-executable applications and soon on our custom Electron.js builds too.
- CMCDragonkai 4y agoDo you have a full example somewhere or using postject. We are currently using vercel/pkg and it is quite opaque.
- jviotti 4y agoThe full example is sadly closed-source for now. That said, we are closely collaborating with the maintainer of PKG as part as the Node.js SEA initiative, and we hope to be kickstarting an experiment on refactoring PKG to use Postject pretty soon.
- jll29 4y agoHere's a standalone (and Rust-implemented) version similar to xxd (if you don't like the vim dependency): https://github.com/jochenleidner/ltools/blob/main/src/bin/bintosrc.rs https://github.com/jochenleidner/ltools/blob/main/src/bin/bi... What I found is that many compilers don't like to compile very large source files; so if the binaries you'd like to integrate are big, it might be better to integrate their constituent objects one by one (if applicable).
- amethyst 4y agoMake a ZIP file containing the blob, and catenate it to the end of the executable binary. The ZIP format specifically puts all of the key metadata at the back of the file, so pretty much any ZIP tool can correctly read/list/extract data from the ZIP portion of the file. Anything that needs to be linked at runtime can just be extracted to a temp dir, and then cleaned up on exit. Bonus points for getting "free" compression on text data blobs. We do this for Python applications, by combining a ZIP containing the "link tree" of sources/packages/modules, with a shell bootstrap script that automatically sets up the environment, import path, etc, and Python itself has built in support for importing pure-python modules from a ZIP file. All that's needed for native modules is a simple import hook that extracts the native objects into temp space and then loads them appropriately.
- deivid 4y agoJust in case you are unaware, take a look at shiv: https://github.com/linkedin/shiv https://github.com/linkedin/shiv which does this quite neatly
- titzer 4y agoIndeed, JARs are just ZIP files (and can even be uncompressed), so this trick works to make self-executable JAR files that consist of a shell script concatenated with the actual JAR. https://blog.frankel.ch/creating-self-contained-executable-jars/ https://blog.frankel.ch/creating-self-contained-executable-j... https://coderwall.com/p/ssuaxa/how-to-make-a-jar-file-linux-executable https://coderwall.com/p/ssuaxa/how-to-make-a-jar-file-linux-...
- deleted 4y ago[deleted]
- EdSchouten 4y agoI think that approach only works on general purpose operating systems, right? It can’t be used to add assets to executables that are used on, say, embedded systems. Or WebAssembly apps.
- ghoward 4y agoThe answer to this is easy. At least it was for me; I didn't know it was such a problem. My solution is [1]. It generates a C file with a specific array name passed in through the command-line. It also has a few other niceties that I need. It works on Windows, Mac OSX, Linux, and the BSD's, no matter the compiler or linker. I use it to generate the arrays for the help texts ([2] and [3]), as well as two math libraries ([4] and [5]). People are welcome to adopt and adapt it. Just follow the license, as per usual. I've even adapted to my other software. [6] [1]: https://git.yzena.com/gavin/bc/src/branch/master/gen/strgen.c https://git.yzena.com/gavin/bc/src/branch/master/gen/strgen.... [2]: https://git.yzena.com/gavin/bc/src/branch/master/gen/bc_help.txt https://git.yzena.com/gavin/bc/src/branch/master/gen/bc_help... [3]: https://git.yzena.com/gavin/bc/src/branch/master/gen/dc_help.txt https://git.yzena.com/gavin/bc/src/branch/master/gen/dc_help... [4]: https://git.yzena.com/gavin/bc/src/branch/master/gen/lib.bc https://git.yzena.com/gavin/bc/src/branch/master/gen/lib.bc [5]: https://git.yzena.com/gavin/bc/src/branch/master/gen/lib2.bc https://git.yzena.com/gavin/bc/src/branch/master/gen/lib2.bc [6]: https://git.yzena.com/Yzena/Yc/src/branch/master/tests/strgen.c https://git.yzena.com/Yzena/Yc/src/branch/master/tests/strge...
- ufo 4y agoHow do you feel about that problem the parent blog post mentioned, of this being slow for large blobs particularly when compiling with Clang?
- hikarudo 4y agoYou can split it up into several files, then concatenate the arrays at runtime.
- CyberDildonics 4y agoThis can actually be solved by writing out to assembly and using an assembler to make the .o file to link in. That can deal with huge files with constant memory.
- ghoward 4y agoI've just run a benchmark on my code. It did 13875457.34 bytes per second, which means my program would have done the 82 MiB file he had in 6.20 seconds, faster than hexdump, GCC, and Clang. It used a max of 3223848 Kbytes, which is about the size of the file it was processing. (The file was 3300000020 bytes exactly.) I also tried with a file as close to 82 MiB as I could get. It used 85168 Kbytes max, and it took 6.77 seconds. My code could probably be optimized too. It tries to skip a header comment. It also reads all of the input file in at once, when it could probably stream it on demand. It is also checking for stuff to exclude, which takes time. If I take out the if statement that begins with: if (!strncmp(in + i, bc_gen_ex_start, strlen(bc_gen_ex_start))) That should remove most of everything else that does not matter. If I run that on the 82 MiB file I made, it uses 85124 Kbytes and 5.40 seconds. This is still far slower than objcopy and incbin, but maybe it's good enough, right? I think I could optimize it more, but I still think that's a pretty good showing against the competition, especially for portability. Edit: I forgot to mention that I did these tests while running a fuzzer (AFL++) on 15 of my 16 cores and while watching YouTube. I didn't want to stop the fuzzer just for this (it's been running for more than 24 hours).
- DethNinja 4y agoAssuming binary blob is relatively small: Just template generate and store the data as a bit array on the language of your choice. For example, if you are using C/C++ you can zip everything then use a small python script to generate a C/C++ header where this data is available as a uint8_t array. Keep in mind that all this data will be loaded to memory, so I don’t recommend this approach for anything north of 10mb.
- kazinator 4y agoOn a modern VM system, the static initialized data will be mapped to memory, not loaded. So you have to worry about its virtual footprint, not physical memory use.
- kelseyfrog 4y agohttps://thephd.dev/finally-embed-in-c23 https://thephd.dev/finally-embed-in-c23
- kazinator 4y agoIn TXR Lisp, I did this: https://www.nongnu.org/txr/txr-manpage.html#N-0389D15E https://www.nongnu.org/txr/txr-manpage.html#N-0389D15E There is a 128-byte area prefixed by the character sequence @(txr):. It normally contains all zeros (empty null-terminated string). If you put a non-empty UTF-8 string there, it gets executed. Of course, the problem of including a binary blob is trivial if it can just be declared as an array; the interesting problem is doing it to the executable, without doing any compiling or linking.
- branon 4y agoSomething like https://justine.lol/ape.html https://justine.lol/ape.html perhaps?
- tomn 4y agoMy colleague wrote this solution for C++ and cmake: https://github.com/ebu/libear/commit/40a4000296190c3f91eba79e5b92141e368bd72a https://github.com/ebu/libear/commit/40a4000296190c3f91eba79... This is a cmake function which generates C++ files using no external tools. It's probably not very fast, but if you don't need to handle big files and are already using cmake this is easy to integrate, adds no dependencies and works on all platforms.
- mrlonglong 4y agoC23 will soon have the #embed attribute to include such blobs. This will ease portability concerns.
- avrionov 4y agoThis was discussed a few days ago "Embed is in C23": https://news.ycombinator.com/item?id=32201951 https://news.ycombinator.com/item?id=32201951 C++ added "std::embed" https://open-std.org/JTC1/SC22/WG21/docs/papers/2020/p1040r6.html https://open-std.org/JTC1/SC22/WG21/docs/papers/2020/p1040r6...
- vgel 4y agoIf you're already using Clang (and thus LLVM & its platform constraints), I wonder if the best way would be to link in a tiny Rust / Zig `.o` using `include_bytes!` / `@embedFile`...
- a-dub 4y agoi'd probably use a constant string by writing a script and build tooling that takes my binary and spits out a source file (c?) which is then compiled in...
- UncleEntity 4y agoBlender has been using this exact technique forever for embedded fonts, icons, the splash screen, and who knows what else.
- Rizz 4y agoIf the converting to hex step is too slow, could you leverage your makefile or other build script to bootstrap a simple, faster converter to convert your data file?
- raggi 4y agoI had to do this in Fuchsia a while back, and the toolchain team kindly fixed up llvm-objcopy for this purpose. The section replacement flags now work as you would want for this use case (they didn't before).
- raggi 4y agoAlso, fwiw, embed isn't always the answer you want. The thing I was patching up was a large rust program (30+mb release stripped), and so it was undesirable to always have blob changes require a relink of the program. Embed as it is available in a few common languages now/soon is very convenient, but it is quite painful once the program link stage is expensive.
- anitil 4y agoSqlite has an append-vfs that allows you to append the database to your executable (or really any file I suppose) [0] . I believe (but haven't looked for a while and my memory is hazy) that it moves all its metadata to the end of the file rather than at the beginning. [0] https://www.sqlite.org/src/file/ext/misc/appendvfs.c https://www.sqlite.org/src/file/ext/misc/appendvfs.c Edit - the very first comment explains that is uses a trailing string
- anon291 4y agothe only thing you need for this is xxd --include name < binaryblob
- X-Cubed 4y ago> However, perhaps surprisingly, xxd is part of Vim (but not Neovim?), which is a rather heavyweight (and odd) dependency to require just to include a binary blob
- gumby 4y ago> Trying to solve objcopy's ugliness Hey, I wrote objcopy and objdump as debugging tools for developing bfd targets when I was designing bfd. Neither was (originally) intended as a production tool. OK I still use them myself, and I haven’t looked at the bfd source code in almost 30 years. I forget who came up with the tool chain target names, which I think was your real complaint, though I remember when. Perhaps Ian Taylor (later author of gold, which was notable for, among other things, not using bfd). Objdump should have a feature to generate assembly or c code for an arbitrary blob (with correct byte swap, if needed, of course).
- ltratt 4y agoI always try to be respectful towards other software in my writing and I fell short this time. I hope you'll accept my sincere apologies! > the tool chain target names, which I think was your real complaint Yes, this was solely what I was referring to (slightly thoughtlessly) as "ugly", and even then only in the sense of "where did those magic names come from?" I certainly wasn't referring to objcopy itself!
- gumby 4y agodon't worry: I'm not actually insulted. I just think it's funny that something intended for development debugging turn out to be actually useful (as I said I still use them too, and not for debugging bfd).
- mmastrac 4y agoI've been using objcopy and objdump for a _long_ time. You saved my bacon on some really obscure archs (QCOM hexagon, for instance). Thanks!
- warmist 4y agoMy approach to this problem was to write file_to_obj converter. It creates a x64 (msvc only currently) file that you can link to. The file contains "unsigned char[]" and size of it or if using c++ you can generate obj for std::array of bytes. Imho the binary format is not that difficult and every tool should target that (i.e. if using gcc target gcc obj format)