12 ms·
Embedding Binary Objects in C
- endless90 7y agoIs it possible to automate this with cmake?
- idoby 7y agoI used to have this automated as part of a makefile, so I'd say yes.
- tambre 7y agoYou may want to checkout the CMRC library (CMake Resource Compiler) [0]. [0]: https://github.com/vector-of-bool/cmrc https://github.com/vector-of-bool/cmrc
- je42 7y agonice!
- lolaso 7y agoperfect
- MaxBarraclough 7y agoNeat. Looks a little more complex than it needs to be though, and I'm surprised it targets C++ (and even relies on exceptions) with no support for C.
- kevin_thibedeau 7y agoYou can use add_custom_command() to run a script that generates an assembly file using the incbin directive.
- saagarjha 7y agoOr, as any CTFer worth their salt will tell you, __asm__(“.incbin file”);
- gpvos 7y agoCTF = ?
- cosarara 7y agoA capture the flag challenge https://ctftime.org/ctf-wtf/ https://ctftime.org/ctf-wtf/
- jagrsw 7y agoA real world example of use: https://github.com/google/honggfuzz/blob/075756bea8d1f08eb19d5de18e48d264038b853c/hfuzz_cc/hfuzz-cc.c#L28 https://github.com/google/honggfuzz/blob/075756bea8d1f08eb19... I think it's a bit better than ld -b binary ... cause it doesn't depend on the static linker generating symbols, which feels a bit like a non-standard/non-portable feature. But, who knows, maybe the vast majority of ld variants actually implement it.
- quietbritishjim 7y agoWhere did they define the symbols `_binary_quine_c_start` and `_binary_quine_c_end`? I would expect that the symbol names would need to be passed to the command that produced the object file from the binary file: ld -r -b binary quine.c -o myself.o -m elf_amd64
- CraneWorm 7y ago> I would expect that the symbol names would need to be passed to the command that produced the object file from the binary file: It's the other way around. The command produces these symbols basing on the file name: $ readelf --symbols myself.o Symbol table '.symtab' contains 5 entries: Num: Value Size Type Bind Vis Ndx Name 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND 1: 0000000000000000 0 SECTION LOCAL DEFAULT 1 2: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 1 _binary_quine_c_start 3: 0000000000000113 0 NOTYPE GLOBAL DEFAULT ABS _binary_quine_c_size 4: 0000000000000113 0 NOTYPE GLOBAL DEFAULT 1 _binary_quine_c_end
- idoby 7y agoThose are standardized named generated by ld, derived from the file name (in this case "quine", which is treated by the author both as C source and a binary data file). I bet there's a CLI switch to change the defaults.
- tyingq 7y agoThey are assuming gnu ld, and that's the pattern it uses. It will change some characters, like "." to "_" to keep valid symbol names. GNU objcopy has a way to change the generated names. As far as I can tell, ld does not.
- tyingq 7y agoThere's also gnu's objcopy. This post covers a lot of different approaches, and has lots of tips: https://www.devever.net/~hl/incbin https://www.devever.net/~hl/incbin
- asdfqwerty18 7y agothanks
- deleted 7y ago[deleted]
- fao_ 7y agoThe image background on that webpage is extremely distracting
- tyingq 7y agoMake your browser width narrow, and it drops to the bottom. Not that you should have to :)
- tyhoff 7y agoThanks for the link! That’s got a lot of goodies. One of us from Memfault also wrote a post about tips with Binutils, and this approach is covered in it. https://interrupt.memfault.com/blog/gnu-binutils#objcopy https://interrupt.memfault.com/blog/gnu-binutils#objcopy
- saagarjha 7y agoTIL objdump --visualize-jumps…
- regularfry 7y agoI got about 90% of the way to having a working ruby single-file executable builder which used objcopy to embed a sqlite database of the source files into an MRI build. Then YARV happened and the ruby build chain changed just enough that I needed to throw it out and start again. Every now and again I ponder having another go with mruby...
- bigfoot 7y agoRelated Twitter discussion: https://twitter.com/taviso/status/1250518818197245953 https://twitter.com/taviso/status/1250518818197245953
- ddevault 7y agoI wrote this tool a long time ago, which allows you to embed files and then access them as a "filesystem" with stdio. May be of interest: https://git.sr.ht/~sircmpwn/koio https://git.sr.ht/~sircmpwn/koio If you just want the contents of the file as a symbol, the approach described in the article is 100% the way to go.
- haberman 7y agoIf we're in the realm of "non-standard linker tricks"... Compilers will concatenate sections of the same name. You can use this trick to produce a concatenation of arrays across several files: $ cat t1.c __attribute__((section("some_array"))) int a[] = {1, 2, 3}; $ cat t2.c __attribute__((section("some_array"))) int b[] = {4, 5, 6}; $ cat t.c #include <stdio.h> extern const int __start_some_array; extern const int __stop_some_array; int main() { const int* ptr = &__start_some_array; const int n = &__stop_some_array - ptr; for (int i = 0; i < n; i++) { printf("some_array[%d] = %d\n", i, ptr[i]); } return 0; } $ gcc -std=c99 -o t t.c t1.c t2.c && ./t some_array[0] = 1 some_array[1] = 2 some_array[2] = 3 some_array[3] = 4 some_array[4] = 5 some_array[5] = 6 This is the mechanism that linkers use "under the hood" to get a list of C++ object initializers that need to run pre-main(). It's unfortunate that there is no standard way of getting at this functionality in portable C, or to get it in C++ without actually running code pre-main(). Sometimes you really want a linker-initialized list things (of some sort) that have been linked in, without actually running code pre-main() (which has all kind of issues). I would love to see a thing like this standardized in both C and C++. C++ compilers need it under the hood anyway.
- sheidavanesa 7y agonice
- haneefmubarak 7y agoHow exactly is the ordering of items across the section decided? Is it relatively random or is it based on the order that the object files are passed to the linker?
- monocasa 7y agoIt's unspecified, and I've seen it change over time as the linker changes it's data structures.
- RealityVoid 7y ago
- hikarudo 7y agoMy favorite is simply using a tool to create a C file with your binary data: static uint8_t mydata[] = {0xDE, 0xAD, 0xBE, 0xEF, ... }; The advantage is that it works anywhere, with any compiler. The disadvantage is that it can increase compile time. I would limit each .c file to 10MB; it seems there's a quadratic increase in build time with file size, at least with gcc. Also, instead of "0x%02x" I use decimal notation and no spaces in order to decrease the .c file size: static uint8_t mydata[] = { 42,7,105,0,0,..., 8,0,1,20,..., ..., };
- zamadatix 7y agoI wonder what causes it to be nonlinear?
- saurik 7y agoThe article specifically leads with this as the default technique and then wanted to describe a fun alternative.
- gpvos 7y agoIf you click through from the article to the mailing list post that inspired it, you'll see that that's what they originally used, but it created a memory problem.
- KMnO4 7y agoFor those wondering, the easiest way to do this is with: xxd -i filename.bin
- dasyatidprime 7y agoYes, so long as you're willing to take a build-dep on vim. An od+awk+sed+sh combo will get you there from a POSIX base—and you have to add some wrapper text around the output of xxd anyway…
- anthk 7y agoxxd exists in standalone form. Compile it. https://github.com/ConorOG/xxd https://github.com/ConorOG/xxd
- JoeAltmaier 7y agoI needed this last week! Building an embedded firmware image for a dashboard display, with lots of PNG files for icons etc. 61 of them. The original developer wrote a tool to expand the PNGs to BMPs (arrays of 32-bit pixel values) and generate a C array definition as a text file. Which is lots bigger than the original PNG (13K => 100K sort of thing). Then included that C source in the build. Used up 700K of my firmware image space which was only 1MB to begin with. So I wrote a tool to represent the raw PNG file as a C declaration, then added parts of libpng to the boot code to decompress the images during initialization. Even with libpng, I saved over 400K. Now the images use RAM instead of ROM, but that's ok I had buttloads of RAM. Anyway, this is a much slicker way of including binary files in an image. I may go back and change my build.
- JoeAltmaier 7y agoNope. Dumb Visual Studio doesn't support embedded binary in cross-compile. That's the tool the client wanted, so I'm stuck with it. Funny, they (VisualGDB) even have a tutorial on doing it. But the option 'Embedded Resource' doesn't exist in my (modern) Visual Studio. Ha! And the feedback I gave VisualGDB about their outdated tutorial, they responded "Just pay HERE and we'll be glad to solve your problem!" They can go ahead and have outdated documentation, screw them.
- RealityVoid 7y agoCould you not try using the objcopy method mentioned here outside Visual Studio (or somewhere before the build process) and then just using the .o at link time, just referencing it from the outside at linkage? I expect at least a way to: - hook custom tool calls - link external objects into the project Also, hello fellow automotive embedded SW eng! I know your suffering! :))
- JoeAltmaier 7y agoVisualGDB uses a tool-created makefile for cross-compiling and building. Not a lot of room for me to innovate. I suppose the .o trick could work. But it still requires careful hand-scripted builds. I already have that - I run a script to read a binary file and emit .c declaration, which I pipe to a C file and include in the project. I suppose it lets me drop support of the script, which is something.
- dirtydroog 7y agoC++ has a proposal for std::embed http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1040r0.html http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p104...
- beefhash 7y agoC has a proposal for #embed. http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2499.pdf http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2499.pdf
- jws 7y agoIn the section on search paths… It follows the same implementation experience guidelines as #include by leaving the search paths implementation defined, with the understand that implementations are not monsters and will generally provide… Really? We are going to "understand that implementations are not monsters" after what they've done with "undefined"? I think maybe these standards should be written from the perspective that implementations are sociopathic demons summoned against their will. (But I really want this feature. I regularly use xxd and some Makefile rules to embed assets in my executables. For instance, in a web service I might have all my default configuration and templates in the executable with command line options to send them to standard output and override them from external files. Then on the chance someone needs to make a change they can just make their own file and use it.)
- rwj 7y agoMay I ask what's wrong with using xxd? I don't have an opinion about C, but C++ is already pretty complicated, and adding features to the language when there are already well-known solutions doesn't seem wise. (For the record, I'm aware that there are size limitation when using xxd, but there are also other solutions).
- deleted 7y ago[deleted]
- 7y ago
- nemanjaboric 7y agoAccording to https://news.ycombinator.com/item?id=22865842 https://news.ycombinator.com/item?id=22865842 #embed proposal is in the review process http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2499.pdf http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2499.pdf . This similar (same?) proposal is also presented and in the review at WG21 to be included in C++ standard. The author puts a lot of work towards making this proposal a reality (as far as I as a casual twitter/slack observer can see) and I'm looking forward to it.
- Jasper_ 7y agoUnfortunately, the proposal has been stalled by the C++ committee and the author is uninterested in continuing it. See their post here https://thephd.github.io/full-circle-embed https://thephd.github.io/full-circle-embed
- saagarjha 7y agoIs Circle something that is actually coming to C++?
- mmm_grayons 7y agoThere's also Drew Devault's Koio: https://git.sr.ht/~sircmpwn/koio https://git.sr.ht/~sircmpwn/koio It works pretty well, though I don't believe it works on Windows.
- megous 7y agoUsing GNU assembly .inbbin is also quite flexible. And you can use .dc.* and .asciz and .byte to add arbitrary data inline. Generating assembly files is my favorite way of including outside data into my C programs.
- tyingq 7y agoFound Chromium is using a shell script that calls od and sed. https://chromium.googlesource.com/chromiumos/platform/ec/+/master/util/bin2h.sh https://chromium.googlesource.com/chromiumos/platform/ec/+/m...
- andrewshadura 7y agoBack in the day Borland shipped a tool called binobj which did exactly this.
- kazinator 7y agoI wouldn't do it this way, within the toolchain. Better to just append the data to the finished executable, with a header that contains an identifying marker. The program can scan its own image and look for that marker to get at the data. That will work on any OS with any executable and linker format and can be done post-production (users or downstream distributors can receive the binary and add customized data to it without requiring a dev environment with a toolchain, just tiny utility you can bundle with the program). Scanning for the marker can be avoided, if we do the following: /* inside the program, at file scope */ struct { char marker[16] = "dW5pcXVlbWFyawo=" uint32 offset; } binary_stuff; Then your tiny utility programs opens the executable and looks for the sequence "dW5pcXVlbWFyawo=" inside it. Having found it, it puts the offset of the data into the 32 bit offset field which follows, and writes the data at that offset. When the program runs, if it sees a nonzero value in binary_stuff.offset, it can open its image and directly proceed to that offset to read the stuff or map it: no searching.
- tantalor 7y agoSounds like a great vector for malware.
- kazinator 7y agoAny unsigned native executable is a vector for malware. Images with stuff appended to them can be signed. For instance, think of signed Linux kernel images on secure embedded systems. The signature includes the embedded initramfs and device tree blob.
- mwcampbell 7y agoThe advantage of the approach described in the article is that there's less work to do at startup time. For applications that run on end-users' machines, we should do whatever we can to minimze work done at startup, getting it as close as possible to just mapping a big blob into memory and jumping to the code that does the real work of the program.
- kazinator 7y ago
- shaklee3 7y agoTed has one of the best, and longest running tech blogs I've seen, and it covers such a wide array of topics. If you're reading Ted, thanks.
- pdw 7y agoUnfortunately, this technique is a bit problematic with modern C compilers. Because the `start` and `end` symbols are unrelated objects, as far as the compiler is concerned, the subtraction `&end - &start` to get the length of the data invokes undefined behavior. Just for that reason, I feel the include file with a hex dump is the better method.
- ghostpepper 7y agoCan you elaborate on this? Does "unrelated" have any special meaning in this context? Why is this operation undefined?
- saagarjha 7y agoIn C, accessing data outside the bounds of an object is undefined (like, you can’t legally “go past the end” of an array and end up in some other object). The “start” and “end” pointers, as far as the compiler is concerned, are totally different objects, so it may optimize a loop from one pointer to the other out since it’s impossible to increment an address so it’ll go from pounding at one thing to another.
- ghostpepper 7y agoSo is the idea that just because the two objects happen to appear sequentially in memory under a certain implementation, the compiler (or linker in this case?) has no obligation to ensure that assumption holds?
- saagarjha 7y agoCorrect. (Although, the pointers being defined that way seems to make it quite unlikely that the compiler could optimize this incorrectly…)
- 08-15 7y agoWhere is data outside any object accessed? Note that forming a pointer just past the end of an object is well-defined; only dereferencing it isn't.
- damnyou 7y agoThis is simply the include_bytes! macro in Rust. There's also the include_str! macro which does a compile-time check for UTF-8 validity.
- mhh__ 7y agoI'm pretty sure this is trivial in D too although you have to tell the compiler the file exists (you can't do arbitrary FS reads at compile time)
- zevv 7y agoTotally not relevant for this post, but this is how we do this in Nim: const a = readFile("mydata") The `const` makes the expression evaluate at compile time, conveniently slurping the file into `a`, where it will be available at run time.
- mikepurvis 7y agoThat is pretty cool, but also not the behaviour I'd expect coming from other languages where this would be "get the contents of this file from the current directory of my runtime environment, and assign them to immutable variable `a`."
- zevv 7y agoUnderstandable; there's `let` for run time and `const` for compile time evaluation.
- RealityVoid 7y agoHeh, I used this trick when 2 years ago went to Hackaday Belgrade and did a port of a NES emulator on the badge. The ROM that was being played would be compiled using the same basic method. Neat!
- pieterk 7y agoIf you do this with a dynamically loaded library that you write to during execution, can C become a dynamic language?
- simias 7y agoAny turing language can be as dynamic as you want if you're motivated enough. After all, many interpreted languages are implemented in C... Many dynamic recompiler/JIT implementations effectively do something like what you describe: they map some executable portion of memory and output native code that is then executed.
- loeg 7y agoYou've invented JIT.
- rgovostes 7y agoOne of the Capture the Flag challenges at CCC last year was to figure out a way to leak the contents of a file on a remote compiler server. The server would accept some C code and just give you a boolean true/false value whether the code compiled or not---without ever executing it. Others came up with a solution abusing the C preprocessor by defining macros that would make the known structure of the file valid C and therefore they could just #include it. But my solution works with arbitrary files, without knowing the structure beforehand. As others have pointed out here, you can use inline assembly and the .incbin directive to include a file. But how could that influence whether the compilation succeeds or fails? I figured out how to guess a byte of the file and create metadata sections only accepted by the linker if the guess was correct. https://ryan.govost.es/2019/12/18/compilerbot.html https://ryan.govost.es/2019/12/18/compilerbot.html
- saagarjha 7y agoWe didn't actually use .incbin on that challenge, interestingly; however, we did use it (along with some nested static constructor function trickery) for Online Calc from TokyoWesterns CTF. For that we had some straightforward tricks to get around the forbidden characters, then we abused the flag format to get the #include to work. After that we could leak the flag byte-by-byte using Linux's BUILD_BUG_ON_ZERO, which is basically an upgraded static_assert. I think this is the code we ran: """ _Pragma("clang diagnostic push") _Pragma("clang diagnostic ignored \\"-Wtrigraphs\\"") ??= define STRINGIZE(...) ??=__VA_ARGS__ ??= define EXPAND_AND_STRINGIZE(...) STRINGIZE(__VA_ARGS__) ??= define hxp EXPAND_AND_STRINGIZE( ??= define BUILD_BUG_ON_ZERO(e) (sizeof(struct <% int:-!!(e); %>)) ??= define BUILD_BUG_ON(condition) ((void)BUILD_BUG_ON_ZERO(condition)) const char flag = ??= include "flag" )[{}]; BUILD_BUG_ON(flag == {}); _Pragma("clang diagnostic pop") """ (The {}, of course, being Python format parameters from our script.)
- abaines 7y agoYou may want to add -z noexecstack or linking the object will give your program an executable stack. e.g: ld -r -b binary -z noexecstack input.bin -o output.o
- nicoburns 7y agoIn Rust: `let foo = include_bytes!("path/to/file")`. And people wonder why I think C is difficult.
- jfkebwjsbx 7y agoC does not have the feature (yet), it is not about being difficult or not.
- nicoburns 7y agoThe fact that C doesn't have the feature makes it difficult if I want to perform that task!
- Gibbon1 7y agoSeems like a trivial feature to add though. Like all the other features C has been missing for the last 30 years.
- jfkebwjsbx 7y agoThing is, embedding binary files is not that useful, and there are easy enough ways to do it for those few that really needed it. The C standard has been very cautious about adding features that would make the standard and the compilers more complex since it is the lingua franca of computing. Nowadays there is really just a handful of archs in use and their compilers are all backends for GCC/LLVM, so they are relaxing a bit the gating of features.
- samrat 7y agoI am in India and I cannot access the site. The DNS resolves to `23.227.131.12` but the server doesn't seem to respond to pings either. I can access the site through a VPN. Anyone else facing the same issue? Anyone know what's up with that?
- balnaphone 7y agoSame, from Canada. My guess is that it will come back up later.
- felixguendling 7y agoThe linked page is currently not available. Archive link: https://web.archive.org/web/20200416183057/https://flak.tedunangst.com/post/embedding-binary-objects-in-c https://web.archive.org/web/20200416183057/https://flak.tedu... There are also ways to do this on Windows and Mac OS X. This library provides CMake support to do this: https://github.com/motis-project/res https://github.com/motis-project/res
- MaxBarraclough 7y agoDo you know how it compares with CMakeRC? See https://news.ycombinator.com/item?id=22888879 https://news.ycombinator.com/item?id=22888879
- ryanmjacobs 7y agoDamn I want to read this, but looks like it's down? 4/16/20 ~2 PM PST
- soylentgraham 7y agoUsed to do this for embedding stuff into homebrew GBA roms! :)