19 ms·

Embed is in C23

483 points by aw1621107 4y ago

thrwyoilarticle 4y ago
Other stuff: https://twitter.com/rcs/status/1550526425211584512 https://twitter.com/rcs/status/1550526425211584512 nullptr! auto! constexpr!
- phkahler 4y ago
  Not sure about the value of nullptr! Also not sure about auto! In C.
  - camel-cdr 4y ago
    nullptr since we have type detection now, and NULL mustn't be a pointer. auto, because otherwise everybody would create their own hacky auto using the new typeof.
trinovantes 4y ago
Off the top of my head, I think there's some niche use in embedding shaders so that they don't need to be stored as strings (no IDE support) or read at runtime (slower performance).
- matthews2 4y ago
  Nice for binary shaders too, e.g. SPIR-V bytecode generated by glslc.
- spicyjpeg 4y ago
  There are a lot of use cases for baking binary data directly into the program, especially in embedded applications. For instance, if you are writing a bootloader for a device that has some kind of display you might want to include a splash screen, or a font to be able to show error messages before a filesystem or an external storage medium is initialized. Similarly, on a microcontroller with no external storage at all you need to embed all your assets into the binary; the current way to do that is to either use whatever non-standard tools the manufacturer's proprietary toolchain provides, or to use xxd to (inefficiently) generate a huge C source file from the contents of the binary file. Both require custom build steps and neither is ideal.
- phoboslab 4y ago
  You can get some IDE support with a simple preprocessor macro[1]. It's a crutch, but at least you don't need to stuff the shader into multiple "strings" or have string continuations (\) at the end of every line. Plus you get some syntax highlighting from the embedding language. I.e. the shader is highlighted as C code, which for the most part seems to be close enough. [1] https://github.com/phoboslab/pl_mpeg/blob/master/pl_mpeg_player.c#L102-L131 https://github.com/phoboslab/pl_mpeg/blob/master/pl_mpeg_pla...
  - trinovantes 4y ago
    That's pretty clever Thanks, I'll remember to use that in my future shaders
- avianes 4y ago
  Another typical use is embedding a public-key in an application or firmware.
junon 4y ago
This will simplify a lot of build pipelines for sure. One thing that isn't clear from skimming the article, how do you refer to the embedded data again?
- jsnell 4y ago
  > The directive is well-specified, currently, in all cases to generate a comma-delimited list of integers I.e. you most likely use it to initialize a static variable, and then refer to that variable.
  - junon 4y ago
    Ah so this basically? static const char d[]={ #embed ... }; EDIT: ah it was showing up more like a comment which made it hard to spot.
    - jonathrg 4y ago
      Yes. There is an example in the article
throwaway38583 4y ago
Congratulations to the author. Things like this are why I hope Carbon exists. Evolving c++ seems like a dumpster fire, despite whatever compelling arguments about comparability you are going to drop on me.
- ActorNightly 4y ago
  The issue is that a lot of people just think about languages in a wrong way, which is the whole reason for pointless things like C++ expansions, Carbon, Rust, and stuff like this. One of the fundamental ideas that people run with in language creation/expansion is "programmer is stupid and/or make mistakes" -> "lets add language features that intercept and control his stupidity/mistakes". And there is a very valid reason for this - it allows programmers of lesser skill and knowledge base to pick up codebases and develop safe software, which has economic advantages in being able to higher less experienced devs to write software at lower salary points and spend next to no time fixing segfault issues due to complex memory management. The whole reason Java got so popular over C++ was because of its GC - both C++ and Java supported fairly strong typing with classes, but C++ still had a lot of semantics around memory management that had to be taken care of, whereas with Java you just simply don't do anything. However, people are applying this idea towards lower level languages, because they want the high performance of a compiled language with a whole bunch of features that make writing code as mistake free as possible. And my challenge to that is this - why not spend the time making just smarter compilers/tooling? Think about a hypothetical case where Rust gets all the features added to it that people want, and is widely used as the main language over all others. Looking at all the code bases, there will be a lot of common use patterns, a lot of the safety code duplicated over and over in predictable patterns, e.t.c. And you will see these common things added to Rust. Just like with Java a lot of the predictable use patterns got abstracted into widely used libraries like Lombok, Spring, e.t.c, where you don't have to worry about correctness in lieu of using a library. And you essentially will start to move towards more and more stuff being handled for you automagically, which is all part of the compiler/toolchain In the same way, #embed can be solved by smart compiler. Have a static string that opens a file, and read contents into a buffer that doesn't change? Auto include that file in a binary if you want to target performance rather than executable size. No need for special instruction, just be smart about how you handle an open call, and leave the fine tuning of this to specific compiler options. And from an economic perspective of ease of use from above, you would have a language like Python which is super easy to pick up and program in, except instead of the interpreter, you would have a compiler that will spit out binaries. Python is already widely adopted primarily of how easy it is to set up and use. Now imagine if you had the option to run a super smart compiler that highlights any potential issues that come with dynamic typing because it understands what you are trying to do, fixes any that it can, and once everything is addressed, it spits out an optimized memory safe executable. With Rust, you code, compile, see you made a mistake somewhere with a reference, fix it, repeat. With this, you would code, compile, fix the mistake somewhere that the compiler warns you about, repeat. No difference. Focusing on the toolchain also lets you think about integrating features from languages like Coq with provability, where you can focus not only on correctness processing/memory wise, but also "is the output actually correct". I.e, any piece of code for all given input can be specified to have guaranteed bounded output set, which you can integrate into IDE tools to provide you real time feedback on this for you to design the code in a way that avoids things like URL parsing mistakes, which all the languages safety features of Rust won't catch. As for C, you leave it a version that has a stable, robust ABI, and then anything that you need to support will be delegated to custom tools. That way, in the future where compute will likely be full of specialized ML chips, instead of worrying about writing the frontend to support every feature, you quickly get a notional tool chain made and are able to run existing C code.
aaaaaaaaaaab 4y ago
C++ keeps kicking ass! Feel sorry for crab people.
- jonathrg 4y ago
  That's not really the right conclusion to draw from this article
- orf 4y ago
  The article definitely isn’t a glowing praise of C/C++. In fact, including this simple, useful feature that rust has had for a decade now has taken an immense amount of effort and received so much pushback from various parties, in part due to the strangled mess of various compiler limitations and in part because of design-by-committee stupidity. C/C++ seems to be kicking it’s own ass.
  - gilnaa 4y ago
    Not to mention that it didn't even get into C++
  - msla 4y ago
    The article doesn't even mention C++/Java.
    - tialaramex 4y ago
      Second paragraph of the title article: > Surprisingly, despite this journey starting with C++ and WG21, the C Committee is the one that managed to get there first Later it mentions presenting their first formal attempt at this to Belfast 2019, that's a C++ meeting, it's too late for this to go into C++ 20 at that point, but it easily could have been in C++ 23 (it is not).
- agluszak 4y ago
  "crab people" means Rust people?
  - 4y ago
Cloudef 4y ago
I've always used xxd -i for embedding, doesn't have the mentioned problems and works everywhere, as it simply outputs a header file with byte array.
- jsnell 4y ago
  The article spends a fair bit of time discussing the build speed and memory use problems with that approach. Like, the benchmark results [0] linked to from this post literally have xxd as one of the rows. It's not a viable option for embedding megabytes of data. [0] https://thephd.dev/embed-the-details#results https://thephd.dev/embed-the-details#results
  - tempodox 4y ago
    And even if the data is small enough, not every C programmer uses Unix or knows their way around it.
- mikepurvis 4y ago
  But you have to have build system stuff for that and it's obviously non portable.
  - Cloudef 4y ago
    True, i dont personally ever have problem with this because i always compile from unix system anyways. (Even for windows)
- pdw 4y ago
  Well, congratulations, you now have a build dependency on Vim. (xxd is not a standard tool, it ships with Vim.) It's also only suitable for tiny files: compile time and RAM requirements will blow up once you go beyond a couple of megabytes.
  - Cloudef 4y ago
    Yeah, these are reasonable arguments against it
eps 4y ago
Re: #embed </dev/urandom> Just a random thought, but I'd expect a compiler to do exactly what's described if I tell it: static char foo[123] = { #embed </dev/urandom> }; This would address the most common case with infinity files, and then just let the compiler error out if the array size is not specified.
- tazjin 4y ago
  Reproducible builds crowd wailing in agony
  - huhtenberg 4y ago
    The context is that of infinite files, not of the urandom specifically. Give the linked post a read for details.
  - astrange 4y ago
    They just need a reproducible urandom.
    - elcritch 4y ago
      Looks like you forgot to add your sponsor disclaimer "message sponsored by NSA". ;)
      - astrange 4y ago
        Well, just don’t do networking in the VM your builds run in and you’re good, maybe.
        tazjin 4y ago
        Remove /dev/urandom and replace it with /dev/zero in your sandbox ;)
- kibwen 4y ago
  I would expect that to produce an error, though. If I had a regular file that was not infinite in size, and I specified the wrong length for the array, I would find it more useful to have the compiler inform me as to the discrepancy rather than truncate my file.
ascar 4y ago
This is a cool feature, but the author doesn't do himself any favors with his style of writing that greatly overestimates the importance of his own feature in the great scheme of things. Remarks like "an extension that should’ve existed 40-50 years ago" make me think, if we should've really bothered all compiler vendors with implementing this 40-50 years ago. After all, you can already a) directly put your binary data in the source file like shown after the preprocessor step and b) read a file at runtime. I'm not saying this isn't useful, but it's a rather niché performance improvement than a core language feature.
someweirdperson 4y ago
How to read unsigned data? Is there a stadardized parameter, or does this require a vendor extension?
- jonathrg 4y ago
  You just make your array type uint8_t or whatever you need as long as it supports integer literals. See section 4 in https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#intro https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#i...
  - someweirdperson 4y ago
    Scary, it's as if the preprocessor has become type-aware. I guess I better don't imagine the result of the preprocessing to look similar to and following the same rules as something I would have written by hand. This might make manual inspection of the preprocessed file a bit painful.
    - elcritch 4y ago
      Its not really a pre-processor stage. Probably better to think of it more like a pointer cast to some binary blob. Though it'd be interesting to see what `gcc -E` would produce.
tempodox 4y ago
> Even among people who control all the cards, they are in many respects fundamentally incapable of imagining a better world or seizing on that opportunity to try and create one, let alone doing so in a timely fashion. That does sound soul-crushing. Congrats on this achievement!
- quelsolaar 4y ago
  This is simply wrong. We (the ISO wg14) don't hold the cards, compilers are free to implement what ever they want, users are free to use what ever tools or languages they want. We exist only as long as we are trusted to be good stewards, and only go forward with the consensus of the wider community.
  - unreal37 4y ago
    You're both right. It's amazing that you and the ISO team are good stewards of the C standard. Thank you for being part of that. And it can also be true that it was "hell" and "hardly worth it" for the OP to get a new feature added to the language. I believe it was a miserable experience that has him questioning how he spends his time. Both can be true. Thank you for your efforts. And thank the OP for his efforts too.
  - morelisp 4y ago
    > > Even among people who control all the cards, they are in many respects fundamentally incapable of imagining a better world or seizing on that opportunity to try and create one, let alone doing so in a timely fashion. > This is simply wrong. We (the ISO wg14) don't hold the cards, compilers are free to implement what ever they want, users are free to use what ever tools or languages they want. This is an incredibly oblivious realization of JeanHeyd's point.
  - gtirloni 4y ago
    > (the ISO wg14) don't hold the cards That "standard" card seem to be a pretty huge one though.
    - 4y ago
kibwen 4y ago
This serves the same use as Rust's `include_bytes!` macro, right? Presumably most people just use this feature as a way to avoid having to stuff binary data into a massive array literal, but in our case it's essential because we're actually using it to stuff binaries from earlier in our build step into a binary built later in the build step. Not something you often need, but very handy when you do.
- kzrdude 4y ago
  for both Rust and C, these features "just" make something you could otherwise do with the build system and generated code easier, I think.
  - masklinn 4y ago
    As the article quotes, in C the lack of standardisation makes this tricky when you want to support more than one compiler, or even when you want to support just one compiler (cf email about the hacks to make it work on GCC with PIE).
- tialaramex 4y ago
  This has different affordances than std::include_bytes! but I agree that if you were writing Rust and had this problem you'd reach for std::include_bytes! and probably not instead think "We should have an equivalent of #embed". include_bytes! gives you a &'static [u8; N] which for non-Rust programmers means we're making a fixed size array (the size of your file) full of unsigned 8-bit integers (ie bytes) which lives for the life of the program, and we get an immutable reference to it. Rust's arrays know how big they are (so we can ask, now or later) but cannot grow. #embed gets you a bunch of integers. The as-if rule means your compiler is likely to notice if what you're actually doing is putting those integers into an array of unsigned 8-bit integers and just stick all the file bytes in the array, short cutting what you wrote, but you could reasonably do other things, especially with smaller files.
oefrha 4y ago
> vendor extensions ... were now a legal part of the syntax. If your implementation does not support a thing, it can issue a diagnostic for the parameters it does not understand. This was great stuff. I can’t be the only one who thinks magic comment is already an ugly escape hatch, adding a mini DSL to it that can mean anything to anyone just makes it ten times worse. It’s neither beautiful nor great. > do extensions on #embed to support different file modes, potentially reading from the network (with a timeout), and other shenanigans. (Emphasis mine.) My god.
- greatgib 4y ago
  I guess you don't know C. "#" is not a symbol for a comment line but the one for a pre-processor directive. Like #include stdlib.h In c/c++ you use // and /* */ for comments.
  - oefrha 4y ago
    I’ve known C for close to two decades, thank you. I’m using the not at all well defined term “magic comment” to loosely refer to everything that’s not strictly speaking code but has special meaning, which include pre-processor directives. cpp is definitely a well-hated part of C.
    - Ensorceled 4y ago
      > I’ve known C for close to two decades, thank you. I’m using the not at all well defined term “magic comment” Please forgive those of us who've been using C since the 80's, or earlier, from assuming you don't know C when you invent your own terminology for preprocessor directives.
      - oefrha 4y ago
        This is not a preprocessor directive though, from reading the post I don’t think cpp is expanding the #embed into an array initializer, otherwise there’s no performance benefit at all.
orbifold 4y ago
Can’t you just add binary data into a custom section of your ELF executable?
- nicoburns 4y ago
  The article address this directly. If you're only targeting one platform then this is reasonably easy (albeit still not as easy as #embed), but if you need to be portable then it becomes a nightmare of multiple proprietary methods.
- rleigh 4y ago
  Yes, but it's linker-specific and non-portable. It can also come with some annoying limitations, like having to separately provide the data size of each symbol. In some cases this might be introspectable, but again comes at the expense of portability. ELF-based variants of the IAR toolchain, for example, provide a means of directly embedding a file as an ELF symbol, but without the size information being directly accessible. GNU ld and LLVM lld do not provide any embedding functionality at all (as far as I can see). You would have to generate a custom object file with some generated C or ASM encoding the binary content. MSVC link.exe doesn't support this either, but there is the "resource compiler" to embed binary bits and link them in so they can be retrieved at runtime. Having a universal and portable mechanism which works everywhere will be a great benefit. I'll be using it for compiled or text shaders, compiled or text lua scripts, small graphics, fonts and all sorts.
  - acka 4y ago
    This article[1] shows how you can use GCC toolchain along with objcopy to create an object file from a binary blob, link it, and use the data within in your own code. [1] https://balau82.wordpress.com/2012/02/19/linking-a-binary-blob-with-gcc/ https://balau82.wordpress.com/2012/02/19/linking-a-binary-bl...
    - spc476 4y ago
      Great for Linux. Less so for Mac OS-X and Solaris (all three of which I use at work).
moffkalast 4y ago
This reminds me, I'd argue that the explosion of JS frameworks can be mainly blamed on one thing: the lack of an <include src="somemodule.html"> tag. If you have that you basically have 80% of vue.js already natively supported. No clue why this was never added in any fashion. Change my mind.
- polskibus 4y ago
  Is <script type="module" /> not sufficient for your needs? If not then what is missing?
  - nkozyra 4y ago
    Seems to be arguing for modular layout/templating, which is what virtual includes did (the cgi in the example would hypothetically output html)
- lkschubert8 4y ago
  Wouldn't the include still need some templating functionality? Or are people using vue that heavily for just importing static html?
  - ear7h 4y ago
    Not the parent comment, but my personal use case is for rendering a selectable list. The server side would render a static list with fragment links (ex. `#item-10`) and include elements with corresponding IDs, and a `:target` css rule to unhide the element. This would hopefully be paired with lazy loading the include elements. edit: My goal is to avoid reloading the page for each selection and rendering all items eagerly. JS frameworks are the only ones that really allow this behavior.
- TheAceOfHearts 4y ago
  HTML imports were part of the original concept of Web Components, and I think they were supported in Chrome. If you look up examples of things built with Polymer 1.x, it was used extensively. It was actually pretty neat, because you could have an HTML file with a template, style, and script section. Safari rejected the proposal, so it had to get dropped. But ESM makes it a bit redundant anyway. The end-goal is to allow you to import any kind of asset, not just JS. There have been demos and examples of tools supporting this going back over half a decade at this point.
quelsolaar 4y ago
I represent Sweden in the ISO WG14, and I voted for the inclusion of Embed in to C23. Its a good feature. But its not a necessary feature and I think JeanHeyd is wrong in his criticism of the pace of wg14 work. I have found everyone in wg14 to be very hardworking and serious about their work. Cs main strengthen is its portability and simplicity. Therefore we should be very conservative, and not add anything quickly. There are plenty of languages to choose form if you want a "modern" language with lots of conveniences. If you want a truly portable language there is really only C. And when I say truly, I mean for platforms without file systems, or operating systems or where bytes aren't 8 bits, that doesn't use ASCI or Unicode, where NULL isn't on address 0 and so on. We are the stewards of this, and the work we put in, while large, is tiny compared to the impact we have. Any change we makes, needs to be addressed by every compiler maintainer. There are millions of lines of code that depend on every part of the standard. A 1% performance loss is millions of tons of CO2 released, and billions in added hardware and energy costs. In this privileged position, we have to be very mindful of the concerns of our users, and take the time too look at every corner case in detail before adding any new features. If we add something, then people will depend on its behavior, no matter how bad, and we therefor will have great difficulty in fixing it in the future without breaking our users work, so we have to get it right the first time.
- pif 4y ago
  Thank you for your post! Thank you especially for reminding everybody that programming is much more than web programming and information systems.
  - quelsolaar 4y ago
    Thank you, Its also worth remembering that a lot of higher level languages have runtimes / VMs are implemented in C. Web applications rely heavily on databases, java script VM, network-stacks, system calls and operating system features, all of which are impemented in C. If you are a software developer and want to do something about climate change, consider becomming a compiler engineer. If you manage to get a couple of tenths of a percent performance increase in one of the big compilers during your career, you will have materially impacted global warming. Compiler engineers are the unsung heroes of software engineering.
jesprenj 4y ago
What about creating object files from raw binary files and then linking against them? That's what I (and of course many others) do for linking textures and shaders into the program. It's a bit ugly though that with this approach you can't generate custom symbol names, at least with the GNU linker. This #embed feature might be a nice alternative for small files. Well for large files you usually don't even want to store them inside the binary, so the compilation overhead might be miniscule, since the files are, by intention, small. When I read the introduction of the article - about allowing us to cram anything we want into the binary - I was hoping to see a standard way to disable optimizations (When the compiler deletes your code and you don't even notice).
- dark-star 4y ago
  One reason against this is mentioned in the letter that is quoted in the article
- jonathrg 4y ago
  It depends on your definition of small files. A few hundred kB to a few megabytes will make compilation speed and memory usage explode if you embed it as text, see section 3.2 in https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#intro https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#i...
- mastax 4y ago
  You reminded me of Bethesda Softworks games, which always seem to have 1GB+ executables for some reason. I hope it isn't all code. Maybe they embed the most important assets that will always need to be loaded.
  - GuB-42 4y ago
    My guess is that the files are not truly embedded as it would require to load the entire file in memory before running the application, which seems wasteful. More likely, the actual executable is only a small part of the file which accesses the rest of the file as an archive, like a self-extracting zip. There may also be some DRM trickery going on.
pwdisswordfish9 4y ago
> “Touch grass”, some people liked to tell me. “Go outside”, they said (like I would in the middle of Yet Another Gotdang COVID-19 Spike). My dude, I’ve literally gotten company snail mail, and it wasn’t a legal notice or some shenanigans like that! Holy cow, real paper in a real envelope, shipped through German Speed Mail!! This letter alone probably increases my Boomer Cred™ by at least 50; who needs Outside anymore after something like this? Touch grass indeed. Sure, #embed is a nice feature, but this self-indulgent writing style I can’t stand.
- morelisp 4y ago
  Maidenless commenter.
timhh 4y ago
Ha I suggested this on the C++ proposals mailing list 7 years ago: https://groups.google.com/a/isocpp.org/g/std-proposals/c/b6ncBojU8wI/m/blx1GlLqUAUJ https://groups.google.com/a/isocpp.org/g/std-proposals/c/b6n... Enjoy the naysayers if you like! I'm glad someone spent the time and effort to push past them. Bit too late for me - I have moved on to Rust which had support for this from version 1.0.0. > There's also the standard *nix/BSD utility "xxd". > Seems like the niche is filled. Or, at least, if you want to claim that > (A) XPM > (B) incbin > (C) "xxd -i" > (D) various ad-hoc scripts given in http://stackoverflow.com/questions/8707183/script-tool-to-convert-file-to-c-c-source-code-array http://stackoverflow.com/questions/8707183/script-tool-to-co... >...do NOT completely fill this evolutionary niche > This ultimately would encourage a weird sort of resource management philosophy that I think might be damaging in the long run. > Speaking from experience, it is a tremendously bad idea to bake any resource into a binary. > I'll point out that this is a non-issue for Qt applications that can simply use Qt's resources for this sort of business. (Though credit to Matthew Woehlke, he did point out a solution which is basically identical to #embed) > I find this useless specially in embedded environments since there should be some processing of the binary data anyway, either before building the application In fairness there was a decent amount of support. But given the insane amount of negativity around an obviously useful feature I gave up. I wonder if there was a similar response to the proposal to include `string::starts_with()`...
- boywitharupee 4y ago
  what is the Rust equivalent for #embed?
  - zRedShift 4y ago
    https://doc.rust-lang.org/std/macro.include_bytes.html https://doc.rust-lang.org/std/macro.include_bytes.html
  - guipsp 4y ago
    https://doc.rust-lang.org/std/macro.include_bytes.html https://doc.rust-lang.org/std/macro.include_bytes.html
sylware 4y ago
There is way too much in C already. The first commandment of C is: 'writing a naive C compiler should be "reasonable" for a small team or even one individual'. That's getting harder and harder, longer and longer. I did move from C being "the best compromise" to "the less worse compromise". I wish we had a "C-like" language, which would kind of be a high-level assembler which: has no integer promotion or implicit casts, has compile-time/runtime casts (without the horrible c++ syntax), has sized primitive types (u64/s64,f32/f64,etc) at its core, has sized literals (42b,12w,123dw,2qw,etc), has no typedef/generic/volatile/restrict/etc well that sort of horrible things, has compile-time and runtime "const"s, and I am forgetting a lot. From the main issues: the kernel gcc C dialect (roughly speaking, each linux release uses more gcc extensions). Aggressive optimizations can break some code (while programing some hardware for instance). Maybe I should write assembly, expect RISC-V to be a success, and forget about all of this.
- agluszak 4y ago
  > I wish we had a "C-like" language, which would... How about https://ziglang.org/ https://ziglang.org/ ?
  - deleted 4y ago
    [deleted]
- AlexanderDhoore 4y ago
  GCC or Clang with all warnings turned on will give you almost what you want. -Wconversion -Wdouble-promotion and 100s of others. A good way to learn about warning flags (apart from reading the docs) is Clang -Weverything, which will give you many, many warnings.
- quelsolaar 4y ago
  I agree (with a lot of caveats), but a key value of C is that we do not break peoples code and that means that we cant easily remove things. If we do, we create a lot of problems. This makes it very difficult to keep the language as easy to implement as we would like. As a member of the WG14, I intend to propose that we do make this our prime priority going forward.
yakubin 4y ago
> told me this form was non-ideal and it was worth voting against (and that they’d want the pure, beautiful C++ version only[1]) I heard about #embed, but I didn't hear about std::embed before. After looking at the proposal, to me it does look a lot better than #embed, because reading binary data and converting it to text, only to then convert it to binary again seems needlessly complex and wasteful. I also don't like that it extends the preprocessor, when IMHO the preprocessor should at worst be left as is, and at best be slowly deprecated in favour of features which compose well with C proper. Going beyond the gut reaction and moving on to hard data, as you can expect from this design, std::embed of course is faster during compilation than #embed for bigger files (comparable for moderately-sized files, and a bit slower for tiny files). I'm not a huge fan of C++, but the fact that C++ removed trigraphs in C++17 and that it's generally adding features replacing the preprocessor scores a point with me. [1]: <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1040r6.html https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p10...>
- twoodfin 4y ago
  I don’t think the implication is that the C compiler must encode the binary file as a comma-separated integer list and then re-parse it, only act as if it did so.
  - yakubin 4y ago
    How would that work? It would need to depend on the grammar of surrounding C code. This directive isn't limited to variable initialisers. You can use it anywhere. So e.g. you can use it inside structure declaration, or between "int main()" and "{". etc. etc. Those will generate errors in subsequent phases, but during preprocessing the compiler doesn't know about it. Then there is also just that: int main () { return #embed "file.bin" ; } There are plenty of cases, where it will all behave differently. And if you're going to pretend even more that the preprocessor understands C syntax, then why not just give this job to compiler proper, which actually understands it?
    - deleted 4y ago
      [deleted]
owalt 4y ago
Honestly I'm usually very wary of additions to C, as one of its greatest strengths (to me) is how rather straightforward it is as a language in terms of conceptual simplicity. There just aren't that many big concepts to understand in the language. (On the other hand there's _many_ footguns but that's another issue.) That said, to me this seems like a great addition to the language. It's very single-purpose in its usage (so it doesn't seem to add much conceptual complexity to the language) and it replaces something genuinely painful (arcane linker hacks). I'm very much looking forward to using this as I often make single-executable programs in C. The only thing that's unfortunate is I'm sure it'll take decades before proprietary embedded toolchains add support for this.
- pjmlp 4y ago
  C23 and C26 are basically heading into C++ without classes.
nuc1e0n 4y ago
This is a cool feature and I'll likely be using it in the years to come. However, the posix standard command xxd and its -i option can achieve this capability portably today. It will be useful to achieve it directly in the preprocessor however. I wonder how quickly can it be added to cpp?
- ErikCorry 4y ago
  I'm pretty sure xxd is not part of POSIX https://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html https://pubs.opengroup.org/onlinepubs/9699919799/idx/utiliti...
mgaunard 4y ago
It takes literally 5 minutes to write a python script that does this. It took a long time to get this adopted because people are most likely busy with things that cannot be already solved trivially.
- jonathrg 4y ago
  I think it's nice that this will soon be possible to do without adding Python as a dependency in your build system
- jjnoakes 4y ago
  > trivially The article covers quite a few reasons why the way things are done without #embed are not quite as trivial as they seem.
  - mgaunard 4y ago
    I've been doing it for 20 years without any single issue, on fairly large files. This proposal doesn't even allow to compress or encrypt the data.
    - jjnoakes 4y ago
      And yet the issues in the article are real and non-trivial. Perhaps you just never hit them, or you have a high tolerance for non-trivial solutions.
buserror 4y ago
Haven't the people making the standards other things to do, like, integrating useful features instead of duplicating incbin.h [0] years after that feature worked? https://github.com/graphitemaster/incbin/blob/main/incbin.h https://github.com/graphitemaster/incbin/blob/main/incbin.h
- Jasper_ 4y ago
  That doesn't work on MSVC without an external source-generating tool.
- deleted 4y ago
  [deleted]
thraw_oway 4y ago
Personally I feel the C committee should've disbanded after the first standard (the C++ one, after the 2003 technical corrigendum). I didn't mind C99 much, but it looks like C(++)reeping featuritis is a nasty habit. These gratuitous standards prompt newbies to use the new features (it's "modern") and puzzled veterans to keep up and reinternalize understanding of new variants of languages they've been using for decades. There's no real improvement, just churn. Possibly it's one of the instruments of ageism. More incompatibility with existing software and existing programmers.
- josephcsible 4y ago
  Having to learn new things throughout your career isn't ageism.
  - thraw_oway 4y ago
    The question is: which other things, and why? One problem of this new millennium is that the field has developed a tendency to do old things with new languages instead of doing new things with old languages. Reinventing another polygonal wheel approximation while constantly tweaking the theory used serves to segregate the experienced (who may have trouble accepting such tweaks or their necessity - they already know about the existence of wheels anyway) from the newbies (who have no previous intuitions and no taste for legitimate and spurious novelty). Newbies are cheap, and new ideas are hard. Let's do some mental rent seeking.
csmpltn 4y ago
C89 is where C should've stayed at. If you need to convert a file to a buffer and stick that somewhere in your translation unit, use a build system. Don't fuck with C.
- macintux 4y ago
  Did you read the snail mail letter from someone who does just that?
  - csmpltn 4y ago
    > "Did you read the snail mail letter from someone who does just that?" I did. The author struggled embedding files into their executables with makefiles. We don't know anything else beyond that. So what? People also struggle with memory management in C, an arguably much more difficult and widespread problem. Should we introduce a garbage collector into the C spec? How about we just pull in libsodium into the C standard library because people struggle with getting cryptography right? OP mentions #embed was a multi-year long uphill battle, with a lot of convincing needed at every turn. That in itself is enough proof that people aren't in clear agreement over there being a single "right" solution. Hence, leave this task to bespoke build systems and be done with it. Let different build systems offer different solutions. Allow for different syntaxes, etc. Leave the core language lean.
- GuB-42 4y ago
  Nothing stops you from sticking to C89 if that's you want. Many projects do, and the -std=c89 option will not disappear anytime soon.
cowtools 4y ago
Interesting. I look forward to this. What I've been doing now to embed a source.png file is something like this, where I generate source code from a file's data: in embed_dump.cpp: #include <fstream> #include <iostream> int main(){ std::ifstream f; f.open("./source.png"); std::cout << "//automatically generated by embed_dump from project files:" << std::endl << "const char embedded_tex[] = {"; char a; while(f.good()){ f.read(&a,1); std::cout << int(a) << ","; } std::cout << "};" << std::endl; f.close(); } Then I set up my makefile like this (main_stuff.cpp #includes embedded_files.h): main_stuff: main_stuff.cpp embedded_files.h c++ main_stuff.cpp embeded_files.h: embed_dump source.png ./embed_dump > embeded_files.h embed_dump: embed_dump.cpp c++ embed_dump.cpp -o embed_dump
iasay 4y ago
This is wrong. It belongs in the linker and you can pull the symbol it creates in with extern. I’ve been doing this for about 25 years.
- dafelst 4y ago
  ...and now your solution is non-portable and as a cross-platform developer you need to implement N different build scripts. This is far more elegant.
  - iasay 4y ago
    No C toolchain is portable! If that's a problem, use Go or another higher level language.
    - xigoi 4y ago
      Isn't one of the supposed advantages of C that it works on many platforms?
      - iasay 4y ago
        The last 40 odd years have proven that it does work but what you write isn’t portable.
        iainmerrick 4y ago
        That’s not the case, if you take a bit of care. Look at STB for example (https://github.com/nothings/stb https://github.com/nothings/stb) -- I’ve successfully used STB functions on a bunch of different platforms. In both C and C++, even!
        iasay 4y ago
        Ah yes the “if you take a bit of care” argument. Welcome to the 1% of C programmers club :)
david2ndaccount 4y ago
This is a really, really good feature and I am so glad it is finally getting standardized. C23 is shaping up to be a very good revision to the C standard. I’m hoping the proposal to allow redeclaration of identical structs gets in as well as you would finally be able to write code using common types without having to coordinate which would allow interoperability between independently written libraries.
zonovar 4y ago
5 years ago I wrote a small python script [1] to help me solve "the same problem". It reads files in a folder and generates an header file containing the files' data and filenames. Is very simple and was to helping me on a job. It has limitations, don't be too hard on me :) [1] https://github.com/daxliar/pyker https://github.com/daxliar/pyker
juunpp 4y ago
Looks great. I've been writing cmake hacks to include assets in executables for too long.
zzo38computer 4y ago
It is something that I had wanted in C too, for a while, so I am glad that they added this #embed command.
einpoklum 4y ago
Everlasting glory to JeanHeyd Meneide, @thephantomderp, for getting this feature into C. I am wondering, though - where does this stand in C++?
kevin_thibedeau 4y ago
> The directive is well-specified, currently, in all cases to generate a comma-delimited list of integers. While a noble act, this is nearly as inefficient as using a code generator tool to convert binary data into intermediate C source. Other routes to embed binary data don't force the compiler to churn through text bloat. It would be much better if a new keyword were introduced that could let the backend fill in the data at link time.
- elcritch 4y ago
  You should read or re-read the article and references. There are multiple benchmarks showing this not to be the case. Actually half the article is a (well deserved) rant about how wrong compiler devs were in thinking that parsing intermediate C sources could ever match the new directive. Compiler internal representation of an array of integers also doesn't require a big pile of integer ast's. According to the benchmarking data this extension is even 2x faster than using the linker `objcopy` to insert a binary at link time as you suggest.
  - deleted 4y ago
    [deleted]
deleted 4y ago
[deleted]
iainmerrick 4y ago
I’m really amazed at how divisive this one is, and the number of comments here questioning what seems to me a really useful and well-thought-out feature, something I’d have loved to have used many many times over the years. I guess the heated arguments here help me understand how it could have taken so long to get this standardised, though, so that’s something! Congratulations and thank you to the OP for doing this, and thanks also for this really interesting (if depressing) view of the process.
pabs3 4y ago
What situations is this useful for?
- ajxs 4y ago
  One particular scenario that people have highlighted is developing for an embedded system that doesn't have any storage except flash memory, and no filesystem. In this kind of system, embedding static resources in the executable is the only reasonable option you have.
AlbertoGP 4y ago
If you want to start playing with this now, my C preprocessor Cedro (https://news.ycombinator.com/item?id=28166125 https://news.ycombinator.com/item?id=28166125, https://news.ycombinator.com/item?id=31193138 https://news.ycombinator.com/item?id=31193138) will insert the byte literals for the compiler: https://sentido-labs.com/en/library/cedro/202106171400/#binary-include https://sentido-labs.com/en/library/cedro/202106171400/#bina... That’s the same that `xxd` does, for instance, and it works with C89/C99/C11. The advantage is that since it uses the same syntax as C23, it is easier to switch to using the compiler: just remove the Cedro pragma `#pragma Cedro 1.0 #embed` and it will compile as C23. The source code (Apache 2.0) and GitHub link are at: https://sentido-labs.com/en/library/ https://sentido-labs.com/en/library/

483 points by aw1621107 4y ago

Other stuff: https://twitter.com/rcs/status/1550526425211584512 https://twitter.com/rcs/status/1550526425211584512 nullptr! auto! constexpr!

phkahler 4y ago
Not sure about the value of nullptr! Also not sure about auto! In C.
- camel-cdr 4y ago
  nullptr since we have type detection now, and NULL mustn't be a pointer. auto, because otherwise everybody would create their own hacky auto using the new typeof.

trinovantes 4y ago

Off the top of my head, I think there's some niche use in embedding shaders so that they don't need to be stored as strings (no IDE support) or read at runtime (slower performance).

matthews2 4y ago
Nice for binary shaders too, e.g. SPIR-V bytecode generated by glslc.
spicyjpeg 4y ago
There are a lot of use cases for baking binary data directly into the program, especially in embedded applications. For instance, if you are writing a bootloader for a device that has some kind of display you might want to include a splash screen, or a font to be able to show error messages before a filesystem or an external storage medium is initialized. Similarly, on a microcontroller with no external storage at all you need to embed all your assets into the binary; the current way to do that is to either use whatever non-standard tools the manufacturer's proprietary toolchain provides, or to use xxd to (inefficiently) generate a huge C source file from the contents of the binary file. Both require custom build steps and neither is ideal.
phoboslab 4y ago
You can get some IDE support with a simple preprocessor macro[1]. It's a crutch, but at least you don't need to stuff the shader into multiple "strings" or have string continuations (\) at the end of every line. Plus you get some syntax highlighting from the embedding language. I.e. the shader is highlighted as C code, which for the most part seems to be close enough. [1] https://github.com/phoboslab/pl_mpeg/blob/master/pl_mpeg_player.c#L102-L131 https://github.com/phoboslab/pl_mpeg/blob/master/pl_mpeg_pla...
- trinovantes 4y ago
  That's pretty clever Thanks, I'll remember to use that in my future shaders
avianes 4y ago
Another typical use is embedding a public-key in an application or firmware.

junon 4y ago

This will simplify a lot of build pipelines for sure. One thing that isn't clear from skimming the article, how do you refer to the embedded data again?

jsnell 4y ago
> The directive is well-specified, currently, in all cases to generate a comma-delimited list of integers I.e. you most likely use it to initialize a static variable, and then refer to that variable.
- junon 4y ago
  Ah so this basically? static const char d[]={ #embed ... }; EDIT: ah it was showing up more like a comment which made it hard to spot.
  - jonathrg 4y ago
    Yes. There is an example in the article

throwaway38583 4y ago

Congratulations to the author. Things like this are why I hope Carbon exists. Evolving c++ seems like a dumpster fire, despite whatever compelling arguments about comparability you are going to drop on me.

ActorNightly 4y ago
The issue is that a lot of people just think about languages in a wrong way, which is the whole reason for pointless things like C++ expansions, Carbon, Rust, and stuff like this. One of the fundamental ideas that people run with in language creation/expansion is "programmer is stupid and/or make mistakes" -> "lets add language features that intercept and control his stupidity/mistakes". And there is a very valid reason for this - it allows programmers of lesser skill and knowledge base to pick up codebases and develop safe software, which has economic advantages in being able to higher less experienced devs to write software at lower salary points and spend next to no time fixing segfault issues due to complex memory management. The whole reason Java got so popular over C++ was because of its GC - both C++ and Java supported fairly strong typing with classes, but C++ still had a lot of semantics around memory management that had to be taken care of, whereas with Java you just simply don't do anything. However, people are applying this idea towards lower level languages, because they want the high performance of a compiled language with a whole bunch of features that make writing code as mistake free as possible. And my challenge to that is this - why not spend the time making just smarter compilers/tooling? Think about a hypothetical case where Rust gets all the features added to it that people want, and is widely used as the main language over all others. Looking at all the code bases, there will be a lot of common use patterns, a lot of the safety code duplicated over and over in predictable patterns, e.t.c. And you will see these common things added to Rust. Just like with Java a lot of the predictable use patterns got abstracted into widely used libraries like Lombok, Spring, e.t.c, where you don't have to worry about correctness in lieu of using a library. And you essentially will start to move towards more and more stuff being handled for you automagically, which is all part of the compiler/toolchain In the same way, #embed can be solved by smart compiler. Have a static string that opens a file, and read contents into a buffer that doesn't change? Auto include that file in a binary if you want to target performance rather than executable size. No need for special instruction, just be smart about how you handle an open call, and leave the fine tuning of this to specific compiler options. And from an economic perspective of ease of use from above, you would have a language like Python which is super easy to pick up and program in, except instead of the interpreter, you would have a compiler that will spit out binaries. Python is already widely adopted primarily of how easy it is to set up and use. Now imagine if you had the option to run a super smart compiler that highlights any potential issues that come with dynamic typing because it understands what you are trying to do, fixes any that it can, and once everything is addressed, it spits out an optimized memory safe executable. With Rust, you code, compile, see you made a mistake somewhere with a reference, fix it, repeat. With this, you would code, compile, fix the mistake somewhere that the compiler warns you about, repeat. No difference. Focusing on the toolchain also lets you think about integrating features from languages like Coq with provability, where you can focus not only on correctness processing/memory wise, but also "is the output actually correct". I.e, any piece of code for all given input can be specified to have guaranteed bounded output set, which you can integrate into IDE tools to provide you real time feedback on this for you to design the code in a way that avoids things like URL parsing mistakes, which all the languages safety features of Rust won't catch. As for C, you leave it a version that has a stable, robust ABI, and then anything that you need to support will be delegated to custom tools. That way, in the future where compute will likely be full of specialized ML chips, instead of worrying about writing the frontend to support every feature, you quickly get a notional tool chain made and are able to run existing C code.

aaaaaaaaaaab 4y ago

C++ keeps kicking ass! Feel sorry for crab people.

jonathrg 4y ago
That's not really the right conclusion to draw from this article
orf 4y ago
The article definitely isn’t a glowing praise of C/C++. In fact, including this simple, useful feature that rust has had for a decade now has taken an immense amount of effort and received so much pushback from various parties, in part due to the strangled mess of various compiler limitations and in part because of design-by-committee stupidity. C/C++ seems to be kicking it’s own ass.
- gilnaa 4y ago
  Not to mention that it didn't even get into C++
- msla 4y ago
  The article doesn't even mention C++/Java.
  - tialaramex 4y ago
    Second paragraph of the title article: > Surprisingly, despite this journey starting with C++ and WG21, the C Committee is the one that managed to get there first Later it mentions presenting their first formal attempt at this to Belfast 2019, that's a C++ meeting, it's too late for this to go into C++ 20 at that point, but it easily could have been in C++ 23 (it is not).
agluszak 4y ago
"crab people" means Rust people?
- 4y ago

Cloudef 4y ago

I've always used xxd -i for embedding, doesn't have the mentioned problems and works everywhere, as it simply outputs a header file with byte array.

jsnell 4y ago
The article spends a fair bit of time discussing the build speed and memory use problems with that approach. Like, the benchmark results [0] linked to from this post literally have xxd as one of the rows. It's not a viable option for embedding megabytes of data. [0] https://thephd.dev/embed-the-details#results https://thephd.dev/embed-the-details#results
- tempodox 4y ago
  And even if the data is small enough, not every C programmer uses Unix or knows their way around it.
mikepurvis 4y ago
But you have to have build system stuff for that and it's obviously non portable.
- Cloudef 4y ago
  True, i dont personally ever have problem with this because i always compile from unix system anyways. (Even for windows)
pdw 4y ago
Well, congratulations, you now have a build dependency on Vim. (xxd is not a standard tool, it ships with Vim.) It's also only suitable for tiny files: compile time and RAM requirements will blow up once you go beyond a couple of megabytes.
- Cloudef 4y ago
  Yeah, these are reasonable arguments against it

eps 4y ago

Re: #embed </dev/urandom> Just a random thought, but I'd expect a compiler to do exactly what's described if I tell it: static char foo[123] = { #embed </dev/urandom> }; This would address the most common case with infinity files, and then just let the compiler error out if the array size is not specified.

tazjin 4y ago
Reproducible builds crowd wailing in agony
- huhtenberg 4y ago
  The context is that of infinite files, not of the urandom specifically. Give the linked post a read for details.
- astrange 4y ago
  They just need a reproducible urandom.
  - elcritch 4y ago
    Looks like you forgot to add your sponsor disclaimer "message sponsored by NSA". ;)
    - astrange 4y ago
      Well, just don’t do networking in the VM your builds run in and you’re good, maybe.
      - tazjin 4y ago
        Remove /dev/urandom and replace it with /dev/zero in your sandbox ;)
kibwen 4y ago
I would expect that to produce an error, though. If I had a regular file that was not infinite in size, and I specified the wrong length for the array, I would find it more useful to have the compiler inform me as to the discrepancy rather than truncate my file.

ascar 4y ago

This is a cool feature, but the author doesn't do himself any favors with his style of writing that greatly overestimates the importance of his own feature in the great scheme of things. Remarks like "an extension that should’ve existed 40-50 years ago" make me think, if we should've really bothered all compiler vendors with implementing this 40-50 years ago. After all, you can already a) directly put your binary data in the source file like shown after the preprocessor step and b) read a file at runtime. I'm not saying this isn't useful, but it's a rather niché performance improvement than a core language feature.

someweirdperson 4y ago

How to read unsigned data? Is there a stadardized parameter, or does this require a vendor extension?

jonathrg 4y ago
You just make your array type uint8_t or whatever you need as long as it supports integer literals. See section 4 in https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#intro https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#i...
- someweirdperson 4y ago
  Scary, it's as if the preprocessor has become type-aware. I guess I better don't imagine the result of the preprocessing to look similar to and following the same rules as something I would have written by hand. This might make manual inspection of the preprocessed file a bit painful.
  - elcritch 4y ago
    Its not really a pre-processor stage. Probably better to think of it more like a pointer cast to some binary blob. Though it'd be interesting to see what `gcc -E` would produce.

tempodox 4y ago

> Even among people who control all the cards, they are in many respects fundamentally incapable of imagining a better world or seizing on that opportunity to try and create one, let alone doing so in a timely fashion. That does sound soul-crushing. Congrats on this achievement!

quelsolaar 4y ago
This is simply wrong. We (the ISO wg14) don't hold the cards, compilers are free to implement what ever they want, users are free to use what ever tools or languages they want. We exist only as long as we are trusted to be good stewards, and only go forward with the consensus of the wider community.
- unreal37 4y ago
  You're both right. It's amazing that you and the ISO team are good stewards of the C standard. Thank you for being part of that. And it can also be true that it was "hell" and "hardly worth it" for the OP to get a new feature added to the language. I believe it was a miserable experience that has him questioning how he spends his time. Both can be true. Thank you for your efforts. And thank the OP for his efforts too.
- morelisp 4y ago
  > > Even among people who control all the cards, they are in many respects fundamentally incapable of imagining a better world or seizing on that opportunity to try and create one, let alone doing so in a timely fashion. > This is simply wrong. We (the ISO wg14) don't hold the cards, compilers are free to implement what ever they want, users are free to use what ever tools or languages they want. This is an incredibly oblivious realization of JeanHeyd's point.
- gtirloni 4y ago
  > (the ISO wg14) don't hold the cards That "standard" card seem to be a pretty huge one though.
  - 4y ago

kibwen 4y ago

This serves the same use as Rust's `include_bytes!` macro, right? Presumably most people just use this feature as a way to avoid having to stuff binary data into a massive array literal, but in our case it's essential because we're actually using it to stuff binaries from earlier in our build step into a binary built later in the build step. Not something you often need, but very handy when you do.

kzrdude 4y ago
for both Rust and C, these features "just" make something you could otherwise do with the build system and generated code easier, I think.
- masklinn 4y ago
  As the article quotes, in C the lack of standardisation makes this tricky when you want to support more than one compiler, or even when you want to support just one compiler (cf email about the hacks to make it work on GCC with PIE).
tialaramex 4y ago
This has different affordances than std::include_bytes! but I agree that if you were writing Rust and had this problem you'd reach for std::include_bytes! and probably not instead think "We should have an equivalent of #embed". include_bytes! gives you a &'static [u8; N] which for non-Rust programmers means we're making a fixed size array (the size of your file) full of unsigned 8-bit integers (ie bytes) which lives for the life of the program, and we get an immutable reference to it. Rust's arrays know how big they are (so we can ask, now or later) but cannot grow. #embed gets you a bunch of integers. The as-if rule means your compiler is likely to notice if what you're actually doing is putting those integers into an array of unsigned 8-bit integers and just stick all the file bytes in the array, short cutting what you wrote, but you could reasonably do other things, especially with smaller files.

oefrha 4y ago

> vendor extensions ... were now a legal part of the syntax. If your implementation does not support a thing, it can issue a diagnostic for the parameters it does not understand. This was great stuff. I can’t be the only one who thinks magic comment is already an ugly escape hatch, adding a mini DSL to it that can mean anything to anyone just makes it ten times worse. It’s neither beautiful nor great. > do extensions on #embed to support different file modes, potentially reading from the network (with a timeout), and other shenanigans. (Emphasis mine.) My god.

greatgib 4y ago
I guess you don't know C. "#" is not a symbol for a comment line but the one for a pre-processor directive. Like #include stdlib.h In c/c++ you use // and /* */ for comments.
- oefrha 4y ago
  I’ve known C for close to two decades, thank you. I’m using the not at all well defined term “magic comment” to loosely refer to everything that’s not strictly speaking code but has special meaning, which include pre-processor directives. cpp is definitely a well-hated part of C.
  - Ensorceled 4y ago
    > I’ve known C for close to two decades, thank you. I’m using the not at all well defined term “magic comment” Please forgive those of us who've been using C since the 80's, or earlier, from assuming you don't know C when you invent your own terminology for preprocessor directives.
    - oefrha 4y ago
      This is not a preprocessor directive though, from reading the post I don’t think cpp is expanding the #embed into an array initializer, otherwise there’s no performance benefit at all.

orbifold 4y ago

Can’t you just add binary data into a custom section of your ELF executable?

nicoburns 4y ago
The article address this directly. If you're only targeting one platform then this is reasonably easy (albeit still not as easy as #embed), but if you need to be portable then it becomes a nightmare of multiple proprietary methods.
rleigh 4y ago
Yes, but it's linker-specific and non-portable. It can also come with some annoying limitations, like having to separately provide the data size of each symbol. In some cases this might be introspectable, but again comes at the expense of portability. ELF-based variants of the IAR toolchain, for example, provide a means of directly embedding a file as an ELF symbol, but without the size information being directly accessible. GNU ld and LLVM lld do not provide any embedding functionality at all (as far as I can see). You would have to generate a custom object file with some generated C or ASM encoding the binary content. MSVC link.exe doesn't support this either, but there is the "resource compiler" to embed binary bits and link them in so they can be retrieved at runtime. Having a universal and portable mechanism which works everywhere will be a great benefit. I'll be using it for compiled or text shaders, compiled or text lua scripts, small graphics, fonts and all sorts.
- acka 4y ago
  This article[1] shows how you can use GCC toolchain along with objcopy to create an object file from a binary blob, link it, and use the data within in your own code. [1] https://balau82.wordpress.com/2012/02/19/linking-a-binary-blob-with-gcc/ https://balau82.wordpress.com/2012/02/19/linking-a-binary-bl...
  - spc476 4y ago
    Great for Linux. Less so for Mac OS-X and Solaris (all three of which I use at work).

moffkalast 4y ago

This reminds me, I'd argue that the explosion of JS frameworks can be mainly blamed on one thing: the lack of an <include src="somemodule.html"> tag. If you have that you basically have 80% of vue.js already natively supported. No clue why this was never added in any fashion. Change my mind.

polskibus 4y ago
Is <script type="module" /> not sufficient for your needs? If not then what is missing?
- nkozyra 4y ago
  Seems to be arguing for modular layout/templating, which is what virtual includes did (the cgi in the example would hypothetically output html)
lkschubert8 4y ago
Wouldn't the include still need some templating functionality? Or are people using vue that heavily for just importing static html?
- ear7h 4y ago
  Not the parent comment, but my personal use case is for rendering a selectable list. The server side would render a static list with fragment links (ex. `#item-10`) and include elements with corresponding IDs, and a `:target` css rule to unhide the element. This would hopefully be paired with lazy loading the include elements. edit: My goal is to avoid reloading the page for each selection and rendering all items eagerly. JS frameworks are the only ones that really allow this behavior.
TheAceOfHearts 4y ago
HTML imports were part of the original concept of Web Components, and I think they were supported in Chrome. If you look up examples of things built with Polymer 1.x, it was used extensively. It was actually pretty neat, because you could have an HTML file with a template, style, and script section. Safari rejected the proposal, so it had to get dropped. But ESM makes it a bit redundant anyway. The end-goal is to allow you to import any kind of asset, not just JS. There have been demos and examples of tools supporting this going back over half a decade at this point.

quelsolaar 4y ago

I represent Sweden in the ISO WG14, and I voted for the inclusion of Embed in to C23. Its a good feature. But its not a necessary feature and I think JeanHeyd is wrong in his criticism of the pace of wg14 work. I have found everyone in wg14 to be very hardworking and serious about their work. Cs main strengthen is its portability and simplicity. Therefore we should be very conservative, and not add anything quickly. There are plenty of languages to choose form if you want a "modern" language with lots of conveniences. If you want a truly portable language there is really only C. And when I say truly, I mean for platforms without file systems, or operating systems or where bytes aren't 8 bits, that doesn't use ASCI or Unicode, where NULL isn't on address 0 and so on. We are the stewards of this, and the work we put in, while large, is tiny compared to the impact we have. Any change we makes, needs to be addressed by every compiler maintainer. There are millions of lines of code that depend on every part of the standard. A 1% performance loss is millions of tons of CO2 released, and billions in added hardware and energy costs. In this privileged position, we have to be very mindful of the concerns of our users, and take the time too look at every corner case in detail before adding any new features. If we add something, then people will depend on its behavior, no matter how bad, and we therefor will have great difficulty in fixing it in the future without breaking our users work, so we have to get it right the first time.

pif 4y ago
Thank you for your post! Thank you especially for reminding everybody that programming is much more than web programming and information systems.
- quelsolaar 4y ago
  Thank you, Its also worth remembering that a lot of higher level languages have runtimes / VMs are implemented in C. Web applications rely heavily on databases, java script VM, network-stacks, system calls and operating system features, all of which are impemented in C. If you are a software developer and want to do something about climate change, consider becomming a compiler engineer. If you manage to get a couple of tenths of a percent performance increase in one of the big compilers during your career, you will have materially impacted global warming. Compiler engineers are the unsung heroes of software engineering.

jesprenj 4y ago

What about creating object files from raw binary files and then linking against them? That's what I (and of course many others) do for linking textures and shaders into the program. It's a bit ugly though that with this approach you can't generate custom symbol names, at least with the GNU linker. This #embed feature might be a nice alternative for small files. Well for large files you usually don't even want to store them inside the binary, so the compilation overhead might be miniscule, since the files are, by intention, small. When I read the introduction of the article - about allowing us to cram anything we want into the binary - I was hoping to see a standard way to disable optimizations (When the compiler deletes your code and you don't even notice).

dark-star 4y ago
One reason against this is mentioned in the letter that is quoted in the article
jonathrg 4y ago
It depends on your definition of small files. A few hundred kB to a few megabytes will make compilation speed and memory usage explode if you embed it as text, see section 3.2 in https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#intro https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#i...
mastax 4y ago
You reminded me of Bethesda Softworks games, which always seem to have 1GB+ executables for some reason. I hope it isn't all code. Maybe they embed the most important assets that will always need to be loaded.
- GuB-42 4y ago
  My guess is that the files are not truly embedded as it would require to load the entire file in memory before running the application, which seems wasteful. More likely, the actual executable is only a small part of the file which accesses the rest of the file as an archive, like a self-extracting zip. There may also be some DRM trickery going on.

pwdisswordfish9 4y ago

> “Touch grass”, some people liked to tell me. “Go outside”, they said (like I would in the middle of Yet Another Gotdang COVID-19 Spike). My dude, I’ve literally gotten company snail mail, and it wasn’t a legal notice or some shenanigans like that! Holy cow, real paper in a real envelope, shipped through German Speed Mail!! This letter alone probably increases my Boomer Cred™ by at least 50; who needs Outside anymore after something like this? Touch grass indeed. Sure, #embed is a nice feature, but this self-indulgent writing style I can’t stand.

morelisp 4y ago
Maidenless commenter.

timhh 4y ago

Ha I suggested this on the C++ proposals mailing list 7 years ago: https://groups.google.com/a/isocpp.org/g/std-proposals/c/b6ncBojU8wI/m/blx1GlLqUAUJ https://groups.google.com/a/isocpp.org/g/std-proposals/c/b6n... Enjoy the naysayers if you like! I'm glad someone spent the time and effort to push past them. Bit too late for me - I have moved on to Rust which had support for this from version 1.0.0. > There's also the standard *nix/BSD utility "xxd". > Seems like the niche is filled. Or, at least, if you want to claim that > (A) XPM > (B) incbin > (C) "xxd -i" > (D) various ad-hoc scripts given in http://stackoverflow.com/questions/8707183/script-tool-to-convert-file-to-c-c-source-code-array http://stackoverflow.com/questions/8707183/script-tool-to-co... >...do NOT completely fill this evolutionary niche > This ultimately would encourage a weird sort of resource management philosophy that I think might be damaging in the long run. > Speaking from experience, it is a tremendously bad idea to bake any resource into a binary. > I'll point out that this is a non-issue for Qt applications that can simply use Qt's resources for this sort of business. (Though credit to Matthew Woehlke, he did point out a solution which is basically identical to #embed) > I find this useless specially in embedded environments since there should be some processing of the binary data anyway, either before building the application In fairness there was a decent amount of support. But given the insane amount of negativity around an obviously useful feature I gave up. I wonder if there was a similar response to the proposal to include `string::starts_with()`...

boywitharupee 4y ago
what is the Rust equivalent for #embed?
- zRedShift 4y ago
  https://doc.rust-lang.org/std/macro.include_bytes.html https://doc.rust-lang.org/std/macro.include_bytes.html
- guipsp 4y ago
  https://doc.rust-lang.org/std/macro.include_bytes.html https://doc.rust-lang.org/std/macro.include_bytes.html

sylware 4y ago

There is way too much in C already. The first commandment of C is: 'writing a naive C compiler should be "reasonable" for a small team or even one individual'. That's getting harder and harder, longer and longer. I did move from C being "the best compromise" to "the less worse compromise". I wish we had a "C-like" language, which would kind of be a high-level assembler which: has no integer promotion or implicit casts, has compile-time/runtime casts (without the horrible c++ syntax), has sized primitive types (u64/s64,f32/f64,etc) at its core, has sized literals (42b,12w,123dw,2qw,etc), has no typedef/generic/volatile/restrict/etc well that sort of horrible things, has compile-time and runtime "const"s, and I am forgetting a lot. From the main issues: the kernel gcc C dialect (roughly speaking, each linux release uses more gcc extensions). Aggressive optimizations can break some code (while programing some hardware for instance). Maybe I should write assembly, expect RISC-V to be a success, and forget about all of this.

agluszak 4y ago
> I wish we had a "C-like" language, which would... How about https://ziglang.org/ https://ziglang.org/ ?
- deleted 4y ago
  [deleted]
AlexanderDhoore 4y ago
GCC or Clang with all warnings turned on will give you almost what you want. -Wconversion -Wdouble-promotion and 100s of others. A good way to learn about warning flags (apart from reading the docs) is Clang -Weverything, which will give you many, many warnings.
quelsolaar 4y ago
I agree (with a lot of caveats), but a key value of C is that we do not break peoples code and that means that we cant easily remove things. If we do, we create a lot of problems. This makes it very difficult to keep the language as easy to implement as we would like. As a member of the WG14, I intend to propose that we do make this our prime priority going forward.

yakubin 4y ago

> told me this form was non-ideal and it was worth voting against (and that they’d want the pure, beautiful C++ version only[1]) I heard about #embed, but I didn't hear about std::embed before. After looking at the proposal, to me it does look a lot better than #embed, because reading binary data and converting it to text, only to then convert it to binary again seems needlessly complex and wasteful. I also don't like that it extends the preprocessor, when IMHO the preprocessor should at worst be left as is, and at best be slowly deprecated in favour of features which compose well with C proper. Going beyond the gut reaction and moving on to hard data, as you can expect from this design, std::embed of course is faster during compilation than #embed for bigger files (comparable for moderately-sized files, and a bit slower for tiny files). I'm not a huge fan of C++, but the fact that C++ removed trigraphs in C++17 and that it's generally adding features replacing the preprocessor scores a point with me. [1]: <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1040r6.html https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p10...>

twoodfin 4y ago
I don’t think the implication is that the C compiler must encode the binary file as a comma-separated integer list and then re-parse it, only act as if it did so.
- yakubin 4y ago
  How would that work? It would need to depend on the grammar of surrounding C code. This directive isn't limited to variable initialisers. You can use it anywhere. So e.g. you can use it inside structure declaration, or between "int main()" and "{". etc. etc. Those will generate errors in subsequent phases, but during preprocessing the compiler doesn't know about it. Then there is also just that: int main () { return #embed "file.bin" ; } There are plenty of cases, where it will all behave differently. And if you're going to pretend even more that the preprocessor understands C syntax, then why not just give this job to compiler proper, which actually understands it?
  - deleted 4y ago
    [deleted]

owalt 4y ago

Honestly I'm usually very wary of additions to C, as one of its greatest strengths (to me) is how rather straightforward it is as a language in terms of conceptual simplicity. There just aren't that many big concepts to understand in the language. (On the other hand there's _many_ footguns but that's another issue.) That said, to me this seems like a great addition to the language. It's very single-purpose in its usage (so it doesn't seem to add much conceptual complexity to the language) and it replaces something genuinely painful (arcane linker hacks). I'm very much looking forward to using this as I often make single-executable programs in C. The only thing that's unfortunate is I'm sure it'll take decades before proprietary embedded toolchains add support for this.

pjmlp 4y ago
C23 and C26 are basically heading into C++ without classes.

nuc1e0n 4y ago

This is a cool feature and I'll likely be using it in the years to come. However, the posix standard command xxd and its -i option can achieve this capability portably today. It will be useful to achieve it directly in the preprocessor however. I wonder how quickly can it be added to cpp?

ErikCorry 4y ago
I'm pretty sure xxd is not part of POSIX https://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html https://pubs.opengroup.org/onlinepubs/9699919799/idx/utiliti...

mgaunard 4y ago

It takes literally 5 minutes to write a python script that does this. It took a long time to get this adopted because people are most likely busy with things that cannot be already solved trivially.

jonathrg 4y ago
I think it's nice that this will soon be possible to do without adding Python as a dependency in your build system
jjnoakes 4y ago
> trivially The article covers quite a few reasons why the way things are done without #embed are not quite as trivial as they seem.
- mgaunard 4y ago
  I've been doing it for 20 years without any single issue, on fairly large files. This proposal doesn't even allow to compress or encrypt the data.
  - jjnoakes 4y ago
    And yet the issues in the article are real and non-trivial. Perhaps you just never hit them, or you have a high tolerance for non-trivial solutions.

buserror 4y ago

Haven't the people making the standards other things to do, like, integrating useful features instead of duplicating incbin.h [0] years after that feature worked? https://github.com/graphitemaster/incbin/blob/main/incbin.h https://github.com/graphitemaster/incbin/blob/main/incbin.h

Jasper_ 4y ago
That doesn't work on MSVC without an external source-generating tool.
deleted 4y ago
[deleted]

thraw_oway 4y ago

Personally I feel the C committee should've disbanded after the first standard (the C++ one, after the 2003 technical corrigendum). I didn't mind C99 much, but it looks like C(++)reeping featuritis is a nasty habit. These gratuitous standards prompt newbies to use the new features (it's "modern") and puzzled veterans to keep up and reinternalize understanding of new variants of languages they've been using for decades. There's no real improvement, just churn. Possibly it's one of the instruments of ageism. More incompatibility with existing software and existing programmers.

josephcsible 4y ago
Having to learn new things throughout your career isn't ageism.
- thraw_oway 4y ago
  The question is: which other things, and why? One problem of this new millennium is that the field has developed a tendency to do old things with new languages instead of doing new things with old languages. Reinventing another polygonal wheel approximation while constantly tweaking the theory used serves to segregate the experienced (who may have trouble accepting such tweaks or their necessity - they already know about the existence of wheels anyway) from the newbies (who have no previous intuitions and no taste for legitimate and spurious novelty). Newbies are cheap, and new ideas are hard. Let's do some mental rent seeking.

csmpltn 4y ago

C89 is where C should've stayed at. If you need to convert a file to a buffer and stick that somewhere in your translation unit, use a build system. Don't fuck with C.

macintux 4y ago
Did you read the snail mail letter from someone who does just that?
- csmpltn 4y ago
  > "Did you read the snail mail letter from someone who does just that?" I did. The author struggled embedding files into their executables with makefiles. We don't know anything else beyond that. So what? People also struggle with memory management in C, an arguably much more difficult and widespread problem. Should we introduce a garbage collector into the C spec? How about we just pull in libsodium into the C standard library because people struggle with getting cryptography right? OP mentions #embed was a multi-year long uphill battle, with a lot of convincing needed at every turn. That in itself is enough proof that people aren't in clear agreement over there being a single "right" solution. Hence, leave this task to bespoke build systems and be done with it. Let different build systems offer different solutions. Allow for different syntaxes, etc. Leave the core language lean.
GuB-42 4y ago
Nothing stops you from sticking to C89 if that's you want. Many projects do, and the -std=c89 option will not disappear anytime soon.

cowtools 4y ago

Interesting. I look forward to this. What I've been doing now to embed a source.png file is something like this, where I generate source code from a file's data: in embed_dump.cpp: #include <fstream> #include <iostream> int main(){ std::ifstream f; f.open("./source.png"); std::cout << "//automatically generated by embed_dump from project files:" << std::endl << "const char embedded_tex[] = {"; char a; while(f.good()){ f.read(&a,1); std::cout << int(a) << ","; } std::cout << "};" << std::endl; f.close(); } Then I set up my makefile like this (main_stuff.cpp #includes embedded_files.h): main_stuff: main_stuff.cpp embedded_files.h c++ main_stuff.cpp embeded_files.h: embed_dump source.png ./embed_dump > embeded_files.h embed_dump: embed_dump.cpp c++ embed_dump.cpp -o embed_dump

iasay 4y ago

This is wrong. It belongs in the linker and you can pull the symbol it creates in with extern. I’ve been doing this for about 25 years.

dafelst 4y ago
...and now your solution is non-portable and as a cross-platform developer you need to implement N different build scripts. This is far more elegant.
- iasay 4y ago
  No C toolchain is portable! If that's a problem, use Go or another higher level language.
  - xigoi 4y ago
    Isn't one of the supposed advantages of C that it works on many platforms?
    - iasay 4y ago
      The last 40 odd years have proven that it does work but what you write isn’t portable.
      - iainmerrick 4y ago
        That’s not the case, if you take a bit of care. Look at STB for example (https://github.com/nothings/stb https://github.com/nothings/stb) -- I’ve successfully used STB functions on a bunch of different platforms. In both C and C++, even!
        iasay 4y ago
        Ah yes the “if you take a bit of care” argument. Welcome to the 1% of C programmers club :)

david2ndaccount 4y ago

This is a really, really good feature and I am so glad it is finally getting standardized. C23 is shaping up to be a very good revision to the C standard. I’m hoping the proposal to allow redeclaration of identical structs gets in as well as you would finally be able to write code using common types without having to coordinate which would allow interoperability between independently written libraries.

zonovar 4y ago

5 years ago I wrote a small python script [1] to help me solve "the same problem". It reads files in a folder and generates an header file containing the files' data and filenames. Is very simple and was to helping me on a job. It has limitations, don't be too hard on me :) [1] https://github.com/daxliar/pyker https://github.com/daxliar/pyker

juunpp 4y ago

Looks great. I've been writing cmake hacks to include assets in executables for too long.

zzo38computer 4y ago

It is something that I had wanted in C too, for a while, so I am glad that they added this #embed command.

einpoklum 4y ago

Everlasting glory to JeanHeyd Meneide, @thephantomderp, for getting this feature into C. I am wondering, though - where does this stand in C++?

kevin_thibedeau 4y ago

> The directive is well-specified, currently, in all cases to generate a comma-delimited list of integers. While a noble act, this is nearly as inefficient as using a code generator tool to convert binary data into intermediate C source. Other routes to embed binary data don't force the compiler to churn through text bloat. It would be much better if a new keyword were introduced that could let the backend fill in the data at link time.

elcritch 4y ago
You should read or re-read the article and references. There are multiple benchmarks showing this not to be the case. Actually half the article is a (well deserved) rant about how wrong compiler devs were in thinking that parsing intermediate C sources could ever match the new directive. Compiler internal representation of an array of integers also doesn't require a big pile of integer ast's. According to the benchmarking data this extension is even 2x faster than using the linker `objcopy` to insert a binary at link time as you suggest.
- deleted 4y ago
  [deleted]

deleted 4y ago

[deleted]

iainmerrick 4y ago

I’m really amazed at how divisive this one is, and the number of comments here questioning what seems to me a really useful and well-thought-out feature, something I’d have loved to have used many many times over the years. I guess the heated arguments here help me understand how it could have taken so long to get this standardised, though, so that’s something! Congratulations and thank you to the OP for doing this, and thanks also for this really interesting (if depressing) view of the process.

pabs3 4y ago

What situations is this useful for?

ajxs 4y ago
One particular scenario that people have highlighted is developing for an embedded system that doesn't have any storage except flash memory, and no filesystem. In this kind of system, embedding static resources in the executable is the only reasonable option you have.

AlbertoGP 4y ago

If you want to start playing with this now, my C preprocessor Cedro (https://news.ycombinator.com/item?id=28166125 https://news.ycombinator.com/item?id=28166125, https://news.ycombinator.com/item?id=31193138 https://news.ycombinator.com/item?id=31193138) will insert the byte literals for the compiler: https://sentido-labs.com/en/library/cedro/202106171400/#binary-include https://sentido-labs.com/en/library/cedro/202106171400/#bina... That’s the same that `xxd` does, for instance, and it works with C89/C99/C11. The advantage is that since it uses the same syntax as C23, it is easier to switch to using the compiler: just remove the Cedro pragma `#pragma Cedro 1.0 #embed` and it will compile as C23. The source code (Apache 2.0) and GitHub link are at: https://sentido-labs.com/en/library/ https://sentido-labs.com/en/library/