5 ms·

I've always used xxd -i for embedding, doesn't have the mentioned problems and works everywhere, as it simply outputs a header file with byte array.

jsnell 4y ago
The article spends a fair bit of time discussing the build speed and memory use problems with that approach. Like, the benchmark results [0] linked to from this post literally have xxd as one of the rows. It's not a viable option for embedding megabytes of data. [0] https://thephd.dev/embed-the-details#results https://thephd.dev/embed-the-details#results
- tempodox 4y ago
  And even if the data is small enough, not every C programmer uses Unix or knows their way around it.
mikepurvis 4y ago
But you have to have build system stuff for that and it's obviously non portable.
- Cloudef 4y ago
  True, i dont personally ever have problem with this because i always compile from unix system anyways. (Even for windows)
pdw 4y ago
Well, congratulations, you now have a build dependency on Vim. (xxd is not a standard tool, it ships with Vim.) It's also only suitable for tiny files: compile time and RAM requirements will blow up once you go beyond a couple of megabytes.
- Cloudef 4y ago
  Yeah, these are reasonable arguments against it
  - nicoburns 4y ago
    The one in the original article I that it performs badly for large files.
    - yakubin 4y ago
      That's putting it mildly: <https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#intro-metrics-speed https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#i...>
- mariusor 4y ago
  > It's also only suitable for tiny files: compile time and RAM requirements will blow up once you go beyond a couple of megabytes. Do you know what makes it so? Is there a technical argument why the compiler could do better, except maybe for xxd not being specifically optimized for this use case?
  - tialaramex 4y ago
    The compiler has an as-if rule on its side. It's allowed to do whatever it wants so long as the results are as-if it did what the standard says. So even though the standard says this is making a big list of integers like your xxd command, the compiler won't do that, because (as a C compiler) it knows perfectly well it would just parse those integers into bytes again, just like the ones it got out of the binary file. It knows the integers would all be valid (it made them) and fit in a byte (duh) and so it can skip the entire back-and-forth.
    - mariusor 4y ago
      If I understand correctly what you're saying it's not a problem with xxd itself, but with the format of the data, which in xxd's case is a pair made out of an array of bytes and its length, while the compiler is free to include the binary verbatim and just provide access to it as it was an array/length pair. Am I right about it?

jsnell 4y ago

The article spends a fair bit of time discussing the build speed and memory use problems with that approach. Like, the benchmark results [0] linked to from this post literally have xxd as one of the rows. It's not a viable option for embedding megabytes of data. [0] https://thephd.dev/embed-the-details#results https://thephd.dev/embed-the-details#results

tempodox 4y ago
And even if the data is small enough, not every C programmer uses Unix or knows their way around it.

mikepurvis 4y ago

But you have to have build system stuff for that and it's obviously non portable.

Cloudef 4y ago
True, i dont personally ever have problem with this because i always compile from unix system anyways. (Even for windows)

pdw 4y ago

Well, congratulations, you now have a build dependency on Vim. (xxd is not a standard tool, it ships with Vim.) It's also only suitable for tiny files: compile time and RAM requirements will blow up once you go beyond a couple of megabytes.

Cloudef 4y ago
Yeah, these are reasonable arguments against it
- nicoburns 4y ago
  The one in the original article I that it performs badly for large files.
  - yakubin 4y ago
    That's putting it mildly: <https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#intro-metrics-speed https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#i...>
mariusor 4y ago
> It's also only suitable for tiny files: compile time and RAM requirements will blow up once you go beyond a couple of megabytes. Do you know what makes it so? Is there a technical argument why the compiler could do better, except maybe for xxd not being specifically optimized for this use case?
- tialaramex 4y ago
  The compiler has an as-if rule on its side. It's allowed to do whatever it wants so long as the results are as-if it did what the standard says. So even though the standard says this is making a big list of integers like your xxd command, the compiler won't do that, because (as a C compiler) it knows perfectly well it would just parse those integers into bytes again, just like the ones it got out of the binary file. It knows the integers would all be valid (it made them) and fit in a byte (duh) and so it can skip the entire back-and-forth.
  - mariusor 4y ago
    If I understand correctly what you're saying it's not a problem with xxd itself, but with the format of the data, which in xxd's case is a pair made out of an array of bytes and its length, while the compiler is free to include the binary verbatim and just provide access to it as it was an array/length pair. Am I right about it?