5 ms·

> I wouldn't be surprised if "parse and compile 8MB of integer literals" isn't a very well optimized code path in the compiler, because nobody is parsing and co

> I wouldn't be surprised if "parse and compile 8MB of integer literals" isn't a very well optimized code path in the compiler, because nobody is parsing and compiling 8MB of integer literals outside of artificial exercises like this Actually, no. There is no portable "incbin" in C/C++, so one way to bake assets into the .rodata section is to convert a binary file into a huge "const unsigned char data[8192567] = { 0xFF, 0xD8, 0xFF, 0xE0, ... }" string and stash it into a .h or .c file. And yes, people do that often enough that gcc and clang actually had to optimize for this case specifically.

tinus_hn 3y ago
Actually the xbm image format is c files like this with a long array of hexadecimal numbers. https://en.m.wikipedia.org/wiki/X_BitMap https://en.m.wikipedia.org/wiki/X_BitMap
- bluedino 3y ago
  I remember paint programs that could export images as C/Pascal/whatever source code files.
  - stefncb 3y ago
    Gimp still does that, it's rather useful if you want to play around with images without a loader.
ajross 3y ago
> Actually, no. There is no portable "incbin" in C/C++ I kinda hate this kind of argument. I mean, it's true, as .incbin is a GNU assembler directive (FWIW: binutils/llvm objcopy is a better mechanism still for this sort of thing in most contexts, as it doesn't involve source compilation of any kind). But it leads ridiculous design decisions, like "I'm going to write 8MB of source code instead of doing the portability work on my own to turn that data into a linkable symbol". There's a huge gulf in the space between "non-portable" and "impossible". In this case, the problem is trivially solved on every platform ever. Trivial problems should employ trivial solutions, even where they have to involve some per-platform engineering.
- duped 3y ago
  > But it leads ridiculous design decisions, like "I'm going to write 8MB of source code instead of doing the portability work on my own to turn that data into a linkable symbol". people use scripts to do this kind of thing and never think about it again, and it's still more portable than writing a custom build step.
- iainmerrick 3y ago
  But it leads ridiculous design decisions, like "I'm going to write 8MB of source code instead of doing the portability work on my own to turn that data into a linkable symbol". Why is that ridiculous? It strikes me as not necessarily the best, but the most obvious approach, the most portable, and possibly the quickest to implement. Your toolchain might have a special way to import binary blobs, but a) you’ll have to dig through the docs to find it, b) you’ll probably need to solve the problem again when porting to a different platform, and c) who knows if it actually works, or if there are hidden gotchas? Sure, if there’s a known tool or option that does the job, you should go ahead and use it. But in general, writing a little script to generate a bunch of boilerplate code is perfectly workable.
  - ajross 3y ago
    > Your toolchain might have a special way to import binary blobs, but a) you’ll have to dig through the docs to find it This is a corrollary: "I don't want to learn my tools, so I'll learn the language standard instead" is fundamentally exactly the problem I'm talking about. Straight up: C linkage is a 1970's paradigm full of tools that had to run on a PDP/11, and it's vastly simpler than learning C++ or Rust. It's just not "modern" and no one taught it to you, so it looks weird and mysterious. That's the problem!
    - maskros 3y ago
      It's not "I don't want to learn my tools." It's "I don't want to learn and debug _everybody else who may possible want to build this otherwise portable C program_'s tools." When those tools change how they do this unportable thing every couple of years in subtle and incompatible ways, which require #ifdef's to handle the different ways those linked against symbols can be accessed, multiplied by dozens of different platforms, then yes, I'm going to compile an 8MB literal.
tialaramex 3y ago
C23 gets #embed https://thephd.dev/finally-embed-in-c23 https://thephd.dev/finally-embed-in-c23 and in practice later C++ compilers will presumably also just do #embed rather than pretend they aren't the same code anyway. However, even though there's a clear need, as that post explains, the compilers varied between "Bad at this" and "Completely awful at this" which is part of what was so frustrating for JeanHyde over years of getting this through committee. In principle #embed is just shoving the bytes in as comma separated values, but in practice by the "as if" rule in C and C++ the compiler won't do that because it's stupid.
- est31 3y ago
  Since Rust 1.67.0, the compiler has a specific optimized code path for the (already older) include_bytes! macro: https://github.com/rust-lang/rust/pull/103812#issuecomment-1299087888 https://github.com/rust-lang/rust/pull/103812#issuecomment-1...
  - tialaramex 3y ago
    Ha, cute, I think I began examining details of include_bytes! only this year so it's possible I happened to first look at this soon after the big improvements landed in the release I was testing against.
armchairhacker 3y ago
finally. #embed https://thephd.dev/finally-embed-in-c23 https://thephd.dev/finally-embed-in-c23 tl;dr It was such a big issue they actually managed to get it through the C Committee
titzer 3y ago
Also, a reminder that JS engines do regularly have to contend with enormous JSON and do not generally display pathological behavior for large literals. I'm not sure who posted that comment, but if they are on the Swift team and blaming users, that should stop. Code like that is out there, and it's the job of tooling to handle it gracefully.
- refulgentis 3y ago
  It wasn’t a Swift team member, Swift community has a quick response for these cases because it’s been around forever and is unique enough of a “simple” case that it, ex. ends up at #1 on HN. My understanding of Swift’s teams thoughts, though a few years out of date, is it’s very very hard to get the Swift type inference engine to handle a huge blob, without type hinting, or with literals that could be multiple types.
  - deleted 3y ago
    [deleted]
  - earthnail 3y ago
    Three years ago I tried putting constants for a matrix (I think it was 1024x80 or sth like that) into a strongly typed array of floats. Still took the compiler 30mins to compile it. In the end I split the matrix up in rows, with each row being one variable, and a final variable to concatenate the rows. No typing hint whatsoever helped. Big wtf moment.
    - refulgentis 3y ago
      Yeah this one's real strange to me. I feel like it used to be as simple as the [int] definition people suggest in the thread. I don't miss Xcode too much. IIRC I built some custom script using arcanda dropped by Apple employees to output per-function build times. It was great! I knew exactly what functions were slow. But now I am old and lazy lol.
  - euiq 3y ago
    Note that someone who seems to manage a Swift team at Apple left a more encouraging response later: <https://forums.swift.org/t/why-is-swift-so-slow-timeout-in-compiling-this-code/61382/5 https://forums.swift.org/t/why-is-swift-so-slow-timeout-in-c...>
jandrese 3y ago
For an example of C using this in real life open a .xpm file in a text editor. There are many implementations of a C tool that converts a binary file into a .h. And yes, it is dumb that we have to do tricks like this in 2023.
tredre3 3y ago
> { 0xFF, 0xD8, 0xFF, 0xE0, ... } You're right of course but experienced developers know to write those as strings instead of arrays, especially for very large content. Otherwise both the compiler and IDE become painfully slow. ie your example would be written "\xFF\xD8\xFF\xE0..."
- tialaramex 3y ago
  Don't try that in MSVC. It deliberately chokes on large strings. If you work hard you might put a few thousand integers in a string, but this example has a million integers. Most ways you can attempt that in MSVC time out, abort compilation or emit an error.
twoodfin 3y ago
I once had good reason (or at least I thought so!) to code-generate nested structures into .rodata by way of thousands of static arrays (each with its own variable), referenced by other static arrays, referenced by static structs… The application was a highly optimized encoding of Aho-Corasick state machines for megabytes of static dictionary strings. The code generation (not in C) was trivial, and along with .rodata came all the benefits of shared pages and lazy loading. Across a number of compilers I only ran into one bug in 32-bit gcc, which was worked around easily by disabling the (unhelpful) optimization pass that was getting snarled.

tinus_hn 3y ago

Actually the xbm image format is c files like this with a long array of hexadecimal numbers. https://en.m.wikipedia.org/wiki/X_BitMap https://en.m.wikipedia.org/wiki/X_BitMap

bluedino 3y ago
I remember paint programs that could export images as C/Pascal/whatever source code files.
- stefncb 3y ago
  Gimp still does that, it's rather useful if you want to play around with images without a loader.

ajross 3y ago

> Actually, no. There is no portable "incbin" in C/C++ I kinda hate this kind of argument. I mean, it's true, as .incbin is a GNU assembler directive (FWIW: binutils/llvm objcopy is a better mechanism still for this sort of thing in most contexts, as it doesn't involve source compilation of any kind). But it leads ridiculous design decisions, like "I'm going to write 8MB of source code instead of doing the portability work on my own to turn that data into a linkable symbol". There's a huge gulf in the space between "non-portable" and "impossible". In this case, the problem is trivially solved on every platform ever. Trivial problems should employ trivial solutions, even where they have to involve some per-platform engineering.

duped 3y ago
> But it leads ridiculous design decisions, like "I'm going to write 8MB of source code instead of doing the portability work on my own to turn that data into a linkable symbol". people use scripts to do this kind of thing and never think about it again, and it's still more portable than writing a custom build step.
iainmerrick 3y ago
But it leads ridiculous design decisions, like "I'm going to write 8MB of source code instead of doing the portability work on my own to turn that data into a linkable symbol". Why is that ridiculous? It strikes me as not necessarily the best, but the most obvious approach, the most portable, and possibly the quickest to implement. Your toolchain might have a special way to import binary blobs, but a) you’ll have to dig through the docs to find it, b) you’ll probably need to solve the problem again when porting to a different platform, and c) who knows if it actually works, or if there are hidden gotchas? Sure, if there’s a known tool or option that does the job, you should go ahead and use it. But in general, writing a little script to generate a bunch of boilerplate code is perfectly workable.
- ajross 3y ago
  > Your toolchain might have a special way to import binary blobs, but a) you’ll have to dig through the docs to find it This is a corrollary: "I don't want to learn my tools, so I'll learn the language standard instead" is fundamentally exactly the problem I'm talking about. Straight up: C linkage is a 1970's paradigm full of tools that had to run on a PDP/11, and it's vastly simpler than learning C++ or Rust. It's just not "modern" and no one taught it to you, so it looks weird and mysterious. That's the problem!
  - maskros 3y ago
    It's not "I don't want to learn my tools." It's "I don't want to learn and debug _everybody else who may possible want to build this otherwise portable C program_'s tools." When those tools change how they do this unportable thing every couple of years in subtle and incompatible ways, which require #ifdef's to handle the different ways those linked against symbols can be accessed, multiplied by dozens of different platforms, then yes, I'm going to compile an 8MB literal.

tialaramex 3y ago

C23 gets #embed https://thephd.dev/finally-embed-in-c23 https://thephd.dev/finally-embed-in-c23 and in practice later C++ compilers will presumably also just do #embed rather than pretend they aren't the same code anyway. However, even though there's a clear need, as that post explains, the compilers varied between "Bad at this" and "Completely awful at this" which is part of what was so frustrating for JeanHyde over years of getting this through committee. In principle #embed is just shoving the bytes in as comma separated values, but in practice by the "as if" rule in C and C++ the compiler won't do that because it's stupid.

est31 3y ago
Since Rust 1.67.0, the compiler has a specific optimized code path for the (already older) include_bytes! macro: https://github.com/rust-lang/rust/pull/103812#issuecomment-1299087888 https://github.com/rust-lang/rust/pull/103812#issuecomment-1...
- tialaramex 3y ago
  Ha, cute, I think I began examining details of include_bytes! only this year so it's possible I happened to first look at this soon after the big improvements landed in the release I was testing against.

armchairhacker 3y ago

finally. #embed https://thephd.dev/finally-embed-in-c23 https://thephd.dev/finally-embed-in-c23 tl;dr It was such a big issue they actually managed to get it through the C Committee

titzer 3y ago

Also, a reminder that JS engines do regularly have to contend with enormous JSON and do not generally display pathological behavior for large literals. I'm not sure who posted that comment, but if they are on the Swift team and blaming users, that should stop. Code like that is out there, and it's the job of tooling to handle it gracefully.

refulgentis 3y ago
It wasn’t a Swift team member, Swift community has a quick response for these cases because it’s been around forever and is unique enough of a “simple” case that it, ex. ends up at #1 on HN. My understanding of Swift’s teams thoughts, though a few years out of date, is it’s very very hard to get the Swift type inference engine to handle a huge blob, without type hinting, or with literals that could be multiple types.
- deleted 3y ago
  [deleted]
- earthnail 3y ago
  Three years ago I tried putting constants for a matrix (I think it was 1024x80 or sth like that) into a strongly typed array of floats. Still took the compiler 30mins to compile it. In the end I split the matrix up in rows, with each row being one variable, and a final variable to concatenate the rows. No typing hint whatsoever helped. Big wtf moment.
  - refulgentis 3y ago
    Yeah this one's real strange to me. I feel like it used to be as simple as the [int] definition people suggest in the thread. I don't miss Xcode too much. IIRC I built some custom script using arcanda dropped by Apple employees to output per-function build times. It was great! I knew exactly what functions were slow. But now I am old and lazy lol.
- euiq 3y ago
  Note that someone who seems to manage a Swift team at Apple left a more encouraging response later: <https://forums.swift.org/t/why-is-swift-so-slow-timeout-in-compiling-this-code/61382/5 https://forums.swift.org/t/why-is-swift-so-slow-timeout-in-c...>

jandrese 3y ago

For an example of C using this in real life open a .xpm file in a text editor. There are many implementations of a C tool that converts a binary file into a .h. And yes, it is dumb that we have to do tricks like this in 2023.

tredre3 3y ago

> { 0xFF, 0xD8, 0xFF, 0xE0, ... } You're right of course but experienced developers know to write those as strings instead of arrays, especially for very large content. Otherwise both the compiler and IDE become painfully slow. ie your example would be written "\xFF\xD8\xFF\xE0..."

tialaramex 3y ago
Don't try that in MSVC. It deliberately chokes on large strings. If you work hard you might put a few thousand integers in a string, but this example has a million integers. Most ways you can attempt that in MSVC time out, abort compilation or emit an error.

twoodfin 3y ago

I once had good reason (or at least I thought so!) to code-generate nested structures into .rodata by way of thousands of static arrays (each with its own variable), referenced by other static arrays, referenced by static structs… The application was a highly optimized encoding of Aho-Corasick state machines for megabytes of static dictionary strings. The code generation (not in C) was trivial, and along with .rodata came all the benefits of shared pages and lazy loading. Across a number of compilers I only ran into one bug in 32-bit gcc, which was worked around easily by disabling the (unhelpful) optimization pass that was getting snarled.