3 ms·

But it leads ridiculous design decisions, like "I'm going to write 8MB of source code instead of doing the portability work on my own to turn that data into a l

But it leads ridiculous design decisions, like "I'm going to write 8MB of source code instead of doing the portability work on my own to turn that data into a linkable symbol". Why is that ridiculous? It strikes me as not necessarily the best, but the most obvious approach, the most portable, and possibly the quickest to implement. Your toolchain might have a special way to import binary blobs, but a) you’ll have to dig through the docs to find it, b) you’ll probably need to solve the problem again when porting to a different platform, and c) who knows if it actually works, or if there are hidden gotchas? Sure, if there’s a known tool or option that does the job, you should go ahead and use it. But in general, writing a little script to generate a bunch of boilerplate code is perfectly workable.

ajross 3y ago
> Your toolchain might have a special way to import binary blobs, but a) you’ll have to dig through the docs to find it This is a corrollary: "I don't want to learn my tools, so I'll learn the language standard instead" is fundamentally exactly the problem I'm talking about. Straight up: C linkage is a 1970's paradigm full of tools that had to run on a PDP/11, and it's vastly simpler than learning C++ or Rust. It's just not "modern" and no one taught it to you, so it looks weird and mysterious. That's the problem!
- maskros 3y ago
  It's not "I don't want to learn my tools." It's "I don't want to learn and debug _everybody else who may possible want to build this otherwise portable C program_'s tools." When those tools change how they do this unportable thing every couple of years in subtle and incompatible ways, which require #ifdef's to handle the different ways those linked against symbols can be accessed, multiplied by dozens of different platforms, then yes, I'm going to compile an 8MB literal.
- josephg 3y ago
  I go back and forth on this argument when it comes to codegen. Like, you could make the same argument that protobuf shouldn't output C code. It should output an object file that you can link into whatever compiled language you want. C, fortran, C++, rust, who cares. As you say, the linking model is simple and works well. Why do we generate big C/C++ strings instead, and compile those? Because object files have lots of compiler/platform/architecture specific stuff in them. Outputting C (or C++) then compiling it is a much more convenient way to generate those object files, because it works on every OS, compiler and architecture. Even systems that you don't know about, or that don't exist yet. I hear what you're saying and I'm torn about it. I mean, aren't binary blobs just a simpler version of the problem protobuf faces? C code is already the most portable way to make object files. Why wouldn't we use it?
  - ajross 3y ago
    > Like, you could make the same argument that protobuf shouldn't output C code. The case at hand is an 8MB static array of integers. Obviously yes, of course, absolutely: you choose the correct/simple/obvious/trivialest implementation strategy for the problem. That's exactly what I'm saying! In the case of protobuf (static generation of an otherwise arbitrarily complicated data structure with reasonably bounded size), code generation makes a ton of sense.
    - Joker_vD 3y ago
      And the simple/obvious/trivialest solution is to write an array literal with 4 millions integers in it, while fighting with Microsoft's link.exe is anything but. Even using rc.exe and loading that data from the resource section is a non-trivial amount of additional work.
      - deleted 3y ago
        [deleted]
      - ajross 3y ago
        Come on. Both binutils and nasm can generate perfectly working PE object files. I don't know the answer off the top of my head, but I bet anything even pure MSVC has a simple answer here. Dealing with the nonsense in the linked article is something you do for a quick hack or to test compiler performance, but (as demonstrated!) it scales poorly. It's terrible engineering, period. Use the right tools, even if they aren't ISO-specified. And if you can't or won't, please don't get into fights on the internet justifying the resulting hackery.
        Joker_vD 3y ago
        > I bet anything even pure MSVC has a simple answer here You lose your bet, because it doesn't, neither its inline assembler nor actual MASM shipped with Visual Studio support any "incbin"-like directives. I guess you can generate an .asm with a huge db/dd, I guess, if you don't like a large literal array in .c files, but that's it.
- Joker_vD 3y ago
  I am pretty certain I've seen linkers routinely writing 0 instead of symbols' actual sizes so getting the actual size of the embedded binary blob is not very pretty. Also, you argument sounds exactly like those that the author of https://thephd.dev/finally-embed-in-c23 https://thephd.dev/finally-embed-in-c23 has been fighting against for 5 years straight.
  - iainmerrick 3y ago
    Yes! That person is a hero.
- iainmerrick 3y ago
  It's just not "modern" and no one taught it to you, so it looks weird and mysterious. I don’t know what to say except that I’ve worked with C linkers for a long time, since before I learned C++ and before Rust even existed, and I still don’t like ‘em.

ajross 3y ago

> Your toolchain might have a special way to import binary blobs, but a) you’ll have to dig through the docs to find it This is a corrollary: "I don't want to learn my tools, so I'll learn the language standard instead" is fundamentally exactly the problem I'm talking about. Straight up: C linkage is a 1970's paradigm full of tools that had to run on a PDP/11, and it's vastly simpler than learning C++ or Rust. It's just not "modern" and no one taught it to you, so it looks weird and mysterious. That's the problem!

maskros 3y ago
It's not "I don't want to learn my tools." It's "I don't want to learn and debug _everybody else who may possible want to build this otherwise portable C program_'s tools." When those tools change how they do this unportable thing every couple of years in subtle and incompatible ways, which require #ifdef's to handle the different ways those linked against symbols can be accessed, multiplied by dozens of different platforms, then yes, I'm going to compile an 8MB literal.
josephg 3y ago
I go back and forth on this argument when it comes to codegen. Like, you could make the same argument that protobuf shouldn't output C code. It should output an object file that you can link into whatever compiled language you want. C, fortran, C++, rust, who cares. As you say, the linking model is simple and works well. Why do we generate big C/C++ strings instead, and compile those? Because object files have lots of compiler/platform/architecture specific stuff in them. Outputting C (or C++) then compiling it is a much more convenient way to generate those object files, because it works on every OS, compiler and architecture. Even systems that you don't know about, or that don't exist yet. I hear what you're saying and I'm torn about it. I mean, aren't binary blobs just a simpler version of the problem protobuf faces? C code is already the most portable way to make object files. Why wouldn't we use it?
- ajross 3y ago
  > Like, you could make the same argument that protobuf shouldn't output C code. The case at hand is an 8MB static array of integers. Obviously yes, of course, absolutely: you choose the correct/simple/obvious/trivialest implementation strategy for the problem. That's exactly what I'm saying! In the case of protobuf (static generation of an otherwise arbitrarily complicated data structure with reasonably bounded size), code generation makes a ton of sense.
  - Joker_vD 3y ago
    And the simple/obvious/trivialest solution is to write an array literal with 4 millions integers in it, while fighting with Microsoft's link.exe is anything but. Even using rc.exe and loading that data from the resource section is a non-trivial amount of additional work.
    - deleted 3y ago
      [deleted]
    - ajross 3y ago
      Come on. Both binutils and nasm can generate perfectly working PE object files. I don't know the answer off the top of my head, but I bet anything even pure MSVC has a simple answer here. Dealing with the nonsense in the linked article is something you do for a quick hack or to test compiler performance, but (as demonstrated!) it scales poorly. It's terrible engineering, period. Use the right tools, even if they aren't ISO-specified. And if you can't or won't, please don't get into fights on the internet justifying the resulting hackery.
      - Joker_vD 3y ago
        > I bet anything even pure MSVC has a simple answer here You lose your bet, because it doesn't, neither its inline assembler nor actual MASM shipped with Visual Studio support any "incbin"-like directives. I guess you can generate an .asm with a huge db/dd, I guess, if you don't like a large literal array in .c files, but that's it.
Joker_vD 3y ago
I am pretty certain I've seen linkers routinely writing 0 instead of symbols' actual sizes so getting the actual size of the embedded binary blob is not very pretty. Also, you argument sounds exactly like those that the author of https://thephd.dev/finally-embed-in-c23 https://thephd.dev/finally-embed-in-c23 has been fighting against for 5 years straight.
- iainmerrick 3y ago
  Yes! That person is a hero.
iainmerrick 3y ago
It's just not "modern" and no one taught it to you, so it looks weird and mysterious. I don’t know what to say except that I’ve worked with C linkers for a long time, since before I learned C++ and before Rust even existed, and I still don’t like ‘em.