3 ms·

While the pursuit of faster build times is definitely a worthy cause, I feel like there's something I'm not quite seeing here. Does the JSON-code change frequen

by occz 2y ago

While the pursuit of faster build times is definitely a worthy cause, I feel like there's something I'm not quite seeing here. Does the JSON-code change frequently enough to incur build cache misses and the full minute penalty? Is there something inherent about the structure of the library that makes it unable to have its compilation be cached? Is the code structured in such a way that editing other code requires also invalidating the cache for the JSON-related code? I guess one way would be to break out the JSON parsing code to its own module and have it produce language-specific structs to be interacted with by the rest of the program.

jart 2y ago
Programming is the process of manipulating data structures, so if you're building a JSON server, then every piece of code in your server is going to be dealing with and operating on JSON data structures. It can't be neatly tucked away in a corner. Because it would be foolish to design a server that makes needless copies of all its inputs and outputs. This truth would be the same if you were using something like protobuf instead. Therefore it's important that your fundamental data structures be something that (a) you can control, and (b) doesn't make everything it touches take forever to build. Do you feel in control of someone else's 24000 line header full of template magic? If that thing is sitting between me and my data structures, then I will wipe it out of existence.
mort96 2y ago
It seems like nlohmann/json is a header-only library, meaning the entire library has to be compiled once for every source file which uses it any time that source file or its includes has updated. So I guess in a JSON-heavy code base or a code base where nlohmann/json has leaked into common headers, you may end up recompiling the library a few dozen times per build where a few dozen of your C++ source files must be recompiled (e.g due to common header changes)... (But don't worry, the linker will then spend a bunch of time throwing away almost all of that work so you only get one copy of the library in your binary)
- occz 2y ago
  I missed that part. That is a pretty significant downside in that case.
  - jart 2y ago
    It places a hard scaling limit on how big an open source project can become. Projects like the Linux Kernel spend enormous amounts of political capital restraining decadent programming practices, since it's the only way a codebase like that can maintain the support of its developers and grow. For example, Linux had a rule until 2018 that everything had to be able to compile with GCC 3.2 from 2003. They're much more laid back these days, since it's difficult to imagine Linux growing bigger than it already has. But I think for a newer project like llama.cpp would be well advised to follow by example what projects like Linux did in their growth phase, rather than following their leadership today. It requires an lot of discipline, toil, and restraint to be a leader in open source, because you're essentially offering the world a pot of gold, and that only works if you keep very little for yourself.
    - occz 2y ago
      Looking at the source code for llama.cpp, it's not strictly speaking set up for success as far as build times are concerned, is it? Having 20k+ lines is a bad idea for many reasons, and this is certainly one of them. I'm sure the authors are brilliant in many ways, but I suspect there's room for improvement in this area.
wiseowise 2y ago
> Does the JSON-code change frequently enough to incur build cache misses and the full minute penalty? The moment you switch branches - it changes. If you develop for Android - it generates build for with hash name from some CMake/Gradle variables, the moment one of those changes (like AGP version) you get a new build dir and essentially have to compile from scratch.
- occz 2y ago
  If you're on something reasonably smart like Bazel it will be able to determine whether the module itself has been changed and requires recompilation instead of running from cache.
  - wiseowise 2y ago
    Nice. We, and majority of Android projects, aren’t on Bazel, though.
    - occz 2y ago
      This is true, and it's kind of a bummer to be honest. There's some serious time being wasted on recompilation that could be avoided with a really sharp build system. Bazel comes with its own bag of sharp edges though so it's unfortunately not like you can just adopt it and be on your merry way.

by occz 2y ago

jart 2y ago

Programming is the process of manipulating data structures, so if you're building a JSON server, then every piece of code in your server is going to be dealing with and operating on JSON data structures. It can't be neatly tucked away in a corner. Because it would be foolish to design a server that makes needless copies of all its inputs and outputs. This truth would be the same if you were using something like protobuf instead. Therefore it's important that your fundamental data structures be something that (a) you can control, and (b) doesn't make everything it touches take forever to build. Do you feel in control of someone else's 24000 line header full of template magic? If that thing is sitting between me and my data structures, then I will wipe it out of existence.

mort96 2y ago

It seems like nlohmann/json is a header-only library, meaning the entire library has to be compiled once for every source file which uses it any time that source file or its includes has updated. So I guess in a JSON-heavy code base or a code base where nlohmann/json has leaked into common headers, you may end up recompiling the library a few dozen times per build where a few dozen of your C++ source files must be recompiled (e.g due to common header changes)... (But don't worry, the linker will then spend a bunch of time throwing away almost all of that work so you only get one copy of the library in your binary)

occz 2y ago
I missed that part. That is a pretty significant downside in that case.
- jart 2y ago
  It places a hard scaling limit on how big an open source project can become. Projects like the Linux Kernel spend enormous amounts of political capital restraining decadent programming practices, since it's the only way a codebase like that can maintain the support of its developers and grow. For example, Linux had a rule until 2018 that everything had to be able to compile with GCC 3.2 from 2003. They're much more laid back these days, since it's difficult to imagine Linux growing bigger than it already has. But I think for a newer project like llama.cpp would be well advised to follow by example what projects like Linux did in their growth phase, rather than following their leadership today. It requires an lot of discipline, toil, and restraint to be a leader in open source, because you're essentially offering the world a pot of gold, and that only works if you keep very little for yourself.
  - occz 2y ago
    Looking at the source code for llama.cpp, it's not strictly speaking set up for success as far as build times are concerned, is it? Having 20k+ lines is a bad idea for many reasons, and this is certainly one of them. I'm sure the authors are brilliant in many ways, but I suspect there's room for improvement in this area.

wiseowise 2y ago

> Does the JSON-code change frequently enough to incur build cache misses and the full minute penalty? The moment you switch branches - it changes. If you develop for Android - it generates build for with hash name from some CMake/Gradle variables, the moment one of those changes (like AGP version) you get a new build dir and essentially have to compile from scratch.

occz 2y ago
If you're on something reasonably smart like Bazel it will be able to determine whether the module itself has been changed and requires recompilation instead of running from cache.
- wiseowise 2y ago
  Nice. We, and majority of Android projects, aren’t on Bazel, though.
  - occz 2y ago
    This is true, and it's kind of a bummer to be honest. There's some serious time being wasted on recompilation that could be avoided with a really sharp build system. Bazel comes with its own bag of sharp edges though so it's unfortunately not like you can just adopt it and be on your merry way.