4 ms·

If someone wanted to reimplement ninja and add support for the GNU Make job server, I'd be sold. https://github.com/ninja-build/ninja/issues/1139 https://githu

If someone wanted to reimplement ninja and add support for the GNU Make job server, I'd be sold. https://github.com/ninja-build/ninja/issues/1139 https://github.com/ninja-build/ninja/issues/1139 At a certain scale (i.e. package management and not a mono build system), having a common job server is essential to scaling in a healthy way.

mastax 5y ago
Aside: the make job server is very simple and UNIXy. There's a pipe shared between `make` and its subprocesses. `make` writes `N-1` bytes to the pipe. When a job wants to do parallel work, it reads a byte from the pipe. When it's done, it writes a byte back to the pipe. (Both rustc and cargo support the make job server)
- stormbrew 5y ago
  It's really an elegant design, which is nice, other than the way that information is passed into the child process (roughly: MAKEFLAGS=" -j --job-server=3,4"), which requires you to argparse out the fd numbers. It would be nice if a new, less cumbersome, environment protocol could be adopted.
- ajconway 5y ago
  What happens when the job crashes?
stormbrew 5y ago
I dunno if they'll accept it but I might actually submit a PR to add this to n2, because I also would very much like "ninja with jobserver" and it being written in rust it should be fairly easy to add since the code cargo uses for jobserver support is broken out into a very simple library.
- jhasse 5y ago
  Do you mean jobserver client or sever? As for "ninja with jobserver client" there's already https://github.com/Kitware/ninja https://github.com/Kitware/ninja
  - stormbrew 5y ago
    > Do you mean jobserver client or sever? Both. But server is more important to me usually (ninja running cargo, in particular -- but also sometimes ninja running make or ninja running ninja). Technically you can obviously kinda trick client-only support into being server support by wrapping ninja in a jobserver like make, and letting that jobserver cascade down, but that's pretty gross. So while client support is nice, it's not really sufficient. I actually don't think I have much use for client-without-server at all, the main configuration I need client for at all is ninja->ninja. There's also that kitware's is a one-purpose fork that's just kind of a pain to install in CI because lack of packages, but it's pretty easy to `cargo install n2` on a machine where I'm building rust anyways, and cache the CARGO_HOME so it doesn't build every time. To be honest, with the fact that compilers are no longer single-threaded almost as a rule, something like jobserver seems like an essential feature in a build system now. It's basically impossible to meaningfully limit concurrency without some level of coordination or fuzzy imperfect indicators like load average. I hope other compilers follow rust's lead here and implement client support eventually. (and then hopefully a more robust common protocol can evolve out of it)

mastax 5y ago

Aside: the make job server is very simple and UNIXy. There's a pipe shared between `make` and its subprocesses. `make` writes `N-1` bytes to the pipe. When a job wants to do parallel work, it reads a byte from the pipe. When it's done, it writes a byte back to the pipe. (Both rustc and cargo support the make job server)

stormbrew 5y ago
It's really an elegant design, which is nice, other than the way that information is passed into the child process (roughly: MAKEFLAGS=" -j --job-server=3,4"), which requires you to argparse out the fd numbers. It would be nice if a new, less cumbersome, environment protocol could be adopted.
ajconway 5y ago
What happens when the job crashes?

stormbrew 5y ago

I dunno if they'll accept it but I might actually submit a PR to add this to n2, because I also would very much like "ninja with jobserver" and it being written in rust it should be fairly easy to add since the code cargo uses for jobserver support is broken out into a very simple library.

jhasse 5y ago
Do you mean jobserver client or sever? As for "ninja with jobserver client" there's already https://github.com/Kitware/ninja https://github.com/Kitware/ninja
- stormbrew 5y ago
  > Do you mean jobserver client or sever? Both. But server is more important to me usually (ninja running cargo, in particular -- but also sometimes ninja running make or ninja running ninja). Technically you can obviously kinda trick client-only support into being server support by wrapping ninja in a jobserver like make, and letting that jobserver cascade down, but that's pretty gross. So while client support is nice, it's not really sufficient. I actually don't think I have much use for client-without-server at all, the main configuration I need client for at all is ninja->ninja. There's also that kitware's is a one-purpose fork that's just kind of a pain to install in CI because lack of packages, but it's pretty easy to `cargo install n2` on a machine where I'm building rust anyways, and cache the CARGO_HOME so it doesn't build every time. To be honest, with the fact that compilers are no longer single-threaded almost as a rule, something like jobserver seems like an essential feature in a build system now. It's basically impossible to meaningfully limit concurrency without some level of coordination or fuzzy imperfect indicators like load average. I hope other compilers follow rust's lead here and implement client support eventually. (and then hopefully a more robust common protocol can evolve out of it)