6 ms·

Good example of how we develop software these days: instead of fixing GNU make by ripping out that antiquated pipe-based jobserver and replacing it with proper

by boris 7y ago

Good example of how we develop software these days: instead of fixing GNU make by ripping out that antiquated pipe-based jobserver and replacing it with proper multi-threaded parallelism[1], we rather optimize the operating system to make the antiquated stuff work better. [1] Having hacked on GNU make I know this won't be easy (it's quite a mess). In fact, it would probably be easier to replace it entirely and fix some of its other deficiencies in the process.

jankotek 7y ago
Maybe there is a good reason for that. There are already dozens of alternatives to GNU Make. Linux already did something similar with Git.
s3cur3 7y ago
Replacing Make in “your” project is relatively easy. Replacing it in all your dependencies is both a technical pain in the ass and a massive political problem if you want to upstream the change. If you have a lot of dependencies, it’s probably easier just to improve Make. ;)
temac 7y ago
I would not call avoiding a classic thundering herd problem a bad choice, especially when the top maintainer of Linux does it himself in Linux: which is arguably more probable than him rewriting GNU Make (well we all know he could do it, but this would still be less probable...) Plus it will not only optimize GNU Make workloads, but potentially other programs that do that. And to be honest, playing with pipe for signaling (and more rarely token counting) is STILL a must-have way to do IPC portably, esp. if you want to integrate with polling on fds. Various OSes have also various new things, but they are non-portable -- maybe someone should try to get eventfd&friends in POSIX? Anyway, if that's a "good example of how we develop software these days", I would say we managed to develop software the right way at last, making basic stuff work well before succumbing to the tentation to rewrite everything all the time just to play with new shinny techs, getting fresh bugs in the process. Also you know what has evolved in the last few years, motivating this perf fix? Availability of HIGHLY parallel computers for not too expensive. Yes, I continue to find it highly refreshing that work is done to get all the boring historical software work better in that context. Maybe you think we can afford rewriting all the things all the time to follow supposedly radical technological changes (and using less well designed OSes?), but if so then this thought is not shared widely or at least not followed with action. Rightly so because economically the outcome would be doubtful: just look at the patch, it is way smaller than an hypothetical shiny new rewrite of GNU Make...
gpderetta 7y ago
Make spawns processes. Using threads would be a terrible idea. fork/execve and pthread_create don't mix well.
imtringued 7y ago
The solution is elegant and it has a much bigger impact than just GNU make. There are performance benefits across the board. Maybe the entire system runs 1% faster now. Add a few hundred patches like these and you get a very fast system.
downerending 7y ago
> proper multi-threaded parallelism So, like buggy and hard to understand?

by boris 7y ago

jankotek 7y ago

Maybe there is a good reason for that. There are already dozens of alternatives to GNU Make. Linux already did something similar with Git.

s3cur3 7y ago

Replacing Make in “your” project is relatively easy. Replacing it in all your dependencies is both a technical pain in the ass and a massive political problem if you want to upstream the change. If you have a lot of dependencies, it’s probably easier just to improve Make. ;)

temac 7y ago

I would not call avoiding a classic thundering herd problem a bad choice, especially when the top maintainer of Linux does it himself in Linux: which is arguably more probable than him rewriting GNU Make (well we all know he could do it, but this would still be less probable...) Plus it will not only optimize GNU Make workloads, but potentially other programs that do that. And to be honest, playing with pipe for signaling (and more rarely token counting) is STILL a must-have way to do IPC portably, esp. if you want to integrate with polling on fds. Various OSes have also various new things, but they are non-portable -- maybe someone should try to get eventfd&friends in POSIX? Anyway, if that's a "good example of how we develop software these days", I would say we managed to develop software the right way at last, making basic stuff work well before succumbing to the tentation to rewrite everything all the time just to play with new shinny techs, getting fresh bugs in the process. Also you know what has evolved in the last few years, motivating this perf fix? Availability of HIGHLY parallel computers for not too expensive. Yes, I continue to find it highly refreshing that work is done to get all the boring historical software work better in that context. Maybe you think we can afford rewriting all the things all the time to follow supposedly radical technological changes (and using less well designed OSes?), but if so then this thought is not shared widely or at least not followed with action. Rightly so because economically the outcome would be doubtful: just look at the patch, it is way smaller than an hypothetical shiny new rewrite of GNU Make...

gpderetta 7y ago

Make spawns processes. Using threads would be a terrible idea. fork/execve and pthread_create don't mix well.

imtringued 7y ago

The solution is elegant and it has a much bigger impact than just GNU make. There are performance benefits across the board. Maybe the entire system runs 1% faster now. Add a few hundred patches like these and you get a very fast system.

downerending 7y ago

> proper multi-threaded parallelism So, like buggy and hard to understand?