27 ms·
This works well for expensive long-running things, but works less well for more short-lived programs, and even for expensive long-running things there may be mo
by arp242 3y ago
This works well for expensive long-running things, but works less well for more short-lived programs, and even for expensive long-running things there may be more considerations than "what's the fastest value?" I intentionally set the default jobs of "make" to "4" rather than "8" because I don't want to utilize my full system when compiling stuff, so I can still use it for other things at a reasonable speed. There's lots of reasons you might want to do this: on a web platform you don't really want to use up all resources by the one expensive thing, but what's "reasonable" and "desired" often differs – often there is no objectively best "optimum setting".
- onetimeuse92304 3y agoYou do not have to target best absolute performance. You can choose your own fitness functions to balance absolute performance and other aspects of the system like resource usage, latency, or any other aspects of the application. Think about what logic the operator would use to establish the parameter and capture it as code or as fitness function. As usual, there is no single universal solution to optimising performance (and in fact it can be trivially proven to not exist in general case). But you do not have to find perfect solution, you just have to be able to find as good or better solution than a human operator would. When you have a large system you can be working with thousands or tens of thousands of parameters. I usually see that even if the application is migrated to different hardware, the configuration parameters like buffer sizes, concurrency, connection pools, etc. are not really revisited. The effect is that usually the application will work suboptimally for the environment it is in. My solution is an attempt to keep this part of application configuration sufficiently close enough to optimal to be usable and do it automatically without operator involvement (and associated cost and possibility for defects).
- vlovich123 3y agoYeah I’m in agreement with you. It can be hard for performance things (vs tuning parameters for numerical algorithm performance) because such tests can be inherently difficult to reproduce and writing the hill climbing can be tricky. It’s a goal to aspire to for sure. Have you found any tooling to do the hill climb search for you in a multidimensional space?
- geysersam 3y agoIf you're using python `scipy.optimize` has several good options for the minimization procedure. I guess you could even use those if you're not using python. But it'd require slightly more plumbing.
- Palomides 3y agoconsider running make with nice, let the scheduler work for you
- arp242 3y agoActually I do both: % alias make make='nice -n 20 make -j4' But I found that even with a nice level of 20 using all CPUs doesn't quite give me the responsiveness I want for other tasks.
- onetimeuse92304 3y agoPut it in a container where you can explicitly limit CPU / memory usage? I run all builds in containers/vms anyway as I am too tired to keep my personal machine able to build hundreds of different things. It is better to have a dedicated development environment frozen for each of the projects I work on.
- arp242 3y agoI just want to run make and not slow down Firefox or maybe some other things I'm doing. This simple alias I've had for over 20 years works, on all (Unix-y) systems. Why make everything a hypercomplex story?
- wtallis 3y agoMake also has an option to only spawn new jobs when the load average is below a given value, which can be used to get make to reduce the number of jobs running in parallel when there are plenty of other processes using CPU time.