4 ms·

Personally, I dislike configuration parameters that let future admins of the system change parameters like concurrency of certain processes, etc. A lot of the

Personally, I dislike configuration parameters that let future admins of the system change parameters like concurrency of certain processes, etc. A lot of the time even I can't tell what is going to be the optimum setting. So for a number of years, rather than expose a configuration parameter, I am building in a performance test that establishes the values of these parameters. This is a little bit of search and a hill climb through the multidimensional space of all relevant parameters. It is not perfect, but in my experience it is almost always better than I can do manually and always better than an operator without intimate understanding of the system. The results are cached in a database and can be re-executed if change to configuration is found (just take a number of parameters from your environment, version of the application, etc. and hash the value and check if you already have parameters in the database).

eropple 3y ago
This is interesting - how long does it take to write? Is it reusable? I'd be really interested in a blog post about this.
- onetimeuse92304 3y ago
  I am not big about writing blog posts. But now that I think about it, I could write a small library to automate most of it while delegating some tasks to the application (providing functionality to be tested, parameters and possible ranges to be searched, a way to save/restore parameters to/from the database, provide fitness function, etc.) It is essentially a small benchmarking framework with some added functionality. I think it could be quite useful to some people.
arp242 3y ago
This works well for expensive long-running things, but works less well for more short-lived programs, and even for expensive long-running things there may be more considerations than "what's the fastest value?" I intentionally set the default jobs of "make" to "4" rather than "8" because I don't want to utilize my full system when compiling stuff, so I can still use it for other things at a reasonable speed. There's lots of reasons you might want to do this: on a web platform you don't really want to use up all resources by the one expensive thing, but what's "reasonable" and "desired" often differs – often there is no objectively best "optimum setting".
- onetimeuse92304 3y ago
  You do not have to target best absolute performance. You can choose your own fitness functions to balance absolute performance and other aspects of the system like resource usage, latency, or any other aspects of the application. Think about what logic the operator would use to establish the parameter and capture it as code or as fitness function. As usual, there is no single universal solution to optimising performance (and in fact it can be trivially proven to not exist in general case). But you do not have to find perfect solution, you just have to be able to find as good or better solution than a human operator would. When you have a large system you can be working with thousands or tens of thousands of parameters. I usually see that even if the application is migrated to different hardware, the configuration parameters like buffer sizes, concurrency, connection pools, etc. are not really revisited. The effect is that usually the application will work suboptimally for the environment it is in. My solution is an attempt to keep this part of application configuration sufficiently close enough to optimal to be usable and do it automatically without operator involvement (and associated cost and possibility for defects).
  - vlovich123 3y ago
    Yeah I’m in agreement with you. It can be hard for performance things (vs tuning parameters for numerical algorithm performance) because such tests can be inherently difficult to reproduce and writing the hill climbing can be tricky. It’s a goal to aspire to for sure. Have you found any tooling to do the hill climb search for you in a multidimensional space?
    - geysersam 3y ago
      If you're using python `scipy.optimize` has several good options for the minimization procedure. I guess you could even use those if you're not using python. But it'd require slightly more plumbing.
- Palomides 3y ago
  consider running make with nice, let the scheduler work for you
fritzo 3y ago
Sounds similar to ATLAS, an automatically tuned BLAS library from the 1990s https://softlib.rice.edu/pub/CRPC-TRs/reports/CRPC-TR98751.pdf https://softlib.rice.edu/pub/CRPC-TRs/reports/CRPC-TR98751.p...
- onetimeuse92304 3y ago
  I am not surprised. I like to say that it is hard to have an original thought. Even if you think you are original, it is more likely somebody else has already done it in the past. Most of the computing novelties I am seeing are really old ideas, revisited and potentially improved upon. Thanks for the link, I will take a look when I have a moment:)
  - bee_rider 3y ago
    Eventually the pendulum swung back for BLAS libraries, the currently popular strategy is hand-tuned assembly kernel in libraries where the most important kernels have been identified. But BLAS is a very weird case, there are only a couple kernels that you care about (and the most important, GEMM, has been studied like crazy). And BLAS is used under the hood by pretty much the whole HPC community, so there’s incentive for tons of expert human time investment. You can look at ilbflame/BLIS for the modern strategy. ATLAS is still neat, perhaps one can even derive more generally applicable lessons from that library, not sure…
    - ska 3y ago
      I thought the point of ATLAS wasn't the idea that automagical tuning was going to necessarily be best. Rather that it was expensive to [edit hand tune] for all the hardware variations that were around and this was a practical approach to get to pretty damn good even if nobody had spent the money to produce a hand optimized library for your hardware (or, they had, but you didn't want to pay what they were asking). e.g. I remember vaguely people using it on AMD cpus because they couldn't use icc/mkl effectively...
      - bee_rider 3y ago
bashinator 3y ago
As a devops type person, I've been advocating for basic performance testing to be part of microservice unit tests. It'd be really wonderful to get rough numbers for peak and idle cpu/memory usage along with other test output.
invalidator 3y ago
FFTW [1] does something like this, not just for parallelism but also for selecting optimal algorithms for a given data set and architecture. It uses "wisdom" to store known-good combinations, falling back on quick heuristics and brief benchmarking when needed. If you're going to do a lot of something you can have it update the wisdom from full benchmarks. [1] https://www.fftw.org/ https://www.fftw.org/

eropple 3y ago

This is interesting - how long does it take to write? Is it reusable? I'd be really interested in a blog post about this.

onetimeuse92304 3y ago
I am not big about writing blog posts. But now that I think about it, I could write a small library to automate most of it while delegating some tasks to the application (providing functionality to be tested, parameters and possible ranges to be searched, a way to save/restore parameters to/from the database, provide fitness function, etc.) It is essentially a small benchmarking framework with some added functionality. I think it could be quite useful to some people.

arp242 3y ago

This works well for expensive long-running things, but works less well for more short-lived programs, and even for expensive long-running things there may be more considerations than "what's the fastest value?" I intentionally set the default jobs of "make" to "4" rather than "8" because I don't want to utilize my full system when compiling stuff, so I can still use it for other things at a reasonable speed. There's lots of reasons you might want to do this: on a web platform you don't really want to use up all resources by the one expensive thing, but what's "reasonable" and "desired" often differs – often there is no objectively best "optimum setting".

onetimeuse92304 3y ago
You do not have to target best absolute performance. You can choose your own fitness functions to balance absolute performance and other aspects of the system like resource usage, latency, or any other aspects of the application. Think about what logic the operator would use to establish the parameter and capture it as code or as fitness function. As usual, there is no single universal solution to optimising performance (and in fact it can be trivially proven to not exist in general case). But you do not have to find perfect solution, you just have to be able to find as good or better solution than a human operator would. When you have a large system you can be working with thousands or tens of thousands of parameters. I usually see that even if the application is migrated to different hardware, the configuration parameters like buffer sizes, concurrency, connection pools, etc. are not really revisited. The effect is that usually the application will work suboptimally for the environment it is in. My solution is an attempt to keep this part of application configuration sufficiently close enough to optimal to be usable and do it automatically without operator involvement (and associated cost and possibility for defects).
- vlovich123 3y ago
  Yeah I’m in agreement with you. It can be hard for performance things (vs tuning parameters for numerical algorithm performance) because such tests can be inherently difficult to reproduce and writing the hill climbing can be tricky. It’s a goal to aspire to for sure. Have you found any tooling to do the hill climb search for you in a multidimensional space?
  - geysersam 3y ago
    If you're using python `scipy.optimize` has several good options for the minimization procedure. I guess you could even use those if you're not using python. But it'd require slightly more plumbing.
Palomides 3y ago
consider running make with nice, let the scheduler work for you

fritzo 3y ago

Sounds similar to ATLAS, an automatically tuned BLAS library from the 1990s https://softlib.rice.edu/pub/CRPC-TRs/reports/CRPC-TR98751.pdf https://softlib.rice.edu/pub/CRPC-TRs/reports/CRPC-TR98751.p...

onetimeuse92304 3y ago
I am not surprised. I like to say that it is hard to have an original thought. Even if you think you are original, it is more likely somebody else has already done it in the past. Most of the computing novelties I am seeing are really old ideas, revisited and potentially improved upon. Thanks for the link, I will take a look when I have a moment:)
- bee_rider 3y ago
  Eventually the pendulum swung back for BLAS libraries, the currently popular strategy is hand-tuned assembly kernel in libraries where the most important kernels have been identified. But BLAS is a very weird case, there are only a couple kernels that you care about (and the most important, GEMM, has been studied like crazy). And BLAS is used under the hood by pretty much the whole HPC community, so there’s incentive for tons of expert human time investment. You can look at ilbflame/BLIS for the modern strategy. ATLAS is still neat, perhaps one can even derive more generally applicable lessons from that library, not sure…
  - ska 3y ago
    I thought the point of ATLAS wasn't the idea that automagical tuning was going to necessarily be best. Rather that it was expensive to [edit hand tune] for all the hardware variations that were around and this was a practical approach to get to pretty damn good even if nobody had spent the money to produce a hand optimized library for your hardware (or, they had, but you didn't want to pay what they were asking). e.g. I remember vaguely people using it on AMD cpus because they couldn't use icc/mkl effectively...
    - bee_rider 3y ago

bashinator 3y ago

As a devops type person, I've been advocating for basic performance testing to be part of microservice unit tests. It'd be really wonderful to get rough numbers for peak and idle cpu/memory usage along with other test output.

invalidator 3y ago

FFTW [1] does something like this, not just for parallelism but also for selecting optimal algorithms for a given data set and architecture. It uses "wisdom" to store known-good combinations, falling back on quick heuristics and brief benchmarking when needed. If you're going to do a lot of something you can have it update the wisdom from full benchmarks. [1] https://www.fftw.org/ https://www.fftw.org/