3 ms·

> The hot paths in most important high-end computing are mostly quite efficient. it’s hard to determine the lower bound on a lot of these things. in scientific

> The hot paths in most important high-end computing are mostly quite efficient. it’s hard to determine the lower bound on a lot of these things. in scientific computing (e.g. physics sims) i still run across a lot of runtime branching that could easily be compiled out if we were using languages that had stronger type systems/templates. we use linear approximations for a lot of things, but it’s been shown that smarter interpolation can let you decrease your resolution without losing accuracy/stability. there’s an easy 2x or more perf gain every time you do that successfully. there’s also the boundary conditions: we always simulate more volume of space than we’re actually interested in so as to avoid certain types of distortion (reflections from the boundary of the simulation). over time we’ve learned tricks for reducing those reflections: we’ve applied a lot of general-purpose optimizations, but there’s evidence that we can go further if we encode more simulation-specific information at these boundaries. this whole area has just been slow, steady, compounded optimizations. i don’t work in ML, but i find it very unlikely that (a) optimizing neural networks is a completely solved problem or (b) that you all know how to train them using the fewest iterations. you make sweeping statements that just aren't obvious to me.

lrem 4y ago
You seem to be arguing against "optimal", while quoting parent's "mostly quite efficient". The reality is that while everything is converging to a locally optimal point on the tradeoff curve, a _lot_ of software out there runs in the "hardware is my driving cost" locale. This makes companies squeeze epsilons out of their servers. There's also the "selling hardware is my profit centre", which end-users get to pay for.
- colinsane 4y ago
  the comment which spawned this chain is: > If push came to shove we might just consider making better software? as you say, "a _lot_ of software out there runs in the 'hardware is my driving cost' locale." if something changed -- say, we lost access to high-end fabs -- then my argument is that we have room to focus higher on the stack to make this far less impactful than it would otherwise be. in most places, software is not "optimal", as you seem to agree. in very few places have we even calculated a bound of what optimal _is_. the HPC example was to highlight the latter part: we shift from something like a linear gradient descent to some different descent algorithm and it works better in practice. ML training techniques are always changing to get faster convergence. very, very few of these HPC areas have actual, well-defined performance bounds. it's just a tradeoff of "how much do we invest in algorithms" v.s. "how much do we invest in hardware". i read GP as suggesting otherwise, based on the content of the comment they were replying to. i'm not even sure there's a disagreement between you and i. maybe not even between me and sneak. i kinda wonder if all of us here just have a different number in mind for what qualifies as "mostly quite efficient". i'm completely confident we have at least 2x perf gains sitting around in just about any computational area that's presently dominated by hardware costs (e.g. HPC, but not e.g. packet-switching/networking). 10x perf gains in many of them: not by micro-optimizing things like memory allocations, but by being inventive with algorithms. that kind of gain is meaningful to me in this area: enough that i wouldn't dismiss it as "mostly quite efficient" as a rebuttal against that original "if push came to show we might just consider making better software".

lrem 4y ago

You seem to be arguing against "optimal", while quoting parent's "mostly quite efficient". The reality is that while everything is converging to a locally optimal point on the tradeoff curve, a _lot_ of software out there runs in the "hardware is my driving cost" locale. This makes companies squeeze epsilons out of their servers. There's also the "selling hardware is my profit centre", which end-users get to pay for.

colinsane 4y ago
the comment which spawned this chain is: > If push came to shove we might just consider making better software? as you say, "a _lot_ of software out there runs in the 'hardware is my driving cost' locale." if something changed -- say, we lost access to high-end fabs -- then my argument is that we have room to focus higher on the stack to make this far less impactful than it would otherwise be. in most places, software is not "optimal", as you seem to agree. in very few places have we even calculated a bound of what optimal _is_. the HPC example was to highlight the latter part: we shift from something like a linear gradient descent to some different descent algorithm and it works better in practice. ML training techniques are always changing to get faster convergence. very, very few of these HPC areas have actual, well-defined performance bounds. it's just a tradeoff of "how much do we invest in algorithms" v.s. "how much do we invest in hardware". i read GP as suggesting otherwise, based on the content of the comment they were replying to. i'm not even sure there's a disagreement between you and i. maybe not even between me and sneak. i kinda wonder if all of us here just have a different number in mind for what qualifies as "mostly quite efficient". i'm completely confident we have at least 2x perf gains sitting around in just about any computational area that's presently dominated by hardware costs (e.g. HPC, but not e.g. packet-switching/networking). 10x perf gains in many of them: not by micro-optimizing things like memory allocations, but by being inventive with algorithms. that kind of gain is meaningful to me in this area: enough that i wouldn't dismiss it as "mostly quite efficient" as a rebuttal against that original "if push came to show we might just consider making better software".