Anyone studying Amdahl's law should also study the inverse, literally 1/(Amdahl's law), which leads to Gustafson's law. Its really strange what a simple recipr

Anyone studying Amdahl's law should also study the inverse, literally 1/(Amdahl's law), which leads to Gustafson's law. Its really strange what a simple reciprocal does to the entire equation. Instead of trying to cut your time down and perform work faster, Gustafson's law holds "time as constant" and measures the "amount of work done". Lo and behold: the parallel-portion of work grows the more-and-more work is available. That is to say, if you double-the-data to process, you can almost always double-the-parallelism in the same amount of time. ------------ So, you have some model (neural net?) that's difficult to parallelize? No problem, simple solution. Double, triple, quadruple the size of the model. Done. Suddenly more of it can be performed in parallel. Shrinking time (ie: Ahmdal's law) is an exercise in futility, and uselessness. It turns out that a lot of people's work days are measured in hours, and going from 30-seconds of wait time to 10-seconds of wait time is not really a crazy benefit. Instead, going from "Analyzed 2-GBs of data in 30-seconds" into "Analyzed 6GBs of data in 30-seconds" does lead to improvements. > The use of std::mutex (especially in frequently accessed structures) will bring your application to a grinding halt. Better solutions abound with concurrent sequential processes, or the Actor architecture. This is a separate issue. Contention is O(n), where n is the amount of contention. 1000 threads writing the same variable can only be done one-at-a-time. So it will take O(1000) time to perform all those reads/writes. It doesn't matter if you use mutexes, atomic-compare-and-swap, or anything. You can't "beat" the laws of physics. 1000 updates requires 1000x time. You "solve" this by splitting up your variables and minimizing contention. Instead of 1000x writes to 1x variable, you can perform 1000x writes to 1000x variables, then read the 1000x variables in parallel one step later.