3 ms·
> Don’t Parallelize when it is already Parallelized For the uninitiated -- there's probably a large list of "don'ts" that would be good here, too. But maybe a
by wyldfire 10y ago
> Don’t Parallelize when it is already Parallelized
For the uninitiated -- there's probably a large list of "don'ts" that would be good here, too. But maybe a prerequisite is ideal: why are you trying to write [this code] to execute in parallel? "It's slow" isn't good enough. You can save yourself a lot of time and headache if you use profilers to understand what the system is doing that makes it slow.
Using that profiler will at least help partition the problem into one of two big domains: compute resource bound (cpu/bus/mem/cache etc) or maybe it's just pending results from some other async task in the system. The latter happens much more often than you might expect. Filesystems, critical locks/exclusion, networks, databases, IPC -- there's lots of stuff that having multiple tasks in parallel for might not help much.
- stcredzero 10y agoThere were times/places when/where programmers in the boonies of the US had funny ideas about pre-optimization and dynamic languages. (Specifically, Ohio!) Basically meetup hecklers who had the weird idea that you should write super-hacky-optimized code 100% of the time, or it would never be fast enough. So then I asked the room to raise their hand if they had ever used a profiler on a real production system. Then I asked them to leave their hand up if they'd never ever been surprised by what the profiler told them!
- jackmott 10y agoSo the lesson is.... Write super hacky optimized code 100% of the time, but constantly profile it too!
- stcredzero 10y agoThe lesson is: If you have a TARDIS and you find yourself in Ohio in the late 90's, don't bother going to a programmer's meetup.
- jsmith0295 10y agoNow we have the opposite problem, at least in my experience. Everyone just says "premature optimization is the root of all evil" and then uses that as an excuse not to have a sane architecture just to save a little effort up front. After all, that would slow down the rate at which they could complete user stories for the week...
- stcredzero 10y agoYet another swing of the pendulum.
- joshvm 10y agoSome other bits: - Always profile first! Then go for low-hanging single-thread fruit. If you also parallelise later you'll get a multiplicative effect with speedup. - Use task manager/activity monitor/top frequently to check what CPU usage your program is actually getting. If your code is using N * 99% (for N cores) continuously then you're doing fine. You should be suspicious if your usage has spikes, implying e.g. an IO bottleneck somewhere. - Rule of thumb: parallelisation alone will at best give you a linear speedup. Super-linear speedup is possible, but only for very specific problems like linear algebra and even then it's usually some weird memory trick like you get more stuff happening in L1 vs RAM. - Build some intuition for how fast your code ought to run. I've chased down many bottlenecks only to realise that the thing I thought was the culprit was actually extremely fast (I'm sure we've all done this). http://norvig.com/21-days.html#answers http://norvig.com/21-days.html#answers , but also make your own measurements of time required to save files, modify images, zip things, etc. You can catch this definitively by profiling, of course, but it's nice to be able to guess where to start prodding. - On the flipside, learn to recognise when things are embarrassingly parallel and act accordingly. Here a profiler won't help you. Sometimes you're just lucky and you can get linear gains for free. - Think about what level you want to parallelise. If you know that your single threaded code is as good as it gets, can you get away with running multiple instances of it? Often it's far simpler to write a wrapper than explicitly paralellising low-level code.