Actually, this holds only up to certain extent. Synchronization, locking, memory access patterns and context-switching from many threads with accompanied cache

Actually, this holds only up to certain extent. Synchronization, locking, memory access patterns and context-switching from many threads with accompanied cache thrashing can bring the effective parallelization to very small levels. I remember from benchmarking a high performance transactional messaging system in Java I was working on seeing how it hit a flat line at some point where it didn't scale at all due to CPU/memory contention (this depended on maximal number of threads allowed). We had to look at alternative designs such as 0MQ to see how they overcame it - basically they put one working/client thread per core to achieve their 1M messages/s. We were using 10s of thousands of threads in our middleware when necessary, and that was faster than async NIO in Java 6 at that time.