5 ms·

More cores, more RAM, more cache, more concurrency will help create higher total throughput but usually also mean higher latency because of increased context sw

by oso2k 8y ago

More cores, more RAM, more cache, more concurrency will help create higher total throughput but usually also mean higher latency because of increased context switching, natural need for more resource locks, increased delay in searching/reading larger stores like cache & RAM, and, higher latency penalties when there are cache misses and/or applications that require write-through policies. Things like faster clocks (which is why we now have dynamic, turbo clocks, aka Intel Turbo Boost), lock-free schedulers, and O(1) (constant-time) schedulers. However, notice that we have few CPU SKUs that go above 4GHz or 5GHz? Why don't we have 10GHz, 25GHz or 50GHz CPUs? It's because silly things like heat, cross-talk, speed of light, and physics at nano scale get in the way. Higher clocks would improve latency. Also, applications can improve their latency if they're written to take advantage of multiple cores, their workloads are inherently parallel or mostly parallel, manage their own thread/process scheduling, and use lock-free work stealing schedulers. CPU OEMs have proven developers prefer the manage all this complexity AND also calculate their ILP (instruction level parallelism) for them. Intel's Itanium failed because it required explicitly declared ILP (from compiler and/or developer) which can be tricky especially when most CS literature/education materials, programming languages, software constrains itself to using simplistic single thread of execution mental/computer models.

speedplane 8y ago
All the concerns in your comment are real, but they can be boiled down to one simple fact: Moore's law is dead. Our processors are now not advancing nearly as quickly as people can create solutions for them. Case in point - Virtual Reality, which has technically been on the market for years, but out of reach for most users. If processors are not improving, everything else slows down. Graphics, games, and the internet ten years from now will look pretty much the same as it does today.
- CodeWriter23 8y ago
  If we had an operating system where every system call used an async calling interface, the type of spinlock latency observed in this article would be greatly reduced. It wouldn’t bring Moore’s Law back but it would make the most of what CPU resource we have.
  - bennofs 8y ago
    Wasn't the lock in the article kind of async? From my understanding, the problem was not the wait time on the lock alone, but that the lock is not fair: so even if you asynchrounously locked, someone else (in this case the WMI scan) can get the lock everytime before you get it so you are blocked a long time (you cannot continue with this particular operation until you get the lock, even in the async case).
    - CodeWriter23 8y ago
      In the article, they illustrate a poor design choice. Or probably more likely, a continuation of the programming practice utilized elsewhere in the kernel. That being, slap a lock on any resource contention. That practice is necessitated solely by the need to support a linear calling interface. Even if this particular issue were to be fixed through queuing instead of locking, the existence of locks elsewhere will still become an issue in a given use case. My point is, by design, an operating system that has a fully async interface can dispense with locks altogether. Resource contention is then properly resolved through queuing. The OS can then strictly enforce a priority scheme on its various queues, preventing an opportunistic thread from dominating the system. Or enabling higher priority requests to be serviced first. But that decision is made intelligently, not as a result of optimizing to minimize context switching.
      - speedplane 8y ago
        Don’t see the fundamental difference between locks and queues. In a system that uses locks, the kernel can still internally use queues to grant some lock requests before others.
        CodeWriter23 8y ago
        The fundamental difference is this. With locking, when a thread’s time slice expires while holding a lock, the dispatcher switches to a different thread and no other thread can gain that particular lock until the dispatcher resumes the thread holding the lock. No locks means this can never happen.
- exikyut 8y ago
  The internet, sure. But GPUs will figure out new ways to parallelize, which is not dead yet.
  - sgift 8y ago
    The problem is not that we cannot parallelize hardware. The problem is that the software making use of it is lacking. 24 cores is plenty, but if most software only uses one core even more cores won't help.
  - speedplane 8y ago
    GPUs can just do fast matrix multiplications in parallel, but it’s up to the developer to reframe their software into matrix multiplications. For some applications it’s great, but for most it isn’t realistic. Also if General processors are now hitting a wall, it’s only a matter of time before GPUs do.
- oso2k 8y ago
  This has nothing to do with Moore's Law or improving processors directly. If anything, this is a software optimization problem. Half of what I mentioned was locks and schedulers, which are directly software problems. Layer on the fact that a helloworld application in most languages and frameworks is 100s of KB to dozens of MBs, and you have really, really, unoptimized software. 30 years ago, lots of software was written in Assembly and some C. But software has gotten fatter and slower at a much much faster rate than hardware has gotten faster/more parallel since the industry introduced ILP, caches, multi-stage instruction pipelines, etc. with the Pentium/5x86 class microcomputers & processors. Most people never learn to write multi-processing, multi-threaded, multi-programming, event-driven, decentralized or distributed applications. The model that browsers and most languages & frameworks provide is usually inherently sequential and single processing. If web developers never learn to write multi-processing, multi-threaded, multi-programming, event-driven, decentralized or distributed applications the web will forever look like it looks today.
  - speedplane 8y ago
    > This has nothing to do with Moore's Law or improving processors directly. If anything, this is a software optimization problem. This is such a contradictory sentence. The whole point of Moore's law was that software developers could do what they want, freed by limitations of hardware because in 18 months, when their product was ready for launch, the hardware became capable enough to handle it. Forcing developers to focus more and more on performance (via threading, distributed systems, and custom die co-processors), rather than innovating new applications, is a huge shift. It's like the hardware developers gave up, and tell the software developers that it's in your hands now. Also, comparing a hello world program 10 years ago to today is a meaningless comparison because no one uses hello world to do anything. A better example would be Excel in 1998 and Excel in 2018. Sure today, it's a 500 megabytes instead of a just a handful, but it does quite a bit more (for better or worse).
    - oso2k 8y ago
      You don't understand Moore's Law. Moore's Law says nothing about software devs doing what they want or processor speed or even hardware capabilities or qualities. It's about the manufacturing capability and integration by semiconductor fabs. Specifically, "Moore's law is the observation that the number of transistors in a dense integrated circuit doubles about every two years.' [0]. Hardware engineers never gave up; they're the reason your bloated apps takes 60 secs to startup instead of 3600 secs. [0] https://en.wikipedia.org/wiki/Moore%27s_law https://en.wikipedia.org/wiki/Moore%27s_law

by oso2k 8y ago

speedplane 8y ago

All the concerns in your comment are real, but they can be boiled down to one simple fact: Moore's law is dead. Our processors are now not advancing nearly as quickly as people can create solutions for them. Case in point - Virtual Reality, which has technically been on the market for years, but out of reach for most users. If processors are not improving, everything else slows down. Graphics, games, and the internet ten years from now will look pretty much the same as it does today.

CodeWriter23 8y ago
If we had an operating system where every system call used an async calling interface, the type of spinlock latency observed in this article would be greatly reduced. It wouldn’t bring Moore’s Law back but it would make the most of what CPU resource we have.
- bennofs 8y ago
  Wasn't the lock in the article kind of async? From my understanding, the problem was not the wait time on the lock alone, but that the lock is not fair: so even if you asynchrounously locked, someone else (in this case the WMI scan) can get the lock everytime before you get it so you are blocked a long time (you cannot continue with this particular operation until you get the lock, even in the async case).
  - CodeWriter23 8y ago
    In the article, they illustrate a poor design choice. Or probably more likely, a continuation of the programming practice utilized elsewhere in the kernel. That being, slap a lock on any resource contention. That practice is necessitated solely by the need to support a linear calling interface. Even if this particular issue were to be fixed through queuing instead of locking, the existence of locks elsewhere will still become an issue in a given use case. My point is, by design, an operating system that has a fully async interface can dispense with locks altogether. Resource contention is then properly resolved through queuing. The OS can then strictly enforce a priority scheme on its various queues, preventing an opportunistic thread from dominating the system. Or enabling higher priority requests to be serviced first. But that decision is made intelligently, not as a result of optimizing to minimize context switching.
    - speedplane 8y ago
      Don’t see the fundamental difference between locks and queues. In a system that uses locks, the kernel can still internally use queues to grant some lock requests before others.
      - CodeWriter23 8y ago
        The fundamental difference is this. With locking, when a thread’s time slice expires while holding a lock, the dispatcher switches to a different thread and no other thread can gain that particular lock until the dispatcher resumes the thread holding the lock. No locks means this can never happen.
exikyut 8y ago
The internet, sure. But GPUs will figure out new ways to parallelize, which is not dead yet.
- sgift 8y ago
  The problem is not that we cannot parallelize hardware. The problem is that the software making use of it is lacking. 24 cores is plenty, but if most software only uses one core even more cores won't help.
- speedplane 8y ago
  GPUs can just do fast matrix multiplications in parallel, but it’s up to the developer to reframe their software into matrix multiplications. For some applications it’s great, but for most it isn’t realistic. Also if General processors are now hitting a wall, it’s only a matter of time before GPUs do.
oso2k 8y ago
This has nothing to do with Moore's Law or improving processors directly. If anything, this is a software optimization problem. Half of what I mentioned was locks and schedulers, which are directly software problems. Layer on the fact that a helloworld application in most languages and frameworks is 100s of KB to dozens of MBs, and you have really, really, unoptimized software. 30 years ago, lots of software was written in Assembly and some C. But software has gotten fatter and slower at a much much faster rate than hardware has gotten faster/more parallel since the industry introduced ILP, caches, multi-stage instruction pipelines, etc. with the Pentium/5x86 class microcomputers & processors. Most people never learn to write multi-processing, multi-threaded, multi-programming, event-driven, decentralized or distributed applications. The model that browsers and most languages & frameworks provide is usually inherently sequential and single processing. If web developers never learn to write multi-processing, multi-threaded, multi-programming, event-driven, decentralized or distributed applications the web will forever look like it looks today.
- speedplane 8y ago
  > This has nothing to do with Moore's Law or improving processors directly. If anything, this is a software optimization problem. This is such a contradictory sentence. The whole point of Moore's law was that software developers could do what they want, freed by limitations of hardware because in 18 months, when their product was ready for launch, the hardware became capable enough to handle it. Forcing developers to focus more and more on performance (via threading, distributed systems, and custom die co-processors), rather than innovating new applications, is a huge shift. It's like the hardware developers gave up, and tell the software developers that it's in your hands now. Also, comparing a hello world program 10 years ago to today is a meaningless comparison because no one uses hello world to do anything. A better example would be Excel in 1998 and Excel in 2018. Sure today, it's a 500 megabytes instead of a just a handful, but it does quite a bit more (for better or worse).
  - oso2k 8y ago
    You don't understand Moore's Law. Moore's Law says nothing about software devs doing what they want or processor speed or even hardware capabilities or qualities. It's about the manufacturing capability and integration by semiconductor fabs. Specifically, "Moore's law is the observation that the number of transistors in a dense integrated circuit doubles about every two years.' [0]. Hardware engineers never gave up; they're the reason your bloated apps takes 60 secs to startup instead of 3600 secs. [0] https://en.wikipedia.org/wiki/Moore%27s_law https://en.wikipedia.org/wiki/Moore%27s_law