7 ms·
How simultaneous multithreading works under the hood
- bee_rider 2y agoIt seems a bit high-level, kind of skimming over a bunch of architecture concepts with a couple references to the fact that this might be duplicated when hyperthreading, this might not… IMO a blog post should be more actionable. This isn’t a textbook chapter. For example we go through the frontend. When discussing the trace cache we have: > … Instruction decoding is an expensive operation and some instructions need to be executed frequently. Having this cache helps the processor cut down the instruction execution latency. … > Trace cache is shared dynamically between the two logical processors on an as needed basis. … > Each entry in the cache is tagged with the thread information to distinguish the instructions of the two threads. The access to the trace cache is arbitrated between the two logical processors each cycle. So the threads share a trace cache, but keep track of which hyperthread used which instructions—but we don’t really know, practically, if we prefer threads that are running very similar computations or if that is a non-issue (that is, does the fact that they share the trace cache mean one thread can benefit from things the other has cached? Or does the tagging keep them separated?). In general, often they say “this is split equally between the two threads” or “this is shared,” which makes me wonder “if I disable SMT does the now single-thread get access to twice as much of this resource, and are there cases where that matters.” This is somewhat covered in: > As we have seen, enabling SMT on a CPU core requires sharing many of the buffers and execution resources between the two logical processors. Even if there is only one thread running on an SMT enabled core, these resources remain unavailable to that thread which reduces its potential performance. But this seems a bit fuzzy to me, I mean, we talk about caches which are shared dynamically between the two threads so at least some resources will be more readily available if only a single thread is running. It also could be interesting—if the author is an expert, perhaps they could share their experience as to which pipeline stages are often bottlenecks that get tighter with hyperthreads on, and which aren’t? We have a sort of even focus on each stage without many hints as to which practically matter. Or how we can help them out. Also it is largely based on a 2002 whitepaper so I guess the specific pipeline stages must have evolved a bit since then. Or maybe they could share some battle stories, favorite tools, some examples of applications and why they put pressure on particular stages, things which surprisingly didn’t scale when hyperthreads were enabled (I’m not asking for all these things, just any would be good).
- mhh__ 2y agoGood summary overall, although seemed a little muddled in places. Would love to know some of the tricks of the trade from insiders (not relating to security at least)
- behnamoh 2y agoTangent: Ever since I became familiar with Erlang and the impressive BEAM, any other async method seems subpar and contrived, and that includes Python, Go, Rust, etc. It's just weird how there's a correct way to do async and parallelism (which erlang does) and literally no other language does it.
- lawn 2y agoI'm a huge fan of the BEAM, but I wonder if you're not overselling it a little? Surely there are trade-offs here that sometimes aren't worth it and alternative ways are better? For example, the BEAM isn't optimized for throughput and if that's a high priority for you then you might want to choose something else (maybe Rust).
- behnamoh 2y ago> For example, the BEAM isn't optimized for throughput... weird take, given that Erlang powers telecom systems with literally millions of connections at a time.
- deagle50 2y agothat's not throughput
- jjtheblunt 2y agobut the traffic on each connection doesn't need bandwidth in the throughput sense mentioned
- JackSlateur 2y agoI worked in telecoms, no erlang found. Perhaps we were too modern ?
- immibis 2y agoI worked in telecoms. We had a third party component written in Erlang. I'd say it was about the second most reliable component of the system. It was susceptible to some memory leaks, but they usually turned out to be caused by misuse of the C client library. The most reliable component was written in C, with what must have been a space shuttle level of effort behind it. No memory allocation, except in code paths that can return an error to the user who asked for something that caused the system to need more memory (we probably got this wrong on our side of the API, and didn't test out of memory scenarios, and they probably would have just resulted in OOM kills anyway). Every line of code commented, sometimes leading to the infamous "//add 1 to i" but most times showing deep design thought. State machines documented with a paragraph for every state transition explaining non-obvious concerns.
- pavlov 2y agoIntel’s next generation Arrow Lake CPUs are supposed to remove hyperthreading (i.e. SMT) completely. The performance gains were always heavily application-dependent, so maybe it’s better to simplify. Here’s a recent discussion of when and where it makes sense: https://news.ycombinator.com/item?id=39097124 https://news.ycombinator.com/item?id=39097124
- PaulKeeble 2y agoMost programs end up with some limitation on the number of threads they can reasonably used. When you have a lot less Cores than that SMT makes a lot of sense to better utilise the resources of the CPU. However once you get to the point where you have enough cores SMT no longer makes any sense. I am not convinced we are necessarily there yet but the P/E cores Intel are using are an alternative towards a similar goal and makes a lot of sense on the desktop given how many workloads are single/low threaded. I can see the value in not having to deal with SMT and E core distinctions in application optimisation. AMD on the other hand intends to keep mostly homogenous cores for now and continue to use SMT. I doubt its going to be simple to work out which strategy in practice is the best, its going to vary widely by application.
- variadix 2y agoIt is my understanding that SMT should be beneficial regardless of core count, as SMT should enable two threads that can stall waiting for memory fetches to fully utilize a single ALU, i.e. SMT improves ALU utilization in memory bound applications with multiple threads by interleaving ALU usage when each thread is waiting on memory. Maybe larger caches are reducing the benefits of SMT, but it should be beneficial as long as there are many threads who are generally bound by memory latency.
- t-3 2y ago> Maybe larger caches are reducing the benefits of SMT, but it should be beneficial as long as there are many threads who are generally bound by memory latency. I thought the reason SMT sometimes resulted in lower performance was that it halved the available cache per thread though - shouldn't larger caches make SMT more effective?
- deleted 2y ago[deleted]
- shaggie76 2y agoA grossly over-simplified argument for SMT that resonated with me was that it could keep a precious ALU busy while a thread stalls on a cache miss. I gather in the early days the LPDDR used on laptops was slower too and since cores were scarce so this was more valuable there. Lately, though, we often have more cores than we can scale with and the value is harder to appreciate. We even avoid scheduling work on a shared with an important thread to avoid cache-contention because we know the single-threaded performance will be the bottleneck. A while back I was testing Efficient/Performance cores and SMT cores for MT rendering with DirectX 12; on my i7-12700K I found no benefit to either: just using P-cores took about the same time to render a complex scene as P+SMT and P+E+SMT. It's not always a wash, though: on the Xbox Series X we found the same test marginally faster when we scheduled work for SMT too.
- bayindirh 2y agoRendering is one of the scenarios which was either same or slower with SMT since the beginning. This is because rendering is already math heavy, and your FPU is always active, esp. dividers (which is always the most expensive operation for processors). SMT shines while waiting for I/O or doing some simple integer stuff. If both your threads can saturate the FPU, SMT is generally slower because of the extra tagging added to the data inside the CPU to note what belongs where.
- hinkley 2y agoBut the way you make rendering embarrassingly parallel is the way you make web servers parallel; treat the system as a large number of discrete tasks with deadlines you work toward and avoid letting them interact with each other as much as possible. You don’t worry about how long it takes to render one frame of a digital movie, you worry about how many CPU hours it takes to render five minutes of the movie.
- bayindirh 2y agoYes, however in an SMT enabled processor, there are one physical FPU per two logical cores. FPU is already busy with other thread's work, so the threads in that SMT enabled core take turns for their computation in the FPU, creating a bottleneck here. As a result, you don't get any speed boost at best case, or lose some time in the worst case. Since SMT doesn't magically increase the number of FPUs available in a processor, if what you're doing is math heavy, SMT just doesn't help. Same is true for scientific simulation, too. I observed the same effect, and verified that indeed saturating the FPU with a thread makes SMT moot.
- jeffbee 2y agoOne of the biggest mistakes users have is a mental model of SMT that imagines the existence of one "real core" and one inferior one. The threads are coequal in all observable respects.
- MBCook 2y agoI suspect that’s a result of the performance. While both threads are capable of the same tasks, you don’t get the 2x performance you would with a “real” second thread, really a second core. So conceptually it’s a little more like having 1.25 single threaded cores, or whatever the ratio is for your application, if you only care about performance in the end.
- LoganDark 2y agoI mean, Intel's new CPUs certainly have both real cores ("P-cores") and inferior cores ("E-cores"). I suspect the reason they introduced E-cores is primarily thermal- and die-space-related, not actually power usage or performance. I always make sure to buy the chips without E-cores because they are better.
- jeffbee 2y agoThat's weird, I buy the one with the most e-cores. They're great for batch processes. For example the 14900K builds the Linux kernel faster than the Ryzen 9 7950X, even though it only has 8 "real cores" versus 16.
- anticensor 2y agoUntil it degrades...
- michaelt 2y agoIf you're compressing a video or a similar highly optimised compute-hungry task, your computer's fans are roaring like jet engines, and task manager says you're at 50% CPU utilisation - I can see how people formed that opinion.
- mgaunard 2y agoThe whole point of SMT is to maximize utilization of a superscalar execution engine. I wonder if that trend means people think superscalar is less important than it used to be.
- superjan 2y agoWhat I think is worth knowing is that compute units in GPU’s also use SMT, usually at a level of 7 to 10 threads per CU. This helps to hide latency.
- adrian_b 2y agoMost GPUs do not use SMT, but the predecessor of SMT, fine-grained multi-threading. In every clock cycle the instruction that is initiated is selected from the many threads that are available, depending on which of them need resources that are not busy. Most GPUs do not initiate multiple instructions per clock cycle (even if multiple instructions may proceed concurrently after initiation) or if they initiate multiple instructions per clock cycle those may have to belong to distinct classes of instructions, which use distinct execution resources, for example scalar instructions and vector instructions. SMT, i.e. simultaneous multi-threading, means that in every clock cycle many instructions are simultaneously initiated from all threads and then those instructions compete for the multiple execution units of a superscalar CPU, in order to keep busy as many of those execution units as possible. For each of the concurrent execution units, e.g. for each of the 6 integer adders available in the latest CPUs, a decision is made separately from the others about which instruction to be executed from the queues that hold instructions belonging to all simultaneous threads.
- gpderetta 2y agoAs far as I know hypertreads on intel share fetchers and decoders, so each clock cycle only one thread is feeding in instructions to the pipeline. That's no different to even for a simple barrel processor. It is true that once fetched an OoO CPU does a significant amount of scheduling and it is possible that in a given clock cycle instructions from both threads are getting fed to an execution unit. But I don't think that's the essence of SMT. For example the original larrabee is described as 4-way SMT, but as P5-derived it was a simple in-order design with very limited superscalar capabilities. I very much doubt that at any time instructions from more than one thread were at the execution stage.
- adrian_b 2y ago
- written-beyond 2y agoPoor AMD their bulldozer architecture got so much flak for not including SMT and now everyone's moving away from it. Yes yes I know bulldozer had a bunch more issues than just no SMT. It actually had the exact opposite with multiple cores sharing the same ALU or something like that. But still they could have been onto something if they had made it marginally more performant.
- RaftPeople 2y ago> for not including SMT and now everyone's moving away from it. The PowerXX architecture isn't moving away from it. Power10 currently supports SMT8 (8 threads per core) effectively and I can't imagine them moving away from it given all of the work they've done over the years to continue to evolve their design around SMT.
- Ahiyftv 2y ago[flagged]
- sweetjuly 2y ago> As we have seen, enabling SMT on a CPU core requires sharing many of the buffers and execution resources between the two logical processors. Even if there is only one thread running on an SMT enabled core, these resources remain unavailable to that thread which reduces its potential performance. This isn't true (anymore?). We've seen a variety of SMT cores which partition the ROB, fetch/decode bandwidth, etc. when running in SMT mode but allow full use when not in SMT mode.
- buildbot 2y agoThis is exactly how the x200 series Phi processors work, you get far more resources per thread in non-smt mode than the 4-way smt mode.
- leloctai 2y agoDoes anyone know how to search for this type of detailed technical articles? I've searched for this exact topic. As expected with kind of end user facing techs, the search results are always end user oriented article that explain nothing.
- layer8 2y agohttps://hn.algolia.com https://hn.algolia.com, assuming that most articles of that sort end up getting submitted to or mentioned on HN.
- stareatgoats 2y agoI find that LLMs with web access are a good fit for this kind of search, at least to point me in the right direction. The URLs provided are mostly hallucinations, however.
- trostaft 2y agoI don't know if google is somehow tracking the surge of interest in this article due to this HN post, but this very blog post is like search result 5 for me when looking up "how does simultaneous multi threading work" (results from a fresh firefox private tab on a different device, which I understand doesn't necessarily stop all tracking / caching but should be a fairly reasonable approx).
- spintin 2y ago[dead]
- snarfy 2y agoDisabling SMT gives about +5% fps in the games I play.
- krackers 2y agoAs a pre-read you should also go through https://www.lighterra.com/papers/modernmicroprocessors/ https://www.lighterra.com/papers/modernmicroprocessors/.
- Sohcahtoa82 2y agoReading about how some of these low-level CPU features work always blows my mind. Back in college, I took a class that I think was called Intro to Computer Hardware, but it really should have been called Intro to CPU Design. We learned how to turn logic gates into adders, latches, flip-flops, etc, and by the end of the course, we were able to design a very rudimentary processor at the gate level. How someone could create something like register renaming figuring how out-of-order execution is beyond me. They're not designing these at the gate level, are they? Or is there a language they use and a "compiler" that arranges the gates and transistors?
- zaptrem 2y agoI took the next level of that course and they taught up to SMT and a few other things. We did all of the homework assignments in a hardware description language called Verilog which allowed us to write abstractions for all the things.