15 ms·

Apple's M1 Ultra comes with a 32MB TLB bottleneck

287 points by retskrad 5y ago

SZJX 5y ago
In general this account (Vadim Yuryev/MaxTech) churns out a lot of specious and empty contents on YouTube at a ridiculous pace. One of the worst highly-subscribed "tech review" channels out there. Not that surprised to see this thread being subject to heavy discussion and inaccuracies being pointed out.
sounds 5y ago
https://threadreaderapp.com/thread/1514295682777059329.html https://threadreaderapp.com/thread/1514295682777059329.html The tweets refer to Tile-Based Deferred Rendering (TBDR) [1] as compared to Tile-Based Immediate Rendering (TBIR). Here's another take on it: https://docs.imgtec.com/Architecture_Guides/PowerVR_Architecture/topics/powervr_architecture_tile_based_deferred_rendering__tbdr.html https://docs.imgtec.com/Architecture_Guides/PowerVR_Architec... Note that TBIR appears to be a term the author just made up to refer to apps that have not be specifically optimized for the Apple M1 Ultra GPU architecture. [1] https://developer.apple.com/documentation/metal/resource_fundamentals/understanding_gpu_family_4 https://developer.apple.com/documentation/metal/resource_fun...
- ksec 5y ago
  >Note that TBIR appears to be a term the author... Tile-Based Immediate Rendering, or Tile-based immediate mode rendering (TBIM) is quite well known and not an invented term. [1] Given this is from MaxTech. I nearly stopped reading when he listed "they started working on the chips 5-7 years ago". But since he quoted hishnash, may be worth a read anyway. I also doubt the the so called GPU limitation is due to this. The GPU TDP usage depends on a lot of other factors. This TBDR limitation, if you can call it that has been well known since PowerVR / Dreamcast's days ( Someone please make a low cost Super-H Chip ). Most of the explanation seems very weird from a cache and memory access perspective. And as @lamchester pointed out later [2], a lot of it doesn't make any sense. And remember, M1 Ultra is two M1 Max. Which means there are many some small details missing in how it works together. [1] https://en.wikipedia.org/wiki/Tiled_rendering https://en.wikipedia.org/wiki/Tiled_rendering [2] https://twitter.com/lamchester/status/1514346083819786240 https://twitter.com/lamchester/status/1514346083819786240
  - vvanders 5y ago
    I've never heard it called TBIR, always TBDR or just tile based rendering. I honestly can't make heads or tails of the Twitter thread. It talks about having too many small reads/writes but that's the whole point of tile based rendering. You use the tile as scratch space to do operations that are expensive in system memory. This approach is well over a decade old at this point and pretty ubiquitous in power constrained rendering scenarios[1]. Best I can tell you need to take architecture into consideration when trying to get the peak GPU usage which has always been true of any platform regardless of tiled or immediate mode rendering. [1] https://developer.qualcomm.com/sites/default/files/docs/adreno-gpu/developer-guide/gpu/overview.html#tile-based-rendering https://developer.qualcomm.com/sites/default/files/docs/adre...
    - brigade 5y ago
      Accessing tile memory doesn't need an address translation; it's not part of virtual memory. Accessing global memory does. I think the claim is that the typical working set can exceed what the TLB can map once the GPU scales to 32 or more cores, so the TLB starts thrashing. Optimizing for TBDR would alleviate this because all the tile memory accesses would bypass the TLB, and also likely reduce the working set because the intermediate buffers don't need VM mapping.
      - vvanders 5y ago
        Yeah but a TLB is for user space virtual memory mapping. When your talking about GPU operations those usually go through opaque handles or allocated buffers and the DMA/read/write is done outside of user space(aside from initial data uploads or the case where you have unsynchronized user space memory buffers, but that's not usually the norm). Some of this changes a bit with Vulkan but it's not really clear from the thread how a TLB fits in. Optimizing for TBR involves reducing drawcalls, buffer target switches and operations that are orthogonal to what a TLB historically does.
        brigade 5y ago
        ...there are modern integrated GPUs that don't use virtual memory for all global resources? I expected that was a given to even just have an MMU enforce security when accessing RAM.
Damogran6 5y ago
Any time there's a massive change in architecture, it should be expected improvements will be found in tweaking.
musicale 5y ago
Good thing that we have Twitter since Apple doesn't know what they're doing, right? ;-) And it's too bad that apparently the page size can't be changed for larger memory systems to reduce TLB misses (at the expense of fragmentation) which is one traditional solution that doesn't require software changes. 16KB seems like a very small page size on a 64GB system. If we scaled an 8MB VAX with 512B pages up to 64GB it would have a page size of something like 4MB.
- jjtheblunt 5y ago
  why do you say the page size apparently can't be changed for larger memory setups?
- scottlamb 5y ago
  It sounds like it might be possible, at least at the hardware level. https://github.com/xmrig/xmrig/issues/2060#issuecomment-770291571 https://github.com/xmrig/xmrig/issues/2060#issuecomment-7702... (bug on some random crypto miner benchmark software, I guess) mentions a 32 MiB page size, although that might be with Linux on M1 rather than macOS on M1. This "32MB TLB bottleneck" is a weird thing to say. The TLB size is typically stated in terms of pages, right? with huge pages (or "superpages", as macOS may call them?), that should be a lot more than 32MB of total memory.
  - Sirened 5y ago
    It absolutely can! On ARM you can just enter a block entree in a non-leaf page table to allocate an entire block of physically contiguous memory in a single TLB entry. From the ARM docs [1], any CPU supporting 16K translation granules will support 32MB L2 blocks. I suspect, however, that apple can't really allocate that many (if any at all) 32MB regions of memory at runtime due to fragmentation unless they've substantially changed their contiguous memory allocator since I last looked. [1] https://developer.arm.com/documentation/den0024/a/The-Memory-Management-Unit/Translation-tables-in-ARMv8-A/Effect-of-granule-sizes-on-translation-tables https://developer.arm.com/documentation/den0024/a/The-Memory...
    - 5y ago
notreallyserio 5y ago
So the "design limitation" in the chip can be overcome by updating apps to support Apple's TBDR tile memory system that was presumably created to maximize processor performance? Am I understanding the design limitation correctly?
- throwmeariver1 5y ago
  It's bait for bad journalists.
- legalcorrection 5y ago
  Many people would call it a design flaw because it leaves a ton of the chip's performance on the table for common workloads in exchange for essentially nothing. TLB size is not where you're cutting costs. If you need to make it bigger so that common workloads can max out the other parts of the chip, it's a no-brainer decision. This isn't a sensible tradeoff.
  - Gigachad 5y ago
    How do you know it was for nothing? Do you have some insider info on the decision?
  - olliej 5y ago
    Do you have insight into the design trade offs made by the apple silicon teams? Because it sounds like Apple's silicon team disagree and took a different design approach to what you would do in your designs?
  - Dylan16807 5y ago
    But those common workloads might just have to tweak their allocator to request larger pages? That's not a very big demand for an architecture change.
  - onphonenow 5y ago
    This was an architecture change. The hoops one had to jump through going from intel to ARM were crazy in lots of other places. Or on gaming platforms (in the past). What I'm confused with is how many silicon engineers are so easily able to make the absolute determination that "This isn't a sensible tradeoff." The tweets terminology frankly seems weak, I've never even heard of a 32MB TLB - that's insanely huge.
dev_tty01 5y ago
Click bait. It is not a "design limitation," it is a different design choice. Of course highly performant code needs to be aware of the selected architecture. Nothing to see here.
- dang 5y ago
  Ok, I've replaced that interpretation with the underlying factual description (32MB TLB) and left the OP's word 'bottleneck' in there. Perhaps there's an interesting discussion to be had about it?
  - dev_tty01 5y ago
    Thank you. Fair enough, there may be an interesting discussion lurking here somewhere about GPU architectures and rendering modes. However, Apple has been using this approach since the A11, so it is not new information. The OP assertion that Apple helped the WOW devs is probably also specious, since the app may have already been optimized (or close to it) for the architecture if they had been following Apple's GPU/Metal programming advice for the last few years. Caches always have limits and abusing those limits often has punitive results.
- legalcorrection 5y ago
  That's just your opinion. This leaves a ton of performance on the table and saves Apple basically nothing in cost. Many of us would call that a design limitation, even a design flaw.
  - olliej 5y ago
    Where do you find that information? I couldn't find anything on apple.com where they detailed the various design tradeoffs they made working on their GPU
- throwaway0x7E6 5y ago
  right, I'm sure that people who write low-level libs will be thrilled to optimize them for 0.1% of the iGPU market
tambourine_man 5y ago
Apple uses a radically different architecture and apps need to be redesigned in order to take full advantage of the hardware. It's nothing new.
ggm 5y ago
can this be fixed by good toolchains or do people have to #ifdef their code around it?
- formerkrogemp 5y ago
  They probably have to #ifdef.
ushakov 5y ago
god why can’t he write a blog post? 5000 tweets in your face
- deleted 5y ago
  [deleted]
- LudwigNagasena 5y ago
  At least he could just take best practices from the Twitter CEO and post text as a picture.
- deleted 5y ago
  [deleted]
atq2119 5y ago
What does he mean by a 32MB TLB? Certainly there isn't an actual 32MB of on-chip memory dedicated to a TLB. So does it refer to the coverable amount of virtual memory? But that can't be true either, given huge pages. All in all, very disappointing thread. Does anybody have real references to what he's talking about?
- jazzyjackson 5y ago
  I don't know anything about TLBs but the M1 is a system on a chip, you can get up to 64GB of RAM shared between CPU and GPU all on the same die.
  - __init 5y ago
    More specifically, it's a system on a _package_. A TLB lives on the _die_, though, with one in each core.
  - heavyset_go 5y ago
    TLB cache isn't the same thing as the unified memory on the M1.
- kimixa 5y ago
  I think it's a confusion of similar terms - there's the TLB (translation lookaside buffer) of the MMU, which is an extremely small fast buffer of recent MMU translations - there's no chance that is anywhere near 32mb. This is often designed to be hit within a single cycle, so has tighter restrictions than even the l1 caches of modern CPUs. Then there's also the GPU tile buffer (which they may also be calling a "TLB" here?) - which I think is what he's referring to here - on a TBDR system there's an advantage if it can calculate multiple passes on the image while keeping all needed info in the on-chip tile buffer (Think stuff like depth and color data when rendering a large scene), as if the intermediate results aren't actually needed (such as a partially drawn scene where more objects will be drawn on top of some of the pixels, or an intermediate output before some post-processing pass) it may never actually touch the main memory so has the ability to save a lot of bandwidth. I'm pretty sure it's the second the OP is referring to here, as if you blow past this buffer size you can hit a performance cliff, and methods to optimise it's use (such as rendering tiles of multiple passes instead of doing each pass on the full screen view separately) doesn't really make any difference to immediate renderers, so is something "new" developers need to take into account for deferred tiler architectures.
jayd16 5y ago
<laughs in mobile developer> Seems a bit apocalyptic. Tile based architectures are everywhere in the mobile space. Is it really so inconceivable that macos devs can adjust to it?
flenserboy 5y ago
He focuses early on with overkill on the cooling — could it be that Apple knew this was a tradeoff, knew it would tend to not push the power envelope, but wanted a cooling system that would let them leave the general design unchanged for years to come, with lots of headroom for future cpus? Why call it an error when there is an actual reason that makes sense as an explanation?
- olliej 5y ago
  I also figured it had been designed so that it could be quiet - if you can keep the volume of the cooler large enough then you can use slower fans right?
- mshockwave 5y ago
  On one hand, I agree with you on the headroom since cooling has become far more complicate than just putting a fan so you really don't want to redesign over and over again. But on the other hand, from what I heard, the cooling on the Ultra edition is super heavy. So in that sense, it's indeed slightly overkill.
- bee_rider 5y ago
  An overkill cooling system could also be nice in a couple years when it is dusty and working at reduced efficiency.
- tomxor 5y ago
  Unlikely. Historically Apple has always designed cooling with very narrow margins around suppliers TDP due to internal pressure to create smaller and smaller machines, which has often resulted in underclocking due to GPU vendors downplaying their TDP requirements. From the explanation it sounds like an artificial limitation. The chip is technically capable of being saturated with the current design given an optimal program, and fulfilling it's TDP, at which point the current cooling system design would be _entirely necessary_ for the current chip design. This is clear from the second tweet: > Mac Studio cooling system is OVERKILL in most apps My emphasis added (not all apps in theory - but practically)
  - 5y ago
lowbloodsugar 5y ago
"If an application has not been optimized for the M1 GPU architecture's tile memory" Huh. What if it's written in crayon? Is the TLB a problem then?
Maursault 5y ago
Everyone calm down. Not every observation on Apple's stuff is earth-shattering. In fact... none of them are, but Apple is of course partly or mostly to blame for revving up everyone's engines. The M1 Ultra Studio is not what you think it is. It is not Apple's flagship high-performance machine. It isn't. That is still the Intel Mac Pro. The Studio is the Mac Mini update that was rumored 6 months before. The Studio is a Mac Mini, hey just gave it a new name. IOW the Studio is the low-end and entry level machine for professionals. Also, the current newest generation Apple hardware is not a revolution. Certainly, Apple deserves praise for seamless platform switching, nice work, truly. But these new Macs are only a little bit more performant than the previous generation. Check the benchmarks. Apple's best years were 2010-2012, where each year they doubled the performance of their machines of the previous year. It took another 6 years for performance to double again. The M1, M1 Pro, M1 Max, and M1 Ultra are all marginally faster than the previous generation, the Intel Macs. Except for 2010-2012, which was amazing, this slight increase in performance is entirely typical of new Macs compared to whatever model's previous iteration. This twitter guy is talking about GPU and games. I realize GPU is important to the computing industry, but not to me. I've been a Mac user since 1989, and even if it was and is included in the machine, I've never once used the GPU. It just sits idle there while I saturate the CPU. Macs are tools for professionals. Professionals don't play games with their tools. If Apple has somehow offended the gaming community by not catering to their needs, and also boxed the GPU performance in the M1 Ultra, it's really a win-win. Ok, it's a little embarrassing, but regardless, doesn't deserve the expected outrage from the sensationalist reporting. How dare Apple! Right? It's more like, ok, well, that sucks a little, but who gives a shit?
- Dylan16807 5y ago
  > Everyone calm down. I'd say the overall tone of comments is pretty skeptical.
- vimy 5y ago
  More and more (professional) apps use the gpu for compute. So does macOS. The days of using a gpu only for games or even graphical apps are long over. I’m pretty sure yours isn’t as idle as you think it is.
Epiphany21 5y ago
Everyone in this comment section is saying "it's nothing new" as if that excuses the issue. From the way it's described this is a software optimization problem. So the question is, how hard is it to optimize an existing application? If it's a few quick changes and a rebuild, then it's nothing. If it's anything more you'll have an Itanium-like product flop. Maybe. It's also possible that node size and computational headroom relative to the software of the day will make it hard for consumers to notice or care. Either way this will be interesting to watch.
- scottlamb 5y ago
  > So the question is, how hard is it to optimize an existing application? TBD how hard it is to optimize an existing application to remove this bottleneck. The tweets seem to suggest it's pretty involved. Alternatively, there might be a simple way to use a larger page size (called "huge pages" on Linux, possibly "superpages" on macOS), which would greatly relieve the TLB as a bottleneck, if TLB = translation lookaside buffer. The tweets seem confused enough that I'm not sure. > If it's a few quick changes and a rebuild, then it's nothing. If it's anything more you'll have an Itanium-like product flop. Meh, if it means many applications only reach 86W rather than 105W, that's probably not world-ending. Most applications on any platform have some bottleneck that they could avoid with more optimization effort, whether it's the TLB or something else. Users usually don't complain in spite of these bottlenecks. And: > It's also possible that node size and computational headroom relative to the software of the day will make it hard for consumers to notice or care. Yeah, people seem pretty happy with these processors.
loser777 5y ago
"Unlike a CPU, when a GPU is waiting for data, it can't just switch to work on something else." https://twitter.com/VadimYuryev/status/1514295693581586434 https://twitter.com/VadimYuryev/status/1514295693581586434 Is this meant to be qualified with (Apple) GPU? Otherwise it sounds like the literal definition of latency hiding that has been the norm on desktop GPUs (and one of the first things taught to newcomers of the OpenCL/CUDA programming model) for a while.
- rayiner 5y ago
  He says the entire thread group needs to be stalled. The thread group is (on M1) 32 SIMD lanes that have to execute basically the same control flow. Presumably other thread groups can continue executing.
- wyldfire 5y ago
  I guess the point here is that GPUs don't have general purpose tasks doing independent work and an OS scheduler. You might end up masking part of the cost of some stalls if you are able to swap in other ready-to-run tasks. The bus latency you're describing doesn't exist when you don't need to copy to a GPU-dedicated GDDR/HBM bank. But while that problem goes away, it sounds like this new one of TLB pressure shows up. I'm no expert but I suspect the mobile SoCs like Bionic and Snapdragon have the same concept as the M1 Ultra with respect to integrated GPU sharing memory with the apps cores. M1 probably inherited it from Bionic? So some of this work of porting compute software to reflect this environment may have already started. I guess the challenge is that the bar is higher for expectation of GPU performance in a desktop system like the Mac Studio.
  - dragontamer 5y ago
    I don't know much about the M1. But I can say for sure that Intel iGPUs, AMD GCN, AMD RDNA, AMD CDNA, and multiple NVidia-generations of GPUs all have hyperthread-like rescheduling of independent workgroups. In fact, something like 8x wavefronts / warps run in parallel on modern GPUs. When one wavefront / warp stalls due to a memory read/write (or a PCIe read/write), the GPUs universally "hyperthread-out" and hide the latency. Its "different" from how CPUs do it, but the fundamental principals are the same. (CPUs have a redundant set of registers tracked in a register file. GPUs on the other hand, have a set of registers and the kernel-scheduler (or whatever handles CUDAstreams) carefully assigns those registers to not conflict with any running wavefronts). ------- The statement so listed is blatantly false, at least for Intel, AMD, and NVidia GPUs. Maybe Apple iGPUs are built different, but I find that unlikely.
t8y 5y ago
If you just use regular vertex and fragment shaders then all of that is automatically handled. A game like World of Warcraft might not use advanced GPU features that would slow down a TBDR.
usaphp 5y ago
> "The M1 family of chips, including the M1 Ultra, has a major limitation that can't be fixed unless apps are properly optimized." So am I understanding it correctly - if an app is properly optimized - then it's not even an issue?
monocasa 5y ago
I think we're missing some info. 32MB of coverage is the same size for the TLB on a SM for a 1080ti. https://dl.acm.org/doi/10.1145/3491218 https://dl.acm.org/doi/10.1145/3491218 I don't think that's gone up a lot since then. And GPUs in general are totally fine with stalling on memory accesses. That's core to how they work, and why you run into issues treating them like CPUs. They approach the problem of the memory hierarchy at a base level differently by accepting the fact that thread groups will stall on pretty much every memory access, and just barrel scheduling enough thread groups that you can keep ALUs saturated despite the memory dependencies due to massive parallelism. Cache and TLB misses aren't that big of a deal if you have enough work to cycle through. If you don't have massively parallel amounts of work, and can't just keep cycling through contexts, this is the main reason why some algorithms don't work on GPUs, but that's a global problem to how all major GPUs approach the problem of memory that's hundreds of cycles away. > Hishnash: "What apps should be doing is loading as much data as possible into the tile mem and flushing it out in large chunks when needed. I bet a lot of the writes (over 95%) are for temporary values that could've been stored in tile mem and never needed to be written at all." I mean, you shouldn't be performing large amounts of writes that aren't a completion of a program on just about any GPU architecture. If you care about the results, heavy random writes screws you just as hard in a traditional GPU architecture too. You really want to be able to look at the data flow and describe it in almost pure functional terms. If you can't look at a program and say "it writes here at this offset or offsets", you're going to have a bad time. And finally, there's all sorts of ways you can crank up the GPU without doing writes at all, so the inability to turn it into a space heater just because of TLB pressure doesn't make a lot of sense. The xor chains that something like the bogomips calculator uses will eat up every cycle you give it and make their way through the compiler and hardware schedulers all without blocking on memory at all.
dragontamer 5y ago
This sequence of posts is written very strangely and imprecisely? You'd almost never say "32MB of TLB". You'd say maybe, 8192 entries of TLB, each with 4kB pages, for instance. (Others here in Hacker News suggest that the M1 uses a 16kB page, so maybe 2048 entries of 16kB each?) The Twitter posts are talking about temperature at first, tile-memory. I admit that I'm ignorant to tile-memory but... The traditional answer to TLB bottlenecks is to enable large-pages or huge-page support. So I'm already feeling a bit weird that large pages / huge pages haven't been discussed in the twitter thread. (Maybe its not the answer on the GPU side of things? But at least acknowledging how we'd solve it in CPU-land would help show off how features like huge-pages could help) Like, none of the logic here makes sense to me at all. Maybe something weird is going on in the Apple / M1 world, but I have a suspicious feeling that maybe the person posting this Twitter thread is at best... misstating things in a very confusing manner and using imprecise language. At worst, they might be fundamentally incorrect on some of these discussion points?
- eyelidlessness 5y ago
  I don’t code for GPUs but it seemed pretty straightforward to me: 1. General purpose GPU code assumes optimization characteristics where memory constraints are less restrictive for this use case 2. Without optimizing for this platform, typical code will not exercise the full potential of the chip The thread is discussing it as a design flaw; I don’t know if it is. But if normal “recompile for ____ target” in XCode or whatever isn’t enough for common workloads to be relatively close to optimal, and you have to do and know to do special work for the memory architecture… yeah that’s probably not ideal. And the other takeaway from the thread, that they’re likely to throw more memory hardware at the problem in the future, sounds right to me.
  - ngcc_hk 5y ago
    The key is apple code especially os and their tools. That is what m1 and future apple cpu and gpu are for. Hence optimisation should not be the issue. The whole thing is all about optimisation. The question is whether it affect other or 3rd party products (that cannot compete, or not shown to the public eg the private api ). And does apple care. Unless it shows is that limitation important ?
hedgehog 5y ago
Some of the main assertions about how common GPUs work are totally backwards: "when a GPU is waiting for data, it can't just switch to work on something else" - GPU designs specifically use the idea of rotating between threads to not block on memory access. "the concept that there is a local on-die memory pool you can read/write from with very very low perf impact is unthinkable in the current desktop GPU space" - Optimizing for the characteristics of the specific device's memory hierarchy including local/shared memory is a big part of GPU coding. If Apple follows the pattern that companies like NVIDIA have used they will keep the overall shape of the GPU similar between generations (in terms of how many threads, memory hierarchy, etc). This will give app devs relatively similar + stable tuning targets, we'll see high dollar apps get aggressively optimized for specific chips, and as that happens power/heat will go up. For an extreme example of this maturation curve look at how console games get more and more out of the same hardware. All that said the Metal tile shading API is a real thing Apple has been pushing for years on the mobile side, presumably because it fits the hardware and performs better, and now Macs use similar chips. No surprises there. And if an app is built around OpenGL and now needs to port to Metal to improve performance, sure, that could be a lot of work.
- SwordOfMyBone 5y ago
  Don't know anything about GPUs and am a CS undergrad. How does this apply to compute workloads? What's wrong with treating it like an IMR GPU when it comes to compute? is the tile buffer used for compute? ELI 5 please.
  - Maursault 5y ago
    > Don't know anything about GPUs and am a CS undergrad. You are the computer in "Computer Science," and the science is really just a lot of math. Maybe you wanted to study CE instead? I made the same damn mistake.
    - SwordOfMyBone 5y ago
      I unironically enjoy CS. I like programming and I like being close to the metal. I have an interest in the engineering part but I don't know if I'd pick it over CS. Life is hard. Also like compilers, and most compiler engineers are CS grads. That's my dream job!
gotaquestion 5y ago
I'm curious how they claimed that the cooling solution is over-designed because they were only able to achieve 80% of the spec'd power target (most likely TDP since people stopped using max power in the 00's). There are a number of factors that go into spec'ing this value, typically quite a few test vectors to determine typical workloads and high-power targeted workloads. A mistake in power budget of 20% is gigantic and would have significant impact on sorting die. Now maybe Apple doesn't care about this because they aren't a silicon vendor, but that budget comes out of battery life and cooling solution space. My point is, I find it highly unlikely Apple overstated their power target by 20% given how much money it would have saved them. But maybe they did?
- jhugo 5y ago
  Well, 20% headroom on a badly optimised workload that can't make full use of the GPU seems decent. Run some better code and you'll need that 20%. As covered in other comments, the thread author has no idea what they're writing about.
kazinator 5y ago
I'm confused. A 32 megabyte TLB is huge. TLB entries are tiny objects; and each one caches the mapping for an entire VM page. The larger the page size, the more "leverage" the TLB has. Maybe the blogger means that the TLB controls a VM working set or footprint of 32 MB?
vmception 5y ago
So it can only use 68w instead of 105w? Thats the crux of the complaint here? This honestly makes me want to throw out all my other computers and GPUs that do whatever they do at any performance metric simply because they use so much more energy to do so
gchadwick 5y ago
I note the only metric they offer is power usage. A poor substitute for actual performance numbers. In particular in any given CPU/GPU you could have all cores busily computing, nothing stalled and still be a good way below maximum power. To hit maximum power you need to carefully construct software that will fully utilize all functional units as well as causing maximum toggling within computation paths (1 + 2 will toggle fewer bits than 0xffffffff + 0xffffffff). The thread doesn't give any clear indication if their explanation that the GPU sees significant stalls due to waiting on TLB miss has actual data behind it or is pure conjecture based upon observed power usage.
ribit 5y ago
The author of those tweets doesn't seem to understand what TLB is (their terminology is off), doesn't seem to understand how GPUs work (they are explicitly designed to hide memory stalls by switching to a different kernel, which is why they have huge register files), doesn't seem to understand the memory hierarchies (why would a GPU have it's own TLB in the first place?). Besides, we know that M1 comes with a 3072-entry TLB that matches it's maximal 48MB cache size, so where did they come up with the "32MB figure anyway"? It is entirely possible that M1 Ultra is bandwidth starved for some workloads, but that can be demonstrated using GPU profiling tools (which Apple readily provides) and not by musing about technical concepts that one barely understands.
- monocasa 5y ago
  I agree with most of what you say but > why would a GPU have it's own TLB in the first place? is because modern GPUs have their own MMUs and have for quite a while. More or less ubiquitous MMUs on GPUs are half the reason why Mantle/Vulkan/DX12/Metal came into being.
  - ribit 5y ago
    Thanks for this! Can you elaborate a bit more and maybe give me some pointers where I can read about these things in more depth? I of course know about the MMU on traditional dGPUs but I was not aware that “integrated” GPUs also have a separate MMU. How does it work and why is it necessary given that the GPU shares the last level cache and the memory controller with the rest of the system?
    - monocasa 5y ago
      Because the TLBs and other MMU hardware aren't in the last level cache or memory controller on the vast majority of systems, they normally sit about at L1 on each core. This is because you want at least L2 to be speaking completely in physical addresses so the coherency protocol isn't confused by the same page mapped in different ways. L1 is more complex typically being virtually indexed, but physically tagged. So when a access is issued with the virtual address, in parallel to the cache set look up, the TLB translates to a physical address, and then the physical address is compared on the tags of the resulting cache line ways in the set that matched the virtual address in order to select the actual cache line for the op.
lvl102 5y ago
This guy knows nothing about computers. He is a Youtuber with zero expertise in anything but creating Youtube videos.
daneel_w 5y ago
I may have misunderstood things, but isn't the gist that software written correctly for this unique hardware isn't subject to the bottleneck? Is it really a hardware problem then, if it can't accommodate suboptimal software with maximum performance? This is after all not an Nvidia or AMD/ATI GPU architecture, and I wonder if anyone ever brought a similar argument forward in the many cases of this or that game having distinctly poor 3D performance on any of their hardware.
bdirgo 5y ago
Y’all just got nerd sniped.

287 points by retskrad 5y ago

SZJX 5y ago

In general this account (Vadim Yuryev/MaxTech) churns out a lot of specious and empty contents on YouTube at a ridiculous pace. One of the worst highly-subscribed "tech review" channels out there. Not that surprised to see this thread being subject to heavy discussion and inaccuracies being pointed out.

sounds 5y ago

https://threadreaderapp.com/thread/1514295682777059329.html https://threadreaderapp.com/thread/1514295682777059329.html The tweets refer to Tile-Based Deferred Rendering (TBDR) [1] as compared to Tile-Based Immediate Rendering (TBIR). Here's another take on it: https://docs.imgtec.com/Architecture_Guides/PowerVR_Architecture/topics/powervr_architecture_tile_based_deferred_rendering__tbdr.html https://docs.imgtec.com/Architecture_Guides/PowerVR_Architec... Note that TBIR appears to be a term the author just made up to refer to apps that have not be specifically optimized for the Apple M1 Ultra GPU architecture. [1] https://developer.apple.com/documentation/metal/resource_fundamentals/understanding_gpu_family_4 https://developer.apple.com/documentation/metal/resource_fun...

ksec 5y ago
>Note that TBIR appears to be a term the author... Tile-Based Immediate Rendering, or Tile-based immediate mode rendering (TBIM) is quite well known and not an invented term. [1] Given this is from MaxTech. I nearly stopped reading when he listed "they started working on the chips 5-7 years ago". But since he quoted hishnash, may be worth a read anyway. I also doubt the the so called GPU limitation is due to this. The GPU TDP usage depends on a lot of other factors. This TBDR limitation, if you can call it that has been well known since PowerVR / Dreamcast's days ( Someone please make a low cost Super-H Chip ). Most of the explanation seems very weird from a cache and memory access perspective. And as @lamchester pointed out later [2], a lot of it doesn't make any sense. And remember, M1 Ultra is two M1 Max. Which means there are many some small details missing in how it works together. [1] https://en.wikipedia.org/wiki/Tiled_rendering https://en.wikipedia.org/wiki/Tiled_rendering [2] https://twitter.com/lamchester/status/1514346083819786240 https://twitter.com/lamchester/status/1514346083819786240
- vvanders 5y ago
  I've never heard it called TBIR, always TBDR or just tile based rendering. I honestly can't make heads or tails of the Twitter thread. It talks about having too many small reads/writes but that's the whole point of tile based rendering. You use the tile as scratch space to do operations that are expensive in system memory. This approach is well over a decade old at this point and pretty ubiquitous in power constrained rendering scenarios[1]. Best I can tell you need to take architecture into consideration when trying to get the peak GPU usage which has always been true of any platform regardless of tiled or immediate mode rendering. [1] https://developer.qualcomm.com/sites/default/files/docs/adreno-gpu/developer-guide/gpu/overview.html#tile-based-rendering https://developer.qualcomm.com/sites/default/files/docs/adre...
  - brigade 5y ago
    Accessing tile memory doesn't need an address translation; it's not part of virtual memory. Accessing global memory does. I think the claim is that the typical working set can exceed what the TLB can map once the GPU scales to 32 or more cores, so the TLB starts thrashing. Optimizing for TBDR would alleviate this because all the tile memory accesses would bypass the TLB, and also likely reduce the working set because the intermediate buffers don't need VM mapping.
    - vvanders 5y ago
      Yeah but a TLB is for user space virtual memory mapping. When your talking about GPU operations those usually go through opaque handles or allocated buffers and the DMA/read/write is done outside of user space(aside from initial data uploads or the case where you have unsynchronized user space memory buffers, but that's not usually the norm). Some of this changes a bit with Vulkan but it's not really clear from the thread how a TLB fits in. Optimizing for TBR involves reducing drawcalls, buffer target switches and operations that are orthogonal to what a TLB historically does.
      - brigade 5y ago
        ...there are modern integrated GPUs that don't use virtual memory for all global resources? I expected that was a given to even just have an MMU enforce security when accessing RAM.

Damogran6 5y ago

Any time there's a massive change in architecture, it should be expected improvements will be found in tweaking.

musicale 5y ago

Good thing that we have Twitter since Apple doesn't know what they're doing, right? ;-) And it's too bad that apparently the page size can't be changed for larger memory systems to reduce TLB misses (at the expense of fragmentation) which is one traditional solution that doesn't require software changes. 16KB seems like a very small page size on a 64GB system. If we scaled an 8MB VAX with 512B pages up to 64GB it would have a page size of something like 4MB.

jjtheblunt 5y ago
why do you say the page size apparently can't be changed for larger memory setups?
scottlamb 5y ago
It sounds like it might be possible, at least at the hardware level. https://github.com/xmrig/xmrig/issues/2060#issuecomment-770291571 https://github.com/xmrig/xmrig/issues/2060#issuecomment-7702... (bug on some random crypto miner benchmark software, I guess) mentions a 32 MiB page size, although that might be with Linux on M1 rather than macOS on M1. This "32MB TLB bottleneck" is a weird thing to say. The TLB size is typically stated in terms of pages, right? with huge pages (or "superpages", as macOS may call them?), that should be a lot more than 32MB of total memory.
- Sirened 5y ago
  It absolutely can! On ARM you can just enter a block entree in a non-leaf page table to allocate an entire block of physically contiguous memory in a single TLB entry. From the ARM docs [1], any CPU supporting 16K translation granules will support 32MB L2 blocks. I suspect, however, that apple can't really allocate that many (if any at all) 32MB regions of memory at runtime due to fragmentation unless they've substantially changed their contiguous memory allocator since I last looked. [1] https://developer.arm.com/documentation/den0024/a/The-Memory-Management-Unit/Translation-tables-in-ARMv8-A/Effect-of-granule-sizes-on-translation-tables https://developer.arm.com/documentation/den0024/a/The-Memory...
  - 5y ago

notreallyserio 5y ago

So the "design limitation" in the chip can be overcome by updating apps to support Apple's TBDR tile memory system that was presumably created to maximize processor performance? Am I understanding the design limitation correctly?

throwmeariver1 5y ago
It's bait for bad journalists.
legalcorrection 5y ago
Many people would call it a design flaw because it leaves a ton of the chip's performance on the table for common workloads in exchange for essentially nothing. TLB size is not where you're cutting costs. If you need to make it bigger so that common workloads can max out the other parts of the chip, it's a no-brainer decision. This isn't a sensible tradeoff.
- Gigachad 5y ago
  How do you know it was for nothing? Do you have some insider info on the decision?
- olliej 5y ago
  Do you have insight into the design trade offs made by the apple silicon teams? Because it sounds like Apple's silicon team disagree and took a different design approach to what you would do in your designs?
- Dylan16807 5y ago
  But those common workloads might just have to tweak their allocator to request larger pages? That's not a very big demand for an architecture change.
- onphonenow 5y ago
  This was an architecture change. The hoops one had to jump through going from intel to ARM were crazy in lots of other places. Or on gaming platforms (in the past). What I'm confused with is how many silicon engineers are so easily able to make the absolute determination that "This isn't a sensible tradeoff." The tweets terminology frankly seems weak, I've never even heard of a 32MB TLB - that's insanely huge.

dev_tty01 5y ago

Click bait. It is not a "design limitation," it is a different design choice. Of course highly performant code needs to be aware of the selected architecture. Nothing to see here.

dang 5y ago
Ok, I've replaced that interpretation with the underlying factual description (32MB TLB) and left the OP's word 'bottleneck' in there. Perhaps there's an interesting discussion to be had about it?
- dev_tty01 5y ago
  Thank you. Fair enough, there may be an interesting discussion lurking here somewhere about GPU architectures and rendering modes. However, Apple has been using this approach since the A11, so it is not new information. The OP assertion that Apple helped the WOW devs is probably also specious, since the app may have already been optimized (or close to it) for the architecture if they had been following Apple's GPU/Metal programming advice for the last few years. Caches always have limits and abusing those limits often has punitive results.
legalcorrection 5y ago
That's just your opinion. This leaves a ton of performance on the table and saves Apple basically nothing in cost. Many of us would call that a design limitation, even a design flaw.
- olliej 5y ago
  Where do you find that information? I couldn't find anything on apple.com where they detailed the various design tradeoffs they made working on their GPU
throwaway0x7E6 5y ago
right, I'm sure that people who write low-level libs will be thrilled to optimize them for 0.1% of the iGPU market

tambourine_man 5y ago

Apple uses a radically different architecture and apps need to be redesigned in order to take full advantage of the hardware. It's nothing new.

ggm 5y ago

can this be fixed by good toolchains or do people have to #ifdef their code around it?

formerkrogemp 5y ago
They probably have to #ifdef.

ushakov 5y ago

god why can’t he write a blog post? 5000 tweets in your face

deleted 5y ago
[deleted]
LudwigNagasena 5y ago
At least he could just take best practices from the Twitter CEO and post text as a picture.
deleted 5y ago
[deleted]

atq2119 5y ago

What does he mean by a 32MB TLB? Certainly there isn't an actual 32MB of on-chip memory dedicated to a TLB. So does it refer to the coverable amount of virtual memory? But that can't be true either, given huge pages. All in all, very disappointing thread. Does anybody have real references to what he's talking about?

jazzyjackson 5y ago
I don't know anything about TLBs but the M1 is a system on a chip, you can get up to 64GB of RAM shared between CPU and GPU all on the same die.
- __init 5y ago
  More specifically, it's a system on a _package_. A TLB lives on the _die_, though, with one in each core.
- heavyset_go 5y ago
  TLB cache isn't the same thing as the unified memory on the M1.
kimixa 5y ago
I think it's a confusion of similar terms - there's the TLB (translation lookaside buffer) of the MMU, which is an extremely small fast buffer of recent MMU translations - there's no chance that is anywhere near 32mb. This is often designed to be hit within a single cycle, so has tighter restrictions than even the l1 caches of modern CPUs. Then there's also the GPU tile buffer (which they may also be calling a "TLB" here?) - which I think is what he's referring to here - on a TBDR system there's an advantage if it can calculate multiple passes on the image while keeping all needed info in the on-chip tile buffer (Think stuff like depth and color data when rendering a large scene), as if the intermediate results aren't actually needed (such as a partially drawn scene where more objects will be drawn on top of some of the pixels, or an intermediate output before some post-processing pass) it may never actually touch the main memory so has the ability to save a lot of bandwidth. I'm pretty sure it's the second the OP is referring to here, as if you blow past this buffer size you can hit a performance cliff, and methods to optimise it's use (such as rendering tiles of multiple passes instead of doing each pass on the full screen view separately) doesn't really make any difference to immediate renderers, so is something "new" developers need to take into account for deferred tiler architectures.

jayd16 5y ago

<laughs in mobile developer> Seems a bit apocalyptic. Tile based architectures are everywhere in the mobile space. Is it really so inconceivable that macos devs can adjust to it?

flenserboy 5y ago

He focuses early on with overkill on the cooling — could it be that Apple knew this was a tradeoff, knew it would tend to not push the power envelope, but wanted a cooling system that would let them leave the general design unchanged for years to come, with lots of headroom for future cpus? Why call it an error when there is an actual reason that makes sense as an explanation?

olliej 5y ago
I also figured it had been designed so that it could be quiet - if you can keep the volume of the cooler large enough then you can use slower fans right?
mshockwave 5y ago
On one hand, I agree with you on the headroom since cooling has become far more complicate than just putting a fan so you really don't want to redesign over and over again. But on the other hand, from what I heard, the cooling on the Ultra edition is super heavy. So in that sense, it's indeed slightly overkill.
bee_rider 5y ago
An overkill cooling system could also be nice in a couple years when it is dusty and working at reduced efficiency.
tomxor 5y ago
Unlikely. Historically Apple has always designed cooling with very narrow margins around suppliers TDP due to internal pressure to create smaller and smaller machines, which has often resulted in underclocking due to GPU vendors downplaying their TDP requirements. From the explanation it sounds like an artificial limitation. The chip is technically capable of being saturated with the current design given an optimal program, and fulfilling it's TDP, at which point the current cooling system design would be _entirely necessary_ for the current chip design. This is clear from the second tweet: > Mac Studio cooling system is OVERKILL in most apps My emphasis added (not all apps in theory - but practically)
- 5y ago

lowbloodsugar 5y ago

"If an application has not been optimized for the M1 GPU architecture's tile memory" Huh. What if it's written in crayon? Is the TLB a problem then?

Maursault 5y ago

Everyone calm down. Not every observation on Apple's stuff is earth-shattering. In fact... none of them are, but Apple is of course partly or mostly to blame for revving up everyone's engines. The M1 Ultra Studio is not what you think it is. It is not Apple's flagship high-performance machine. It isn't. That is still the Intel Mac Pro. The Studio is the Mac Mini update that was rumored 6 months before. The Studio is a Mac Mini, hey just gave it a new name. IOW the Studio is the low-end and entry level machine for professionals. Also, the current newest generation Apple hardware is not a revolution. Certainly, Apple deserves praise for seamless platform switching, nice work, truly. But these new Macs are only a little bit more performant than the previous generation. Check the benchmarks. Apple's best years were 2010-2012, where each year they doubled the performance of their machines of the previous year. It took another 6 years for performance to double again. The M1, M1 Pro, M1 Max, and M1 Ultra are all marginally faster than the previous generation, the Intel Macs. Except for 2010-2012, which was amazing, this slight increase in performance is entirely typical of new Macs compared to whatever model's previous iteration. This twitter guy is talking about GPU and games. I realize GPU is important to the computing industry, but not to me. I've been a Mac user since 1989, and even if it was and is included in the machine, I've never once used the GPU. It just sits idle there while I saturate the CPU. Macs are tools for professionals. Professionals don't play games with their tools. If Apple has somehow offended the gaming community by not catering to their needs, and also boxed the GPU performance in the M1 Ultra, it's really a win-win. Ok, it's a little embarrassing, but regardless, doesn't deserve the expected outrage from the sensationalist reporting. How dare Apple! Right? It's more like, ok, well, that sucks a little, but who gives a shit?

Dylan16807 5y ago
> Everyone calm down. I'd say the overall tone of comments is pretty skeptical.
vimy 5y ago
More and more (professional) apps use the gpu for compute. So does macOS. The days of using a gpu only for games or even graphical apps are long over. I’m pretty sure yours isn’t as idle as you think it is.

Epiphany21 5y ago

Everyone in this comment section is saying "it's nothing new" as if that excuses the issue. From the way it's described this is a software optimization problem. So the question is, how hard is it to optimize an existing application? If it's a few quick changes and a rebuild, then it's nothing. If it's anything more you'll have an Itanium-like product flop. Maybe. It's also possible that node size and computational headroom relative to the software of the day will make it hard for consumers to notice or care. Either way this will be interesting to watch.

scottlamb 5y ago
> So the question is, how hard is it to optimize an existing application? TBD how hard it is to optimize an existing application to remove this bottleneck. The tweets seem to suggest it's pretty involved. Alternatively, there might be a simple way to use a larger page size (called "huge pages" on Linux, possibly "superpages" on macOS), which would greatly relieve the TLB as a bottleneck, if TLB = translation lookaside buffer. The tweets seem confused enough that I'm not sure. > If it's a few quick changes and a rebuild, then it's nothing. If it's anything more you'll have an Itanium-like product flop. Meh, if it means many applications only reach 86W rather than 105W, that's probably not world-ending. Most applications on any platform have some bottleneck that they could avoid with more optimization effort, whether it's the TLB or something else. Users usually don't complain in spite of these bottlenecks. And: > It's also possible that node size and computational headroom relative to the software of the day will make it hard for consumers to notice or care. Yeah, people seem pretty happy with these processors.

loser777 5y ago

"Unlike a CPU, when a GPU is waiting for data, it can't just switch to work on something else." https://twitter.com/VadimYuryev/status/1514295693581586434 https://twitter.com/VadimYuryev/status/1514295693581586434 Is this meant to be qualified with (Apple) GPU? Otherwise it sounds like the literal definition of latency hiding that has been the norm on desktop GPUs (and one of the first things taught to newcomers of the OpenCL/CUDA programming model) for a while.

rayiner 5y ago
He says the entire thread group needs to be stalled. The thread group is (on M1) 32 SIMD lanes that have to execute basically the same control flow. Presumably other thread groups can continue executing.
wyldfire 5y ago
I guess the point here is that GPUs don't have general purpose tasks doing independent work and an OS scheduler. You might end up masking part of the cost of some stalls if you are able to swap in other ready-to-run tasks. The bus latency you're describing doesn't exist when you don't need to copy to a GPU-dedicated GDDR/HBM bank. But while that problem goes away, it sounds like this new one of TLB pressure shows up. I'm no expert but I suspect the mobile SoCs like Bionic and Snapdragon have the same concept as the M1 Ultra with respect to integrated GPU sharing memory with the apps cores. M1 probably inherited it from Bionic? So some of this work of porting compute software to reflect this environment may have already started. I guess the challenge is that the bar is higher for expectation of GPU performance in a desktop system like the Mac Studio.
- dragontamer 5y ago
  I don't know much about the M1. But I can say for sure that Intel iGPUs, AMD GCN, AMD RDNA, AMD CDNA, and multiple NVidia-generations of GPUs all have hyperthread-like rescheduling of independent workgroups. In fact, something like 8x wavefronts / warps run in parallel on modern GPUs. When one wavefront / warp stalls due to a memory read/write (or a PCIe read/write), the GPUs universally "hyperthread-out" and hide the latency. Its "different" from how CPUs do it, but the fundamental principals are the same. (CPUs have a redundant set of registers tracked in a register file. GPUs on the other hand, have a set of registers and the kernel-scheduler (or whatever handles CUDAstreams) carefully assigns those registers to not conflict with any running wavefronts). ------- The statement so listed is blatantly false, at least for Intel, AMD, and NVidia GPUs. Maybe Apple iGPUs are built different, but I find that unlikely.

t8y 5y ago

If you just use regular vertex and fragment shaders then all of that is automatically handled. A game like World of Warcraft might not use advanced GPU features that would slow down a TBDR.

usaphp 5y ago

> "The M1 family of chips, including the M1 Ultra, has a major limitation that can't be fixed unless apps are properly optimized." So am I understanding it correctly - if an app is properly optimized - then it's not even an issue?

monocasa 5y ago

I think we're missing some info. 32MB of coverage is the same size for the TLB on a SM for a 1080ti. https://dl.acm.org/doi/10.1145/3491218 https://dl.acm.org/doi/10.1145/3491218 I don't think that's gone up a lot since then. And GPUs in general are totally fine with stalling on memory accesses. That's core to how they work, and why you run into issues treating them like CPUs. They approach the problem of the memory hierarchy at a base level differently by accepting the fact that thread groups will stall on pretty much every memory access, and just barrel scheduling enough thread groups that you can keep ALUs saturated despite the memory dependencies due to massive parallelism. Cache and TLB misses aren't that big of a deal if you have enough work to cycle through. If you don't have massively parallel amounts of work, and can't just keep cycling through contexts, this is the main reason why some algorithms don't work on GPUs, but that's a global problem to how all major GPUs approach the problem of memory that's hundreds of cycles away. > Hishnash: "What apps should be doing is loading as much data as possible into the tile mem and flushing it out in large chunks when needed. I bet a lot of the writes (over 95%) are for temporary values that could've been stored in tile mem and never needed to be written at all." I mean, you shouldn't be performing large amounts of writes that aren't a completion of a program on just about any GPU architecture. If you care about the results, heavy random writes screws you just as hard in a traditional GPU architecture too. You really want to be able to look at the data flow and describe it in almost pure functional terms. If you can't look at a program and say "it writes here at this offset or offsets", you're going to have a bad time. And finally, there's all sorts of ways you can crank up the GPU without doing writes at all, so the inability to turn it into a space heater just because of TLB pressure doesn't make a lot of sense. The xor chains that something like the bogomips calculator uses will eat up every cycle you give it and make their way through the compiler and hardware schedulers all without blocking on memory at all.

dragontamer 5y ago

This sequence of posts is written very strangely and imprecisely? You'd almost never say "32MB of TLB". You'd say maybe, 8192 entries of TLB, each with 4kB pages, for instance. (Others here in Hacker News suggest that the M1 uses a 16kB page, so maybe 2048 entries of 16kB each?) The Twitter posts are talking about temperature at first, tile-memory. I admit that I'm ignorant to tile-memory but... The traditional answer to TLB bottlenecks is to enable large-pages or huge-page support. So I'm already feeling a bit weird that large pages / huge pages haven't been discussed in the twitter thread. (Maybe its not the answer on the GPU side of things? But at least acknowledging how we'd solve it in CPU-land would help show off how features like huge-pages could help) Like, none of the logic here makes sense to me at all. Maybe something weird is going on in the Apple / M1 world, but I have a suspicious feeling that maybe the person posting this Twitter thread is at best... misstating things in a very confusing manner and using imprecise language. At worst, they might be fundamentally incorrect on some of these discussion points?

eyelidlessness 5y ago
I don’t code for GPUs but it seemed pretty straightforward to me: 1. General purpose GPU code assumes optimization characteristics where memory constraints are less restrictive for this use case 2. Without optimizing for this platform, typical code will not exercise the full potential of the chip The thread is discussing it as a design flaw; I don’t know if it is. But if normal “recompile for ____ target” in XCode or whatever isn’t enough for common workloads to be relatively close to optimal, and you have to do and know to do special work for the memory architecture… yeah that’s probably not ideal. And the other takeaway from the thread, that they’re likely to throw more memory hardware at the problem in the future, sounds right to me.
- ngcc_hk 5y ago
  The key is apple code especially os and their tools. That is what m1 and future apple cpu and gpu are for. Hence optimisation should not be the issue. The whole thing is all about optimisation. The question is whether it affect other or 3rd party products (that cannot compete, or not shown to the public eg the private api ). And does apple care. Unless it shows is that limitation important ?

hedgehog 5y ago

Some of the main assertions about how common GPUs work are totally backwards: "when a GPU is waiting for data, it can't just switch to work on something else" - GPU designs specifically use the idea of rotating between threads to not block on memory access. "the concept that there is a local on-die memory pool you can read/write from with very very low perf impact is unthinkable in the current desktop GPU space" - Optimizing for the characteristics of the specific device's memory hierarchy including local/shared memory is a big part of GPU coding. If Apple follows the pattern that companies like NVIDIA have used they will keep the overall shape of the GPU similar between generations (in terms of how many threads, memory hierarchy, etc). This will give app devs relatively similar + stable tuning targets, we'll see high dollar apps get aggressively optimized for specific chips, and as that happens power/heat will go up. For an extreme example of this maturation curve look at how console games get more and more out of the same hardware. All that said the Metal tile shading API is a real thing Apple has been pushing for years on the mobile side, presumably because it fits the hardware and performs better, and now Macs use similar chips. No surprises there. And if an app is built around OpenGL and now needs to port to Metal to improve performance, sure, that could be a lot of work.

SwordOfMyBone 5y ago
Don't know anything about GPUs and am a CS undergrad. How does this apply to compute workloads? What's wrong with treating it like an IMR GPU when it comes to compute? is the tile buffer used for compute? ELI 5 please.
- Maursault 5y ago
  > Don't know anything about GPUs and am a CS undergrad. You are the computer in "Computer Science," and the science is really just a lot of math. Maybe you wanted to study CE instead? I made the same damn mistake.
  - SwordOfMyBone 5y ago
    I unironically enjoy CS. I like programming and I like being close to the metal. I have an interest in the engineering part but I don't know if I'd pick it over CS. Life is hard. Also like compilers, and most compiler engineers are CS grads. That's my dream job!

gotaquestion 5y ago

I'm curious how they claimed that the cooling solution is over-designed because they were only able to achieve 80% of the spec'd power target (most likely TDP since people stopped using max power in the 00's). There are a number of factors that go into spec'ing this value, typically quite a few test vectors to determine typical workloads and high-power targeted workloads. A mistake in power budget of 20% is gigantic and would have significant impact on sorting die. Now maybe Apple doesn't care about this because they aren't a silicon vendor, but that budget comes out of battery life and cooling solution space. My point is, I find it highly unlikely Apple overstated their power target by 20% given how much money it would have saved them. But maybe they did?

jhugo 5y ago
Well, 20% headroom on a badly optimised workload that can't make full use of the GPU seems decent. Run some better code and you'll need that 20%. As covered in other comments, the thread author has no idea what they're writing about.

kazinator 5y ago

I'm confused. A 32 megabyte TLB is huge. TLB entries are tiny objects; and each one caches the mapping for an entire VM page. The larger the page size, the more "leverage" the TLB has. Maybe the blogger means that the TLB controls a VM working set or footprint of 32 MB?

vmception 5y ago

So it can only use 68w instead of 105w? Thats the crux of the complaint here? This honestly makes me want to throw out all my other computers and GPUs that do whatever they do at any performance metric simply because they use so much more energy to do so

gchadwick 5y ago

I note the only metric they offer is power usage. A poor substitute for actual performance numbers. In particular in any given CPU/GPU you could have all cores busily computing, nothing stalled and still be a good way below maximum power. To hit maximum power you need to carefully construct software that will fully utilize all functional units as well as causing maximum toggling within computation paths (1 + 2 will toggle fewer bits than 0xffffffff + 0xffffffff). The thread doesn't give any clear indication if their explanation that the GPU sees significant stalls due to waiting on TLB miss has actual data behind it or is pure conjecture based upon observed power usage.

ribit 5y ago

The author of those tweets doesn't seem to understand what TLB is (their terminology is off), doesn't seem to understand how GPUs work (they are explicitly designed to hide memory stalls by switching to a different kernel, which is why they have huge register files), doesn't seem to understand the memory hierarchies (why would a GPU have it's own TLB in the first place?). Besides, we know that M1 comes with a 3072-entry TLB that matches it's maximal 48MB cache size, so where did they come up with the "32MB figure anyway"? It is entirely possible that M1 Ultra is bandwidth starved for some workloads, but that can be demonstrated using GPU profiling tools (which Apple readily provides) and not by musing about technical concepts that one barely understands.

monocasa 5y ago
I agree with most of what you say but > why would a GPU have it's own TLB in the first place? is because modern GPUs have their own MMUs and have for quite a while. More or less ubiquitous MMUs on GPUs are half the reason why Mantle/Vulkan/DX12/Metal came into being.
- ribit 5y ago
  Thanks for this! Can you elaborate a bit more and maybe give me some pointers where I can read about these things in more depth? I of course know about the MMU on traditional dGPUs but I was not aware that “integrated” GPUs also have a separate MMU. How does it work and why is it necessary given that the GPU shares the last level cache and the memory controller with the rest of the system?
  - monocasa 5y ago
    Because the TLBs and other MMU hardware aren't in the last level cache or memory controller on the vast majority of systems, they normally sit about at L1 on each core. This is because you want at least L2 to be speaking completely in physical addresses so the coherency protocol isn't confused by the same page mapped in different ways. L1 is more complex typically being virtually indexed, but physically tagged. So when a access is issued with the virtual address, in parallel to the cache set look up, the TLB translates to a physical address, and then the physical address is compared on the tags of the resulting cache line ways in the set that matched the virtual address in order to select the actual cache line for the op.

lvl102 5y ago

This guy knows nothing about computers. He is a Youtuber with zero expertise in anything but creating Youtube videos.

daneel_w 5y ago

I may have misunderstood things, but isn't the gist that software written correctly for this unique hardware isn't subject to the bottleneck? Is it really a hardware problem then, if it can't accommodate suboptimal software with maximum performance? This is after all not an Nvidia or AMD/ATI GPU architecture, and I wonder if anyone ever brought a similar argument forward in the many cases of this or that game having distinctly poor 3D performance on any of their hardware.

bdirgo 5y ago

Y’all just got nerd sniped.