That's one bottleneck, but you're missing another bottleneck entirely. In many problems, we load the entirety of the data into the GPU's RAM anyways (some games

That's one bottleneck, but you're missing another bottleneck entirely. In many problems, we load the entirety of the data into the GPU's RAM anyways (some games do so with the entirety of their assets during load times). It doesn't matter that the data is already in RAM, because we care about how much data can come from GDDR to registers (in GB/s). You have to get the data to your compute units. This is your memory bandwidth. GDDR RAM wants to be accessed in bulk - large rows at once. Things are easy when each thread wants a subsequent byte, but if not things become much slower. Caching and other techniques can help mitigate this, and they're (imo) the place for a lot of creativity in architecture design. Having more potential FLOPS just means more ALUs.