4 ms·

"You don't think about memories when you design hardware" I found that comment interesting because memory is mostly what I think about when I design hardware.

"You don't think about memories when you design hardware" I found that comment interesting because memory is mostly what I think about when I design hardware. Memory access is by far the slowest thing, so managing memory bandwidth is absolutely critical, and it uses a significant portion of the die area. Also, a significant portion of the microarchitecture of a modern CPU is all about managing memory accesses. If there were a hypothetical high-level CPU language that somehow encoded all the information the microarchitecture needs to measure to manage memory access, then it would likely tie in performance, assuming the CPU team did a good job measuring memory. Still, it wouldn't need all the extra stuff that did the measurement. So I see that as a win. The main problem is, I have absolutely no idea how to do that, and unfortunately, I haven't met anyone else who knows either. Hence, tons of bookkeeping logic in CPUs persists.

eigenform 1y ago
Also weird because pipelining is very literally "insert memories to cut some bigger process into parts that can occur in parallel with one another"
foota 1y ago
This is almost the opposite of what you're referring to I think, but interestingly Itanium had instruction level parallelism hard-coded by the compiler as opposed to determined by the processor at runtime.
uticus 1y ago
"memories" could also refer to registers, or even a stack-based approach
- blakepelton 1y ago
  Some folks use the term "state element" to describe the set of things which hold state between clock cycles. Memories (e.g., SRAM) and registers are both examples of state elements. I took the "You don't think about memories when you design hardware" paragraph from Yosef to mean that SRAM is so highly tuned that in most cases your best bet is to just use it "off the shelf" rather than invent some novel type of state element. And if you use SRAM, then you are stuck sucking through a small straw (e.g., an SRAM can hold 1K words, but you can only read 1 word/cycle).
Legend2440 1y ago
Keep in mind this article is almost 20 years old. Memory access used to matter less because CPUs were a lot slower relative to memory. When I was a kid you could get a speedup by using lookup tables for trig functions - you'd never do that today, it's faster to recalculate.
- PeterWhittaker 1y ago
  Wait, what? Oh, yeah. Jeebus. 2008 was SEVENTEEN years ago. I turn 60 soon. Anything 2xxx just seems like almost yesterday, 199x isn't that far away in mind, and even 198x is pretty close. I love math (B.Sc. Physics, so more math than anything else) but sometimes math hurts. Damn you, beloved math.
gchadwick 1y ago
This certainly rings true with my own experiences (worked on both GPUs and CPUs and now doing AI inference at UK startup Fractile). > If there were a hypothetical high-level CPU language that somehow encoded all the information the microarchitecture needs to measure to manage memory access, I think this is fundamentally impossible because of dynamic behaviours. It's very tempting to assume that if you can be clever enough you can work this all out ahead of time, encode what you need in some static information and the whole computation just runs like clockwork on the hardware, no or little area spent on scheduling, stalling, buffering etc. Though I think history has shown over and over this just doesn't work (for more general computation at least, more feasible in restricted domains). There's always lots of fiddly details in real workloads that surprise you and if you've got an inflexible system you've got no give and you end up 'stopping the world' (or some significant part of it) to deal with it killing your performance. Notably running transformer models feels like one of those restrictive domains you could do this well in but dig in and there's plenty enough dynamic behaviour in there that you can't escape the problems they cause.
- throwaway31131 1y ago
  > I think this is fundamentally impossible because of dynamic behaviours. I was thinking that too as it sounds awfully close to a variant of the halting problem. But since I am not sure that it's actually equivalent, and there might be some heuristic that constrains the problem, in theory, to make it solvable "enough" I went with the softer "I don't know how to do it" rather than "it can't be done"

eigenform 1y ago

Also weird because pipelining is very literally "insert memories to cut some bigger process into parts that can occur in parallel with one another"

foota 1y ago

This is almost the opposite of what you're referring to I think, but interestingly Itanium had instruction level parallelism hard-coded by the compiler as opposed to determined by the processor at runtime.

uticus 1y ago

"memories" could also refer to registers, or even a stack-based approach

blakepelton 1y ago
Some folks use the term "state element" to describe the set of things which hold state between clock cycles. Memories (e.g., SRAM) and registers are both examples of state elements. I took the "You don't think about memories when you design hardware" paragraph from Yosef to mean that SRAM is so highly tuned that in most cases your best bet is to just use it "off the shelf" rather than invent some novel type of state element. And if you use SRAM, then you are stuck sucking through a small straw (e.g., an SRAM can hold 1K words, but you can only read 1 word/cycle).

Legend2440 1y ago

Keep in mind this article is almost 20 years old. Memory access used to matter less because CPUs were a lot slower relative to memory. When I was a kid you could get a speedup by using lookup tables for trig functions - you'd never do that today, it's faster to recalculate.

PeterWhittaker 1y ago
Wait, what? Oh, yeah. Jeebus. 2008 was SEVENTEEN years ago. I turn 60 soon. Anything 2xxx just seems like almost yesterday, 199x isn't that far away in mind, and even 198x is pretty close. I love math (B.Sc. Physics, so more math than anything else) but sometimes math hurts. Damn you, beloved math.

gchadwick 1y ago

This certainly rings true with my own experiences (worked on both GPUs and CPUs and now doing AI inference at UK startup Fractile). > If there were a hypothetical high-level CPU language that somehow encoded all the information the microarchitecture needs to measure to manage memory access, I think this is fundamentally impossible because of dynamic behaviours. It's very tempting to assume that if you can be clever enough you can work this all out ahead of time, encode what you need in some static information and the whole computation just runs like clockwork on the hardware, no or little area spent on scheduling, stalling, buffering etc. Though I think history has shown over and over this just doesn't work (for more general computation at least, more feasible in restricted domains). There's always lots of fiddly details in real workloads that surprise you and if you've got an inflexible system you've got no give and you end up 'stopping the world' (or some significant part of it) to deal with it killing your performance. Notably running transformer models feels like one of those restrictive domains you could do this well in but dig in and there's plenty enough dynamic behaviour in there that you can't escape the problems they cause.

throwaway31131 1y ago
> I think this is fundamentally impossible because of dynamic behaviours. I was thinking that too as it sounds awfully close to a variant of the halting problem. But since I am not sure that it's actually equivalent, and there might be some heuristic that constrains the problem, in theory, to make it solvable "enough" I went with the softer "I don't know how to do it" rather than "it can't be done"