4 ms·

I am building a single socket server right now, I can't really justify more than twice the price of a 7443P for a marginally higher base clock and twice the cac

by cm2187 6y ago

I am building a single socket server right now, I can't really justify more than twice the price of a 7443P for a marginally higher base clock and twice the cache. Does the cache makes that much of a difference? I thought these are already very large caches vs lots of Intel CPUs.

dragontamer 6y ago
Hmm, with AMD Threadripper, you're already looking at TLB issues at these L3 sizes. So if you actually want to take advantage of lots of L3, you need either many cores, or hugepages. Case in point: AMD Zen2 has 2048 TLB entries (L2), under a default (in Linux and Windows) of 4kB per TLB entry. That's 8MBs of TLB before your processor starts to page-walk. Emphasis: Your application will pagewalk when the data still fits in L3 cache. ------------ I'm looking at some of this lineup with 3-cores per CCX (32MBs L3 cache), which means under default 4kB pages, those cores will always require a pagewalk to just read/write its 32MBs L3 cache effectively. With that being said: 2048 TLB entries for Zen2 processors. Maybe AMD has increased the TLB entries for Zen3. Either way, you probably should start looking at hugepage configuration settings... These L3 cache sizes are absurd, to the point where its kind of unwieldy. I mean, with enough configuration / programming, you can really make these things fly. But its not exactly plug-and-play.
- justincormack 6y ago
  The 64k page size available on Arm (and Power) makes a lot more sense with these kind of cache sizes. With 2MB amd64 hugepages its only 16 different pages in that L3 cache, which for a cluster of up to 8 CPUs is not much at all when using huge pages.
  - dragontamer 6y ago
    TLB-misses always slows down your code, even out-of-cache. So having 2MB (or even 1GB) hugepages is a big advantage in memory-heavy applications, like databases. No, 1GB pages won't fit in L3 cache, but it still means you won't have to page-walk when looking for memory. 1GB pages might be too big for today's computers, but 2MB pages might be good enough for default now. Historically, 4kB was needed for swap purposes (going to 2MB with Swap would incur too much latency if data paged out), but with 32GBs RAM + SSDs on today's computers... fewer and fewer people seem to need swap. There might be some kind of fragmentation-benefit for using the smaller pages, but it really is a hassle for your CPU's TLB to try to keep track of all that virtual memory and put it back in order. --------- While there is performance hits associated with page-walks, the page-walk process is fortunately pretty fast. So most applications probably won't notice a major speedup... still though, the idea of tons of unnecessary page-walks slowing down untold amounts of code bothers me a bit for some reason. Note: ARM also supports hugepages. So going up to 2MBs (or bigger) on ARM is also possible.

by cm2187 6y ago

dragontamer 6y ago

Hmm, with AMD Threadripper, you're already looking at TLB issues at these L3 sizes. So if you actually want to take advantage of lots of L3, you need either many cores, or hugepages. Case in point: AMD Zen2 has 2048 TLB entries (L2), under a default (in Linux and Windows) of 4kB per TLB entry. That's 8MBs of TLB before your processor starts to page-walk. Emphasis: Your application will pagewalk when the data still fits in L3 cache. ------------ I'm looking at some of this lineup with 3-cores per CCX (32MBs L3 cache), which means under default 4kB pages, those cores will always require a pagewalk to just read/write its 32MBs L3 cache effectively. With that being said: 2048 TLB entries for Zen2 processors. Maybe AMD has increased the TLB entries for Zen3. Either way, you probably should start looking at hugepage configuration settings... These L3 cache sizes are absurd, to the point where its kind of unwieldy. I mean, with enough configuration / programming, you can really make these things fly. But its not exactly plug-and-play.

justincormack 6y ago
The 64k page size available on Arm (and Power) makes a lot more sense with these kind of cache sizes. With 2MB amd64 hugepages its only 16 different pages in that L3 cache, which for a cluster of up to 8 CPUs is not much at all when using huge pages.
- dragontamer 6y ago
  TLB-misses always slows down your code, even out-of-cache. So having 2MB (or even 1GB) hugepages is a big advantage in memory-heavy applications, like databases. No, 1GB pages won't fit in L3 cache, but it still means you won't have to page-walk when looking for memory. 1GB pages might be too big for today's computers, but 2MB pages might be good enough for default now. Historically, 4kB was needed for swap purposes (going to 2MB with Swap would incur too much latency if data paged out), but with 32GBs RAM + SSDs on today's computers... fewer and fewer people seem to need swap. There might be some kind of fragmentation-benefit for using the smaller pages, but it really is a hassle for your CPU's TLB to try to keep track of all that virtual memory and put it back in order. --------- While there is performance hits associated with page-walks, the page-walk process is fortunately pretty fast. So most applications probably won't notice a major speedup... still though, the idea of tons of unnecessary page-walks slowing down untold amounts of code bothers me a bit for some reason. Note: ARM also supports hugepages. So going up to 2MBs (or bigger) on ARM is also possible.