5 ms·

Interesting point about L1 cache sizes and the relationship between page size > Also, first-level caches are usually limited by the page size times the associa

Interesting point about L1 cache sizes and the relationship between page size > Also, first-level caches are usually limited by the page size times the associativity of the cache. If the cache is smaller than that, the bits used to index into the cache are the same regardless if whether you're looking at the virtual address or the physical address, so you don't have to do a virtual to physical translation before indexing into the cache. If the cache is larger than that, you have to first do a TLB lookup to index into the cache (which will cost at least one extra cycle), or build a virtually indexed cache (which is possible, but adds complexity and coupling to software). You can see this limit in modern chips. Haswell has an 8-way associative cache and 4kB pages. Its l1 data cache is 8 * 4kB = 32kB. Having helped build the virtually indexed cache of the arm A55 I can confirm it's a complete nightmare and I can see why Intel and AMD have kept to the L1 data cache limit required to avoid it. Interestingly Apple may have gone down the virtually indexed route (or possibly some other cunning design corner) for the M1 with their 128 kB data cache. However I believe they standardized on 16k pages which would allow still allow physical indexing with an 8 way associative cache. So what do they do when they're running x86 code with 4k pages? Does they drop 75% of their L1 cache to maintain physical indexing? Do they aggressively try and merge the x86 4k pages into 16k pages with some slow back-up when they can't do that? Maybe they've gone with some special purpose hardware support for emulating x86 4k pages on their 16k page architecture. Have they just indeed implemented a virtually indexed cache?

zozbot234 4y ago
> Do they aggressively try and merge the x86 4k pages into 16k pages with some slow back-up when they can't do that? This does not seem feasible because the 16k pages on ARM are not "huge" pages; it's a completely different arrangement of the virtual address space and page tables. The two are not interoperable.
- ddingus 4y ago
  Perhaps that explains the odd cache sizes. Map x86 onto it's own cache when necessary.
heavyset_go 4y ago
Please add an RSS or Atom feed to your blog :)
my123 4y ago
> with their 128 kB data cache Bigger than that. (https://github.com/apple/darwin-xnu/blob/2ff845c2e033bd0ff64b5b6aa6063a1f8f65aa32/osfmk/arm/proc_reg.h#L369 https://github.com/apple/darwin-xnu/blob/2ff845c2e033bd0ff64...) /* I-Cache, 256KB for Firestorm, 128KB for Icestorm, 6-way. / / D-Cache, 160KB for Firestorm, 8-way. 64KB for Icestorm, 6-way. */ Apple also uses a base 16KB page size on macOS, with 4KB being present for backwards compatibility (x86_64 apps) and the wider ecosystem (running Windows for example). > Maybe they've gone with some special purpose hardware support for emulating x86 4k pages on their 16k page architecture Nah, it's that when you run an x86 process, one of the TTBRs is pointed to a page size corresponding to 4KB pages.
- throwawaylinux 4y ago
  Firestorm L1 d$ is 128kB most diagrams and documentation will show that. This latency measurement pretty well confirms it too - https://www.anandtech.com/show/16226/apple-silicon-m1-a14-deep-dive/2 https://www.anandtech.com/show/16226/apple-silicon-m1-a14-de... Apple definitely takes advantage of VI=PI for their caches. 160kB is not a good size for a cache, surely must be a typo. i$ has more options for handling aliases. You don't actually even need to eliminate them at all because it's a read-only cache. So it's not unusual to see these caches exceed the VI=PI geometry.
  - my123 4y ago
    That’s what Apple explicitly says. Why they don’t match with measurements is another matter… (and no, it’s not a typo) ECC? Needs hugepages? Or hardware erratum? I don’t know for sure. Also, at least from the VM perspective icache is reported as PIPT, which is not too surprising but still worth noting.
    - throwawaylinux 4y ago
      > That’s what Apple explicitly says. Why they don’t match with measurements is another matter… (and no, it’s not a typo) Ridiculous. You're just making things up. It clearly doesn't have a 160kB 6-way L1 dcache. Nor are the small cores 64kB 6-way -- those don't even divide that's ridiculous. Why do you say it's not a typo? They "explicitly" say it is 160kB in that backwater code comment. They also explicitly say it's 128kB in this presentation https://www.youtube.com/watch?v=5AwdkGKmZ0I&t=534s https://www.youtube.com/watch?v=5AwdkGKmZ0I&t=534s so what makes your source the authoritative one?? Pretty obviously it's a quick cut and paste job with a typo (two in one line actually) in it. > ECC? Needs hugepages? Or hardware erratum? I don’t know for sure. ECC? Erratum? Come on. It's clearly because it's 128kB.
rhn_mk1 4y ago
This took me a while to click, but: - cache entries are assumed to be looked up by the virtual address (not physical), to save a TLB lookup - within a single page, the page line index is the same in both a physical and a virtual address - if the cache has enough sets, it doesn't just use the line bits, but also some bits of the page address Here's the crucial bit: a single physical page address may correspond to several different virtual addresses, so if the cache kept using virtual addresses, it might have had many copies of the same line. Kaboom. That's why having more sets in a cache than lines in a page is troublesome.

zozbot234 4y ago

> Do they aggressively try and merge the x86 4k pages into 16k pages with some slow back-up when they can't do that? This does not seem feasible because the 16k pages on ARM are not "huge" pages; it's a completely different arrangement of the virtual address space and page tables. The two are not interoperable.

ddingus 4y ago
Perhaps that explains the odd cache sizes. Map x86 onto it's own cache when necessary.

heavyset_go 4y ago

Please add an RSS or Atom feed to your blog :)

my123 4y ago

> with their 128 kB data cache Bigger than that. (https://github.com/apple/darwin-xnu/blob/2ff845c2e033bd0ff64b5b6aa6063a1f8f65aa32/osfmk/arm/proc_reg.h#L369 https://github.com/apple/darwin-xnu/blob/2ff845c2e033bd0ff64...) /* I-Cache, 256KB for Firestorm, 128KB for Icestorm, 6-way. / / D-Cache, 160KB for Firestorm, 8-way. 64KB for Icestorm, 6-way. */ Apple also uses a base 16KB page size on macOS, with 4KB being present for backwards compatibility (x86_64 apps) and the wider ecosystem (running Windows for example). > Maybe they've gone with some special purpose hardware support for emulating x86 4k pages on their 16k page architecture Nah, it's that when you run an x86 process, one of the TTBRs is pointed to a page size corresponding to 4KB pages.

throwawaylinux 4y ago
Firestorm L1 d$ is 128kB most diagrams and documentation will show that. This latency measurement pretty well confirms it too - https://www.anandtech.com/show/16226/apple-silicon-m1-a14-deep-dive/2 https://www.anandtech.com/show/16226/apple-silicon-m1-a14-de... Apple definitely takes advantage of VI=PI for their caches. 160kB is not a good size for a cache, surely must be a typo. i$ has more options for handling aliases. You don't actually even need to eliminate them at all because it's a read-only cache. So it's not unusual to see these caches exceed the VI=PI geometry.
- my123 4y ago
  That’s what Apple explicitly says. Why they don’t match with measurements is another matter… (and no, it’s not a typo) ECC? Needs hugepages? Or hardware erratum? I don’t know for sure. Also, at least from the VM perspective icache is reported as PIPT, which is not too surprising but still worth noting.
  - throwawaylinux 4y ago
    > That’s what Apple explicitly says. Why they don’t match with measurements is another matter… (and no, it’s not a typo) Ridiculous. You're just making things up. It clearly doesn't have a 160kB 6-way L1 dcache. Nor are the small cores 64kB 6-way -- those don't even divide that's ridiculous. Why do you say it's not a typo? They "explicitly" say it is 160kB in that backwater code comment. They also explicitly say it's 128kB in this presentation https://www.youtube.com/watch?v=5AwdkGKmZ0I&t=534s https://www.youtube.com/watch?v=5AwdkGKmZ0I&t=534s so what makes your source the authoritative one?? Pretty obviously it's a quick cut and paste job with a typo (two in one line actually) in it. > ECC? Needs hugepages? Or hardware erratum? I don’t know for sure. ECC? Erratum? Come on. It's clearly because it's 128kB.

rhn_mk1 4y ago

This took me a while to click, but: - cache entries are assumed to be looked up by the virtual address (not physical), to save a TLB lookup - within a single page, the page line index is the same in both a physical and a virtual address - if the cache has enough sets, it doesn't just use the line bits, but also some bits of the page address Here's the crucial bit: a single physical page address may correspond to several different virtual addresses, so if the cache kept using virtual addresses, it might have had many copies of the same line. Kaboom. That's why having more sets in a cache than lines in a page is troublesome.