5 ms·

I've recently benchmarked RAM bandwidth and latency on computers based on i.MX53 (Cortex-A8), Exynos4 (Cortex-A9) and Exynos5 (Cortex-A15). RAM throughput incre

by lgeek 13y ago

I've recently benchmarked RAM bandwidth and latency on computers based on i.MX53 (Cortex-A8), Exynos4 (Cortex-A9) and Exynos5 (Cortex-A15). RAM throughput increases by something like 20 times between the A8 and the A15 platforms. So while it's true that ARM systems used to suffer from low throughput, this is something they've been working on for some time, and the results look quite good from where I'm staying. I found this microbenchmark[0] and (looking at memcpy) the results for new ARM systems look alright to me. Of course, this is an oversimplification, but I'm happy to go into more details if there's interest. Since the first ARMv8 cores are built for server workloads, I expect they're brining significant improvements compared to A15. Disclaimer: Not associated with ARM, but I have an interest in ARM being used more for things other than mobile devices. [0] https://github.com/c2h2/arm_c_benchmark https://github.com/c2h2/arm_c_benchmark

akiselev 13y ago
Kudos for actually trying out a benchmark but it must be seriously flawed. There's no way you'd get a 20x boost in bandwidth between the A8 and A15 (both of which are ARMv7 and DDR3 is not 20x faster than DDR2) in nontrivial cases if you're using technology within the last 5 years. I'm guessing you ran a very trivial benchmark that operated mostly out of the L1/L2 cache (hence you weren't really testing the RAM) on the Exynos5 and mostly out of ram on the i.MX53. Depending on what operating system you ran on the different boards, there could also be major differences in the kernel's implementation of the DMA peripheral, which would also heavily skew results. For example, in the Github link, some code used native memcpys while others used kernel call memcpys. The differences in the specific Ubuntu 13.04 and Android implementations could vary the results quite a bit, even if they have the same exact overhead.
- lgeek 13y ago
  > There's no way you'd get a 20x boost in bandwidth between the A8 and A15 (both of which are ARMv7 and DDR3 is not 20x faster than DDR2) There are a few factors at work here: * A15 was designed to have more memory bandwidth in the first place, since this was a known issue. I think the bottleneck was (is?) the speed of AMBA bus[0] connecting the peripherals (including RAM controller) and the ARM core. * A15 has a vastly more advanced pipeline and multiple-issue capabilities than A8. This should allow it to use its functional units more efficiently. * Finally, my A15 system is running at 1.7 GHz compared to 1 GHz for the A8 system. > I'm guessing you ran a very trivial benchmark that operated mostly out of the L1/L2 cache The buffers were pre-initialized (to remove the kernel's lazy physical memory allocation as a factor), and far larger than the L2. The data caches were explicitly flushed at the beginning. But indeed it was a very simple benchmark since it was a microbenchmark meant to measure memory bandwidth. > Depending on what operating system you ran on the different boards, there could also be major differences in the kernel's implementation of the DMA peripheral. I wasn't using DMA. > For example, in the Github link, some code used native memcpys while others used kernel call memcpys. To clarify: the project I've linked isn't mine, it's a public project that supported my point. But as far as I'm aware there's no memcpy systemcall. memcpy is implemented completely in libc. For example, glibc has multiple optimized implementations written in assembly[1]. But you're right, the glibc version is different between these computers, so I need to repeat with a statically compiled benchmark. Anyway, I've looked at this while trying to improve memcpy speed for A15, which is why I don't have solid comparative results. But I'm doing a write-up and now I'll probably also include this bit. [0] https://en.wikipedia.org/wiki/Advanced_Microcontroller_Bus_Architecture https://en.wikipedia.org/wiki/Advanced_Microcontroller_Bus_A... [1] http://sourceware.org/git/?p=glibc.git;a=tree;f=ports/sysdeps/arm;h=928696e7715c6c454c5792a1b5c5973d0005fc17;hb=3711a167f6b5203b4414db7853f3c57f1260e1bf http://sourceware.org/git/?p=glibc.git;a=tree;f=ports/sysdep...
  - akiselev 13y ago
    Huh wow, I guess I'm behind the times. Didn't know they made that big of a jump within ARMv7. This makes me curious to look at the datasheets. From what I can tell, the i.MX53[1] has a 64 bit AXI @ 200mhz. The Exynos5[2] on the other hand, has 64bit AXIs and also a ton of optimizations. The LCD spec says it operates off of a 200mhz AXI so I wouldn't be surprised if the Exynos5 uses dual 200mhz AXIs for memory. It's tough to tell how much of that 20 times translates to the clock speeds and the rest to optimizations. I agree the A15 is just a sign of things to come since ARMv8, although it has legacy stuff, ARM still gets to improve a lot more on the architecture specifically for servers. For comparison, these guys [3] say 12.8 GB/S which kinda sounds ridiculous. If it's true, they really are in range of Intel. The Sandy Bridge Xeon E3-1220 boasts a theoretical 21 GB/s @ a whopping 80 watts (although the high end E7-8870 [4] is a fucking monster. At this point I can't even begin to think how they compare with all of Intel's memory stuff). The number are clearly within the ballpark, we'll just have to see a real world case. I'm curious about how well GCC and the kernels utilize all the unique processor features and optimizations. If it's mostly down to market forces, Intel's 96% market share might make it a long and difficult journey to an ARM Linux kernel optimized as well for a server environment as x86/x64 ports. I'm too lazy to check what that assembly code is but if it uses NEON PLD optimizations (NEON is definitely in Exynos5) that may give a speed bump in memcpy (even if you preloaded the buffers) because those optimizations would use L1/L2 cache intelligently. It's hard to tell without looking at the code and diving into i.MX53's feature set more whether that played a factor. * I was thinking of malloc and free, they are system calls because of paging. Memcpy is a straight pointer to pointer copy, except maybe for whatever that assembly code does. Time to just find a fluffy article on this topic [5] [1] http://www.freescale.com/files/32bit/doc/data_sheet/IMX53IEC.pdf http://www.freescale.com/files/32bit/doc/data_sheet/IMX53IEC... [2] http://www.samsung.com/global/business/semiconductor/file/product/Exynos_5_Dual_User_Manaul_Public_REV1.00-0.pdf http://www.samsung.com/global/business/semiconductor/file/pr... [3] http://www.maximumpc.com/article/news/samsung_details_exynos_5_dual_chip_better_retina_display_muscle http://www.maximumpc.com/article/news/samsung_details_exynos... [4] http://ark.intel.com/products/53580/Intel-Xeon-Processor-E7-8870-30M-Cache-2_40-GHz-6_40-GTs-Intel-QPI http://ark.intel.com/products/53580/Intel-Xeon-Processor-E7-... [5] http://www.theregister.co.uk/2011/10/20/details_on_big_little_processing/ http://www.theregister.co.uk/2011/10/20/details_on_big_littl...

by lgeek 13y ago

akiselev 13y ago

Kudos for actually trying out a benchmark but it must be seriously flawed. There's no way you'd get a 20x boost in bandwidth between the A8 and A15 (both of which are ARMv7 and DDR3 is not 20x faster than DDR2) in nontrivial cases if you're using technology within the last 5 years. I'm guessing you ran a very trivial benchmark that operated mostly out of the L1/L2 cache (hence you weren't really testing the RAM) on the Exynos5 and mostly out of ram on the i.MX53. Depending on what operating system you ran on the different boards, there could also be major differences in the kernel's implementation of the DMA peripheral, which would also heavily skew results. For example, in the Github link, some code used native memcpys while others used kernel call memcpys. The differences in the specific Ubuntu 13.04 and Android implementations could vary the results quite a bit, even if they have the same exact overhead.

lgeek 13y ago
> There's no way you'd get a 20x boost in bandwidth between the A8 and A15 (both of which are ARMv7 and DDR3 is not 20x faster than DDR2) There are a few factors at work here: * A15 was designed to have more memory bandwidth in the first place, since this was a known issue. I think the bottleneck was (is?) the speed of AMBA bus[0] connecting the peripherals (including RAM controller) and the ARM core. * A15 has a vastly more advanced pipeline and multiple-issue capabilities than A8. This should allow it to use its functional units more efficiently. * Finally, my A15 system is running at 1.7 GHz compared to 1 GHz for the A8 system. > I'm guessing you ran a very trivial benchmark that operated mostly out of the L1/L2 cache The buffers were pre-initialized (to remove the kernel's lazy physical memory allocation as a factor), and far larger than the L2. The data caches were explicitly flushed at the beginning. But indeed it was a very simple benchmark since it was a microbenchmark meant to measure memory bandwidth. > Depending on what operating system you ran on the different boards, there could also be major differences in the kernel's implementation of the DMA peripheral. I wasn't using DMA. > For example, in the Github link, some code used native memcpys while others used kernel call memcpys. To clarify: the project I've linked isn't mine, it's a public project that supported my point. But as far as I'm aware there's no memcpy systemcall. memcpy is implemented completely in libc. For example, glibc has multiple optimized implementations written in assembly[1]. But you're right, the glibc version is different between these computers, so I need to repeat with a statically compiled benchmark. Anyway, I've looked at this while trying to improve memcpy speed for A15, which is why I don't have solid comparative results. But I'm doing a write-up and now I'll probably also include this bit. [0] https://en.wikipedia.org/wiki/Advanced_Microcontroller_Bus_Architecture https://en.wikipedia.org/wiki/Advanced_Microcontroller_Bus_A... [1] http://sourceware.org/git/?p=glibc.git;a=tree;f=ports/sysdeps/arm;h=928696e7715c6c454c5792a1b5c5973d0005fc17;hb=3711a167f6b5203b4414db7853f3c57f1260e1bf http://sourceware.org/git/?p=glibc.git;a=tree;f=ports/sysdep...
- akiselev 13y ago
  Huh wow, I guess I'm behind the times. Didn't know they made that big of a jump within ARMv7. This makes me curious to look at the datasheets. From what I can tell, the i.MX53[1] has a 64 bit AXI @ 200mhz. The Exynos5[2] on the other hand, has 64bit AXIs and also a ton of optimizations. The LCD spec says it operates off of a 200mhz AXI so I wouldn't be surprised if the Exynos5 uses dual 200mhz AXIs for memory. It's tough to tell how much of that 20 times translates to the clock speeds and the rest to optimizations. I agree the A15 is just a sign of things to come since ARMv8, although it has legacy stuff, ARM still gets to improve a lot more on the architecture specifically for servers. For comparison, these guys [3] say 12.8 GB/S which kinda sounds ridiculous. If it's true, they really are in range of Intel. The Sandy Bridge Xeon E3-1220 boasts a theoretical 21 GB/s @ a whopping 80 watts (although the high end E7-8870 [4] is a fucking monster. At this point I can't even begin to think how they compare with all of Intel's memory stuff). The number are clearly within the ballpark, we'll just have to see a real world case. I'm curious about how well GCC and the kernels utilize all the unique processor features and optimizations. If it's mostly down to market forces, Intel's 96% market share might make it a long and difficult journey to an ARM Linux kernel optimized as well for a server environment as x86/x64 ports. I'm too lazy to check what that assembly code is but if it uses NEON PLD optimizations (NEON is definitely in Exynos5) that may give a speed bump in memcpy (even if you preloaded the buffers) because those optimizations would use L1/L2 cache intelligently. It's hard to tell without looking at the code and diving into i.MX53's feature set more whether that played a factor. * I was thinking of malloc and free, they are system calls because of paging. Memcpy is a straight pointer to pointer copy, except maybe for whatever that assembly code does. Time to just find a fluffy article on this topic [5] [1] http://www.freescale.com/files/32bit/doc/data_sheet/IMX53IEC.pdf http://www.freescale.com/files/32bit/doc/data_sheet/IMX53IEC... [2] http://www.samsung.com/global/business/semiconductor/file/product/Exynos_5_Dual_User_Manaul_Public_REV1.00-0.pdf http://www.samsung.com/global/business/semiconductor/file/pr... [3] http://www.maximumpc.com/article/news/samsung_details_exynos_5_dual_chip_better_retina_display_muscle http://www.maximumpc.com/article/news/samsung_details_exynos... [4] http://ark.intel.com/products/53580/Intel-Xeon-Processor-E7-8870-30M-Cache-2_40-GHz-6_40-GTs-Intel-QPI http://ark.intel.com/products/53580/Intel-Xeon-Processor-E7-... [5] http://www.theregister.co.uk/2011/10/20/details_on_big_little_processing/ http://www.theregister.co.uk/2011/10/20/details_on_big_littl...