7 ms·

Playing with a Raspberry Pi 4 64-bit

116 points by _ananos_ 7y ago

nnikoleris 7y ago
Very impressive. In your RPi4 experiments would you benefit from having more DRAM if you didn't have the 1GB limit? I'm wondering how much difference the gic makes. Do you think there is a way to breakdown the performance uplift?
- _ananos_ 7y ago
  More RAM could help us with a scale test, or maybe with a storage-related benchmark. Regarding the GIC, first I think we should spend some time examining the benefits from the A72 upgrade. Then, breaking down the time spent on each step of the VM lifecycle should be fairly straightforward (annotate EL changes, capture numbers with perf or something similar).
  - vardump 7y ago
    I have two 4 GB RPi4s, and can run any tests if you like. Or is this about some 1GB barrier DMA limitation? I did notice VideoCore can only access maximum 1GB of RAM. Related to that?
    - _ananos_ 7y ago
      the issue about the available RAM is this: https://github.com/raspberrypi/linux/commit/cdb78ce891f6c6367a69c0a46b5779a58164bd4b#diff-634f284364ba43ef69912111615b08ef https://github.com/raspberrypi/linux/commit/cdb78ce891f6c636... probably some kind of address mapping, but I'm no expert on this stuff ;-)
      - vardump 7y ago
        Saw that, but that doesn't really tell why, and how much effort is required to solve this. Maybe SD stuff is in the VideoCore side, and it simply can't DMA above 1 GB limit, and requires to have below 1GB DMA buffers + copy for transfers to upper memory. Maybe.
anfractuosity 7y ago
Very interesting! When running KVM on the Pi, does that use virtualisation extensions if the processor has them? (I don't know anything about ARM and virtualisation).
- _ananos_ 7y ago
  yeap, that's the case with the older models too. A53 has also virtualisation extensions. The main difference about the Pi4 is the GIC which removes the need for emulating interrupt handling. Other than that, the A72 handles VMEnters/VMExits by trapping in/out of EL0/EL1/EL2.
Vogtinator 7y ago
Currently u-boot does not support the RPi4 properly (some hardcoded registers and clock numbers), but patches are pending. So EFI support as required by Arch, openSUSE and some others is not available yet. Progress for actual mainline support can be followed here: https://github.com/lategoodbye/rpi-zero/tree/bcm2838-initial https://github.com/lategoodbye/rpi-zero/tree/bcm2838-initial
qalmakka 7y ago
I hope they're going to figure out AArch64 support for the Raspberry Pi soon, especially since they have launched the models with larger amounts of RAM. Having a true AArch64 Raspberry Pi would be really nice.
- Vogtinator 7y ago
  That's what the article is about. Also openSUSE (and SLES) run in 64 bit mode on a RPi 3 by default without issues, using a mainline kernel and FOSS userspace (no proprietary /opt/vc stuff).
- kurtisc 7y ago
  Shortly after the Pi 4 was released I looked into it and found that, for the Pi 3 at least, the foundation weren't really interested. It essentially adds another distro for them to maintain. It's a shame.
  - floatboth 7y ago
    It's not like anyone is forced to use their distro. It's newbie-oriented and extremely annoying for Unix geeks anyway (Debian base with its always-outdated packages, etc) You don't even have to run Linux at all. FreeBSD/aarch64 works great on a Pi3 (as a headless server at least, VC4 is not ported)
    - 0815test 7y ago
      How is Debian annoying for Unix geeks? If outdated packages are an issue, you can always install stable backports, or stuff from testing/unstable.
    - kurtisc 7y ago
      True but the newbies will slowly drift away from all the projects that don't want to split focus from aarch64.
- _ananos_ 7y ago
  actually, the host OS for this post is a ubuntu 18.04.2 server image. root@pi:~# cat /etc/issue.net Ubuntu 18.04.2 LTS root@pi:~# uname -a Linux pi 4.19.57-v8+ #2 SMP PREEMPT Tue Jul 9 20:31:37 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux root@pi:~# file /bin/ls /bin/ls: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-, for GNU/Linux 3.7.0, BuildID[sha1]=de05fcef79d88af9cf9a71ed38e73af0b179bfb2, stripped
LeonM 7y ago
There seems to be some work-in-progress with supporting AArch64 for the rPi. This looks like a fun FOSS subject to contribute to once I get my hands on a Pi4. I like the low-level stuff. Does anyone here have recommendations where to contribute to?
- Y_Y 7y ago
  By the way, the 3B+ is armv7 by default, but also works fine as aarch64. That's how I use mine under nixos.
mikorym 7y ago
I know that this is for 64-bit on the new Raspberry Pi specifically, but would the Pi 4 be able to run a Playstation 2 emulator?
- derefr 7y ago
  Interesting question. Emulating the PS1 is a lot easier on modern hardware than emulating e.g. the SNES, because most of what it does is push polygons, and PS1 games just don’t push very many of them; so even the most anemic mobile GPU core is more than enough to handle the load. I imagine this is still true of the PS2 to an extent. It’s even more just a polygon pusher than the PS1; and the 3D is still generations-old. There’s this forum post linked to on the PCSX2 website: https://forums.pcsx2.net/Thread-Sticky-Will-PCSX2-run-fast-on-my-computer https://forums.pcsx2.net/Thread-Sticky-Will-PCSX2-run-fast-o..., that says the recommended specs are something like: • Intel Core 2 Duo / Core i3 @ 3.2Ghz or faster • Nvidia Geforce 9600GT / 8800GT or better Probably all you need here, to answer this question in theory, is to compare raw benchmark scores between those components and the Pi4’s CPU/GPU. Or, for another angle, here’s a forum post about Play! (an emulator that has aarch64 support) running on the Nintendo Switch’s Tegra X1: https://gbatemp.net/threads/play-ps2-emulator-is-running-on-the-nintendo-switch.538600/ https://gbatemp.net/threads/play-ps2-emulator-is-running-on-... Spoilers: it gets 10FPS. And the Pi4’s GPU is no Maxwell. (And, while Play! isn’t the most optimized of emulators, it’s the one you’d have to use on aarch64. Even if you optimized it 6x, so that the Tegra got 60FPS, the Pi4’s Videocore GPU wouldn’t be pulling nearly that.)
  - davesmith1983 7y ago
    It about how accurate the emulator is. ePSXE used to run fine on a 700mhz PIII with 128gb of ram. There were SNES emulators that also worked fine. However they were inaccurate and lots of games weren't exactly great. MGS for example didn't work well on ePSXe for years because it used many more hardware tricks than same the original ridge Racer. Ridge Racer R4 doesn't even work properly on a PS2 backwards PS1 compatibility because the PS1 on a chip isn't perfect.
    - 7y ago
Havoc 7y ago
>Lightweight virtualization is a natural fit for low power devices No it's not. The pi is punchy enough for it sure, but the above doesn't follow in my mind. Low power = limited resources so ideally you want it run on bare metal to minimise overheads
- _ananos_ 7y ago
  sure! bare-metal would be awesome. But isn't it a shame not to take advantage of all the virtualization/containerization goodies out there? After all, the virtualization overhead nowadays is only referring to I/O (network & storage). CPU/MEM are being virtualized using hardware extensions at (near-)native speeds.
  - Havoc 7y ago
    >CPU/MEM are being virtualized using hardware extensions at (near-)native speeds. Is that really true for rasp level gear? Haven't tested it but my gut feel tells me the hit is sizable. Anyway - don't let that dissuade you from the mission. Container goodies on a rasp is a grand idea...just don't think it's quite as hit free as the article suggests ;)
    - _ananos_ 7y ago
      definitely! that's our initial goal. Quantify the penalty and examine the trade-offs. Clearly, virtualizing workloads on such devices with standard VMMs/hypervisors isn't ideal. And we're working towards this direction; playing with the systems stack is what makes us tick, so, it will be a fun and (hopefully) useful adventure :D
      - Havoc 7y ago
        >Quantify the penalty and examine the trade-offs. If you do have numbers available I'd love to see a write-up on what the real world hit is. I used to run a rasp3b for home server but that just wasn't punchy enough (crappy fake gigabit etc). So my old gaming laptop became a server w/ docker etc. But itching to justify a rasp 4. Anyway...thanks for exploring 64...thought the 32 on rasp 3 was unfortunate.
londons_explore 7y ago
I thought the whole point of things like docker is they gave native performance? Isn't docker just a combination of cgroups, network namespaces, pid namespaces, fancy filesystem mounts, etc. The overhead should be zero by design. What am I missing?
- _ananos_ 7y ago
  there is a number of factors to consider when comparing this kind of technologies. Linux containers provide native performance for almost all applications, true; but there are tons of implications when it comes to multi-tenancy (security, QoS, trusted execution etc.). Moreover, spawning an application as a docker container can incur significant overhead on startup time, fs setup, FS access etc. So in theory yes, containers provide native performance. But do we really want to run apps on multi-tenant edge devices as containers ? I would argue not necessarily ;)

116 points by _ananos_ 7y ago

Very impressive. In your RPi4 experiments would you benefit from having more DRAM if you didn't have the 1GB limit? I'm wondering how much difference the gic makes. Do you think there is a way to breakdown the performance uplift?

_ananos_ 7y ago
More RAM could help us with a scale test, or maybe with a storage-related benchmark. Regarding the GIC, first I think we should spend some time examining the benefits from the A72 upgrade. Then, breaking down the time spent on each step of the VM lifecycle should be fairly straightforward (annotate EL changes, capture numbers with perf or something similar).
- vardump 7y ago
  I have two 4 GB RPi4s, and can run any tests if you like. Or is this about some 1GB barrier DMA limitation? I did notice VideoCore can only access maximum 1GB of RAM. Related to that?
  - _ananos_ 7y ago
    the issue about the available RAM is this: https://github.com/raspberrypi/linux/commit/cdb78ce891f6c6367a69c0a46b5779a58164bd4b#diff-634f284364ba43ef69912111615b08ef https://github.com/raspberrypi/linux/commit/cdb78ce891f6c636... probably some kind of address mapping, but I'm no expert on this stuff ;-)
    - vardump 7y ago
      Saw that, but that doesn't really tell why, and how much effort is required to solve this. Maybe SD stuff is in the VideoCore side, and it simply can't DMA above 1 GB limit, and requires to have below 1GB DMA buffers + copy for transfers to upper memory. Maybe.

anfractuosity 7y ago

Very interesting! When running KVM on the Pi, does that use virtualisation extensions if the processor has them? (I don't know anything about ARM and virtualisation).

_ananos_ 7y ago
yeap, that's the case with the older models too. A53 has also virtualisation extensions. The main difference about the Pi4 is the GIC which removes the need for emulating interrupt handling. Other than that, the A72 handles VMEnters/VMExits by trapping in/out of EL0/EL1/EL2.

Vogtinator 7y ago

Currently u-boot does not support the RPi4 properly (some hardcoded registers and clock numbers), but patches are pending. So EFI support as required by Arch, openSUSE and some others is not available yet. Progress for actual mainline support can be followed here: https://github.com/lategoodbye/rpi-zero/tree/bcm2838-initial https://github.com/lategoodbye/rpi-zero/tree/bcm2838-initial

qalmakka 7y ago

I hope they're going to figure out AArch64 support for the Raspberry Pi soon, especially since they have launched the models with larger amounts of RAM. Having a true AArch64 Raspberry Pi would be really nice.

Vogtinator 7y ago
That's what the article is about. Also openSUSE (and SLES) run in 64 bit mode on a RPi 3 by default without issues, using a mainline kernel and FOSS userspace (no proprietary /opt/vc stuff).
kurtisc 7y ago
Shortly after the Pi 4 was released I looked into it and found that, for the Pi 3 at least, the foundation weren't really interested. It essentially adds another distro for them to maintain. It's a shame.
- floatboth 7y ago
  It's not like anyone is forced to use their distro. It's newbie-oriented and extremely annoying for Unix geeks anyway (Debian base with its always-outdated packages, etc) You don't even have to run Linux at all. FreeBSD/aarch64 works great on a Pi3 (as a headless server at least, VC4 is not ported)
  - 0815test 7y ago
    How is Debian annoying for Unix geeks? If outdated packages are an issue, you can always install stable backports, or stuff from testing/unstable.
  - kurtisc 7y ago
    True but the newbies will slowly drift away from all the projects that don't want to split focus from aarch64.
_ananos_ 7y ago
actually, the host OS for this post is a ubuntu 18.04.2 server image. root@pi:~# cat /etc/issue.net Ubuntu 18.04.2 LTS root@pi:~# uname -a Linux pi 4.19.57-v8+ #2 SMP PREEMPT Tue Jul 9 20:31:37 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux root@pi:~# file /bin/ls /bin/ls: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-, for GNU/Linux 3.7.0, BuildID[sha1]=de05fcef79d88af9cf9a71ed38e73af0b179bfb2, stripped

LeonM 7y ago

There seems to be some work-in-progress with supporting AArch64 for the rPi. This looks like a fun FOSS subject to contribute to once I get my hands on a Pi4. I like the low-level stuff. Does anyone here have recommendations where to contribute to?

Y_Y 7y ago
By the way, the 3B+ is armv7 by default, but also works fine as aarch64. That's how I use mine under nixos.

mikorym 7y ago

I know that this is for 64-bit on the new Raspberry Pi specifically, but would the Pi 4 be able to run a Playstation 2 emulator?

derefr 7y ago
Interesting question. Emulating the PS1 is a lot easier on modern hardware than emulating e.g. the SNES, because most of what it does is push polygons, and PS1 games just don’t push very many of them; so even the most anemic mobile GPU core is more than enough to handle the load. I imagine this is still true of the PS2 to an extent. It’s even more just a polygon pusher than the PS1; and the 3D is still generations-old. There’s this forum post linked to on the PCSX2 website: https://forums.pcsx2.net/Thread-Sticky-Will-PCSX2-run-fast-on-my-computer https://forums.pcsx2.net/Thread-Sticky-Will-PCSX2-run-fast-o..., that says the recommended specs are something like: • Intel Core 2 Duo / Core i3 @ 3.2Ghz or faster • Nvidia Geforce 9600GT / 8800GT or better Probably all you need here, to answer this question in theory, is to compare raw benchmark scores between those components and the Pi4’s CPU/GPU. Or, for another angle, here’s a forum post about Play! (an emulator that has aarch64 support) running on the Nintendo Switch’s Tegra X1: https://gbatemp.net/threads/play-ps2-emulator-is-running-on-the-nintendo-switch.538600/ https://gbatemp.net/threads/play-ps2-emulator-is-running-on-... Spoilers: it gets 10FPS. And the Pi4’s GPU is no Maxwell. (And, while Play! isn’t the most optimized of emulators, it’s the one you’d have to use on aarch64. Even if you optimized it 6x, so that the Tegra got 60FPS, the Pi4’s Videocore GPU wouldn’t be pulling nearly that.)
- davesmith1983 7y ago
  It about how accurate the emulator is. ePSXE used to run fine on a 700mhz PIII with 128gb of ram. There were SNES emulators that also worked fine. However they were inaccurate and lots of games weren't exactly great. MGS for example didn't work well on ePSXe for years because it used many more hardware tricks than same the original ridge Racer. Ridge Racer R4 doesn't even work properly on a PS2 backwards PS1 compatibility because the PS1 on a chip isn't perfect.
  - 7y ago

Havoc 7y ago

>Lightweight virtualization is a natural fit for low power devices No it's not. The pi is punchy enough for it sure, but the above doesn't follow in my mind. Low power = limited resources so ideally you want it run on bare metal to minimise overheads

_ananos_ 7y ago
sure! bare-metal would be awesome. But isn't it a shame not to take advantage of all the virtualization/containerization goodies out there? After all, the virtualization overhead nowadays is only referring to I/O (network & storage). CPU/MEM are being virtualized using hardware extensions at (near-)native speeds.
- Havoc 7y ago
  >CPU/MEM are being virtualized using hardware extensions at (near-)native speeds. Is that really true for rasp level gear? Haven't tested it but my gut feel tells me the hit is sizable. Anyway - don't let that dissuade you from the mission. Container goodies on a rasp is a grand idea...just don't think it's quite as hit free as the article suggests ;)
  - _ananos_ 7y ago
    definitely! that's our initial goal. Quantify the penalty and examine the trade-offs. Clearly, virtualizing workloads on such devices with standard VMMs/hypervisors isn't ideal. And we're working towards this direction; playing with the systems stack is what makes us tick, so, it will be a fun and (hopefully) useful adventure :D
    - Havoc 7y ago
      >Quantify the penalty and examine the trade-offs. If you do have numbers available I'd love to see a write-up on what the real world hit is. I used to run a rasp3b for home server but that just wasn't punchy enough (crappy fake gigabit etc). So my old gaming laptop became a server w/ docker etc. But itching to justify a rasp 4. Anyway...thanks for exploring 64...thought the 32 on rasp 3 was unfortunate.

londons_explore 7y ago

I thought the whole point of things like docker is they gave native performance? Isn't docker just a combination of cgroups, network namespaces, pid namespaces, fancy filesystem mounts, etc. The overhead should be zero by design. What am I missing?

_ananos_ 7y ago
there is a number of factors to consider when comparing this kind of technologies. Linux containers provide native performance for almost all applications, true; but there are tons of implications when it comes to multi-tenancy (security, QoS, trusted execution etc.). Moreover, spawning an application as a docker container can incur significant overhead on startup time, fs setup, FS access etc. So in theory yes, containers provide native performance. But do we really want to run apps on multi-tenant edge devices as containers ? I would argue not necessarily ;)