The borrow checker is mainly a problem when the same data needs to be modified in two different places. Only the PPU writes to a frame buffer(that it owns), so

The borrow checker is mainly a problem when the same data needs to be modified in two different places. Only the PPU writes to a frame buffer(that it owns), so it wasn't a problem there. The game controllers were another case: They are mapped in CPU address space, so the CPU needs a permanent mutable reference to them. But the top-level also needs to update them with inputs read from my Xbox 360 controller.