Yes, Wayland compositors like cage and sway can be nested too. That said, with both the X nesting approach and the Wayland nesting approach, you'd also need to

Yes, Wayland compositors like cage and sway can be nested too. That said, with both the X nesting approach and the Wayland nesting approach, you'd also need to run the screencasting application itself inside the nested server, not the just the application you want to cast. If the compositor supports a way to create headless outputs (as sway and hyprland do) that is much easier.