The sensible way to secure it would be to have a device behind the router, in an isolated network which is the only network/device allowed to access the managem

The sensible way to secure it would be to have a device behind the router, in an isolated network which is the only network/device allowed to access the management interface. Then you tunnel/vpn/wire guard into that device.