You probably don't even need to reverse anything. Just learn how other (open source) toolchains work, read N64 manuals and information from other reverse engine

You probably don't even need to reverse anything. Just learn how other (open source) toolchains work, read N64 manuals and information from other reverse engineers, and work incrementally. It's probably not necessary to reverse this proprietary ROM you're stuck with.