I don't think it's that aarch64 specifically is difficult to build for, it's that adding _any_ additional architecture is a significant upfront amount of work,

I don't think it's that aarch64 specifically is difficult to build for, it's that adding _any_ additional architecture is a significant upfront amount of work, since you need to set up all the dependent platform-specific things for the first time.