I would say the biggest problem of CGNAT is that it is essentially double-NAT: your home router did one layer of NAT, and the ISP also did another layer of NAT

I would say the biggest problem of CGNAT is that it is essentially double-NAT: your home router did one layer of NAT, and the ISP also did another layer of NAT on the edge that is close to your home, not only the latency could add up (although so far it is not a problem in general), but also that another point of failure to be concerned too. I happen to come across this having CGNAT in my parent's house, luckily they have a backup IPv6 so I can access it from remote "directly".