The biggest problem is that in general you need to unlock the bootloader to install custom firmware. On some phones it might be possible to relock the bootloade

The biggest problem is that in general you need to unlock the bootloader to install custom firmware. On some phones it might be possible to relock the bootloader after you flash the firmware, but I think most phones won't let you do this with a custom firmware. Once the bootloader is unlocked, anyone with physical access to the phone can reboot the phone into custom software and run arbitrary code, read data etc. I guess you can still use encryption in case your phone is stolen, but anyone who temporarily gets access to your phone could backdoor it.