FPGA ISP i am using with this project is improved version of ISP that i made for lattice machxo3 FPGA , those FPGA like most FPGAs do not have any MIPI hard PHY

FPGA ISP i am using with this project is improved version of ISP that i made for lattice machxo3 FPGA , those FPGA like most FPGAs do not have any MIPI hard PHY. If you want to port this ISP to Xilinx you would not find hard PHY in many FPGAs and you would need a byte aligner. That is why Byte Aligner was implemented and left enabled in there for the sake of portability to other FPGA it does not hurt (except for may be very very small performance in very edge case or some FPGA resource consumption). I had many issues with Crosslink NX part. I never specifically got/or noticed the issue you have mentioned.