Early on the focus was on the hypothetical CISC advantage, hence wanting ‘compressed’ variable-length instructions, even though the ISA hasn't ended up especial

Early on the focus was on the hypothetical CISC advantage, hence wanting ‘compressed’ variable-length instructions, even though the ISA hasn't ended up especially space efficient now that the actual complex stuff is legacy. But as to why their encoding is so difficult, put that down to lack of foresight. x86 was designed before parallel decode became a thing, and the complexity accumulated over time. Intel have had quite a few ISAs and all the famous ones are bad, so there doesn't have to be a good reason beyond that.