Larger blocks means that they take longer to propagate across the network, meaning there is a much longer period where two or more miners might find alternative

Larger blocks means that they take longer to propagate across the network, meaning there is a much longer period where two or more miners might find alternative "next blocks", which could be exploited by tricking miners into including a different transaction in each split of the chain. This means that you basically have to wait much longer before you can be sure that your transaction will not be reversed. On top of that, larger blocks also means that home users, or small businesses can no longer run their own fully verifying nodes, meaning that we have to move to relying on larger businesses to take care of attacks on the network, and not to collude to change the ruleset. In reality, Bitcoin Cash and Bitcoin SV have not taken off in popularity, due to these issues as well as serious character flaws in their most vocal proponents, meaning that for now their blocks remain small, and the possibility of these attacks isn't actually realized. But if they were to become more popular it would not last long.