Between Cat IV and Cat V languages, colonial influence and other cultural factors will probably outweigh any difference between the languages as you said. But

Between Cat IV and Cat V languages, colonial influence and other cultural factors will probably outweigh any difference between the languages as you said. But I wouldn't be surprised if, as a broad generalization, people who speak Cat I languages find it much easier to learn English than people who speak Cat V languages do.