Castles were less useful because you didn't need internal military structure to defend against the people from one province over. Roman cities for 100s of years

Castles were less useful because you didn't need internal military structure to defend against the people from one province over. Roman cities for 100s of years didn't upgrade their city walls, even communities in Gaul stopped living in hill cities and so on. Defensive structure really only made sense in specific border regions. There however large empires certainty did invest in such defenses. Border cities that were ready to be under siege and that could host an arriving army.