Practically speaking modern Ethernet switches are non blocking though which allows every port to talk to every other port at full speed. With DOCSIS being TDMA

Practically speaking modern Ethernet switches are non blocking though which allows every port to talk to every other port at full speed. With DOCSIS being TDMA there isn't a way to improve speed to the backbone (other than adding more channels / smaller time slots). Also 3-5ms is probably at the low end since it heavily depends on the usage patterns of those on the same CMTS as you.