These products were typically known as "WAN emulators" and were pretty commonly used in the exact situation you describe: developing large scale distributed sys

These products were typically known as "WAN emulators" and were pretty commonly used in the exact situation you describe: developing large scale distributed systems that were going to be used over unreliable connections. This functionality is now available in the Linux kernel via netem and is accessible via the `tc` command.