A few years ago our team rewrote a critical part of our system in a more resource efficient language to cut a large chunk out of our AWS bill. The system handl

A few years ago our team rewrote a critical part of our system in a more resource efficient language to cut a large chunk out of our AWS bill. The system handled around 500K requests per minute. The inputs/outputs of the system looked like this. 1. Receive HTTP request 2. Make a bunch of HTTP requests - Call external web services for business logic - Call reporting services for analytics 3. Render result On a single production box of the legacy system, we instrumented the http clients to dump the requests/replies to disk for 15 minutes. Using the snapshots, we wrote some scripts to allow us to replay the data through the new system. It was a totally ad hoc manual process that took about a week. It found a TON of bugs and greatly increased our confidence in the new system. After we were done we threw it away. I am convinced that using standard unit testing alone would not have found many of these bugs and caused them to surface in production.