I used to do this at Napster, albeit with far less computing resources. I would record traffic to a machine and then play it back at 10-20x realtime to see wher

I used to do this at Napster, albeit with far less computing resources. I would record traffic to a machine and then play it back at 10-20x realtime to see where my performance limits were using a few machines. Of course, it turned out that most of the limits we ran into were hardware related (the max packets per second Cisco switches advertised was... optimistic). Of course, I also just tested in production with real users. I'd redirect more or less users to a particular server to stress it and see where it broke. Since we controlled the protocol and the client, I could hide any problems pretty easily. Users didn't seem to care if a for a couple minutes the service was a little wonky. As Twitter has demonstrated, if people love your service, they will forgive some minor disruptions from time to time. At our peak, we were at 2.3 million simultaneous users, but we had nowhere near the bandwidth requirements of MySpace. It wasn't until recently with the rise of continuous deployment did I think that testing in production was an ok thing to do. I did it because I had to.