I have built one of these map rendering systems on EC2. We decided to go with a 100% on-the-fly + two caching layers (one HTTP on top and one application-specif

I have built one of these map rendering systems on EC2. We decided to go with a 100% on-the-fly + two caching layers (one HTTP on top and one application-specific in between [TileCache]). There were dozens of layers (now there are thousands) so pre-rendering all of them was not feasible. Since I left that gig, the current team has added some processes to "warm" the caches before new data goes live. It just takes a few minutes though, nowhere near 2% of the tilespace. From what I remember, the biggest performance trick was figuring out how to properly pack worldwide street data (geometry, categorization, and label) + indexes into RAM on a single machine without using a custom file format. It involved stripping out every little unnecessary byte and sorting all of the streets geographically to improve access locality. I believe this got down to ~15GB of shapefiles.