I think there are two big things that can be really hard with scalability - 1) There are a lot of dimensions I think it is fairly common to focus on a through

I think there are two big things that can be really hard with scalability - 1) There are a lot of dimensions I think it is fairly common to focus on a throughput-related dimension like requests per second and test that very thoroughly, while not paying enough attention to other aspects. We have services where, if we were told requests per second were going to go up 100x tomorrow, I wouldn't be too concerned, but if we were told request size were going up 2x, I'd be freaking out. Even when you think you've identified all of the dimensions, there are usually ones you missed and/or they interact in weird and unexpected ways. 2) Performance can appear to be linear when it really isn't So many outages I've seen have resulted from everything being fine until some threshold is hit, at which point it doesn't start to slowly degrade, but instead immediately explodes. Often times due to feedback loops (GC activity in garbage-collected languages often can behave this way) or because some cache overflowed (data that is queried at a high rate no longer fitting in memory on a DBMS comes to mind).