5 ms·
You read about these "20 deploys a day" type situations, and it sounds great, and I'm sure it makes VCs all warm and creamy inside. But they're talking about s
by jfm3 16y ago
You read about these "20 deploys a day" type situations, and it sounds great, and I'm sure it makes VCs all warm and creamy inside. But they're talking about small changes, and not all deploys are small. You can't incrementally change a database engine, for example.
What I would be more impressed by is if they could run tests against a full load of real user input, and have useful/reliable metrics come out the other end. There's no reason for the deploy to fail if you have a real mechanism for testing the deploy. I've yet to hear of a shop of the size of an Etsy that does real production/load testing.
- steveklabnik 16y agoIMVU. http://timothyfitz.wordpress.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/ http://timothyfitz.wordpress.com/2009/02/10/continuous-deplo...
- jfm3 16y ago"[...] 4.4 machine hours of automated tests to be exact. Over an hour of these tests are [...] Selenium. The rest of the time is [...] unit tests." Impressive. But this not production test, where you tee the actual input from users into the system. Nor is it load test, where you measure whether or not the site is performant under real load. For that matter, there's nothing here that indicates you debug your operations (provision/deploy/backup) at all. My comment about not seeing such an operation stands.
- steveklabnik 16y agoRead a bit farther. > Load average, cpu usage, php errors and dies and more are sampled by the push script, as a basis line. ... A minute later the push script again samples data across the cluster and if there has been a statistically significant regression then the revision is automatically rolled back.
- jfm3 16y agoPrecisely so, but that's not testing, it's repair. You've just rolled back from a production problem automatically. The user still saw the problem. Sorry if this seems rude. You're definitely way ahead of the game, but you're not addressing the thing that I'm talking about.
- steveklabnik 16y agoNone taken. I don't work for IMVU.
- ericb 16y agoI am working on a service that will offer load testing using ruby scripts. One of the main goals is to enable continual load testing. I'm building out the rest interface now. If anyone wants to be an early adopter, drop me a line.