2 ms·

We have a cut-down but “production shaped” dataset for fast environment creation but even with code reviews it hasn’t been successful at catching the O(n^2) uni

We have a cut-down but “production shaped” dataset for fast environment creation but even with code reviews it hasn’t been successful at catching the O(n^2) unindexed query or accidental O(scary) at the app tier caused by a refactor. Or the data migration which wasn’t batched but needs to be. Developer turnover is only accelerating, with predictable results. At this point we are looking into volume snapshots as a potential speedup for creation of a full-sized per-release staging environments but that still leaves the problem of generating a realistic multi-tenant customer load to solve and maintain over time.

nprateem 5y ago
Catch the 90% of stuff that works with your cut down dataset. The other 10% includes things you might only catch not only with O(n^2) queries over a 10TB, but also when under prod load like you say. Just either gradually roll out those high-risk changes with feature flags, etc. or make sure your monitoring is up-to-scratch so you can catch issues and fix them.
- tharkun__ 5y ago
  Totally seconding the 80/20 rule here. It's probably less frustrating and faster for everyone if you can work on compartmentalizing failures that do slip through or only happen for those two customers with very special data sets that you most probably wouldn't catch in testing either because your test suite won't execise the full 10TB data set anyway (assuming here because we don't know your product)