4 ms·

How do you get realistic data into these environments? Our production has 10TB of database data and 20M files uploaded by customers. We regularly see devs writ

How do you get realistic data into these environments? Our production has 10TB of database data and 20M files uploaded by customers. We regularly see devs write queries or nested loops that work fine on their tiny dev system but explode when run against realistic customer data. Right now we go through a 12 hour restore then anonymization pass to keep staging up to date. Very painful. Who has a better way?

tharkun__ 5y ago
YMMV as always, i.e. this might be faster/easier to implement for some use cases/companies than others. I would argue that for most changes in most companies it does not matter if you have a full data set equivalent of Prod. Especially since it's an ever growing target (hopefully for you :)). As we can see from your use case, that can create challenges. If we were testing every little change with a complete replica of our Prod environment we'd spend a lot more money on this and would also have to wait way too long for these environments to come up. This might then drive us towards keeping a small set of staging environments running all the time and share them. Now I don't know your specific field, what kind of system you provide, what the cost of failures in Prod would be, what kind of guards you have in your systems against a complete outage like that (e.g. can a bad query running for one customer take out your entire product or will it be an isolated incident to that one customer or a small group of customers?) etc. If so maybe you first want to isolate failures between customers more. FWIW, I also see devs write these kinds of queries and code and we catch most of them in PRs and most people learn from this and next time they don't write this kind of query any longer. But let's say they do and we don't catch them. Can you analyze your data set and figure out a small enough example that is not simply an ever growing anonymized version of actual Prod data that would exhibit the same catastrophic characteristics for most queries these devs might write? I'm pretty sure the kinds of queries that fail on 10TB of data w/ 20M files fail just as spectacularly on 9TB and 19M files? Have you tried if they also fail 'badly enough' on 500GB of data and 1M files? How fast would it be to restore a fixed data set like that? Maybe refresh it every month from Prod if that's needed for some reason or another. A lot of what you might actually want to try out in the end will depend on your exact system set up I would argue and it's hard to give general advice that will definitely fit.
- tatersolid 5y ago
  We have a cut-down but “production shaped” dataset for fast environment creation but even with code reviews it hasn’t been successful at catching the O(n^2) unindexed query or accidental O(scary) at the app tier caused by a refactor. Or the data migration which wasn’t batched but needs to be. Developer turnover is only accelerating, with predictable results. At this point we are looking into volume snapshots as a potential speedup for creation of a full-sized per-release staging environments but that still leaves the problem of generating a realistic multi-tenant customer load to solve and maintain over time.
  - nprateem 5y ago
    Catch the 90% of stuff that works with your cut down dataset. The other 10% includes things you might only catch not only with O(n^2) queries over a 10TB, but also when under prod load like you say. Just either gradually roll out those high-risk changes with feature flags, etc. or make sure your monitoring is up-to-scratch so you can catch issues and fix them.
    - tharkun__ 5y ago
      Totally seconding the 80/20 rule here. It's probably less frustrating and faster for everyone if you can work on compartmentalizing failures that do slip through or only happen for those two customers with very special data sets that you most probably wouldn't catch in testing either because your test suite won't execise the full 10TB data set anyway (assuming here because we don't know your product)

tharkun__ 5y ago

YMMV as always, i.e. this might be faster/easier to implement for some use cases/companies than others. I would argue that for most changes in most companies it does not matter if you have a full data set equivalent of Prod. Especially since it's an ever growing target (hopefully for you :)). As we can see from your use case, that can create challenges. If we were testing every little change with a complete replica of our Prod environment we'd spend a lot more money on this and would also have to wait way too long for these environments to come up. This might then drive us towards keeping a small set of staging environments running all the time and share them. Now I don't know your specific field, what kind of system you provide, what the cost of failures in Prod would be, what kind of guards you have in your systems against a complete outage like that (e.g. can a bad query running for one customer take out your entire product or will it be an isolated incident to that one customer or a small group of customers?) etc. If so maybe you first want to isolate failures between customers more. FWIW, I also see devs write these kinds of queries and code and we catch most of them in PRs and most people learn from this and next time they don't write this kind of query any longer. But let's say they do and we don't catch them. Can you analyze your data set and figure out a small enough example that is not simply an ever growing anonymized version of actual Prod data that would exhibit the same catastrophic characteristics for most queries these devs might write? I'm pretty sure the kinds of queries that fail on 10TB of data w/ 20M files fail just as spectacularly on 9TB and 19M files? Have you tried if they also fail 'badly enough' on 500GB of data and 1M files? How fast would it be to restore a fixed data set like that? Maybe refresh it every month from Prod if that's needed for some reason or another. A lot of what you might actually want to try out in the end will depend on your exact system set up I would argue and it's hard to give general advice that will definitely fit.

tatersolid 5y ago
We have a cut-down but “production shaped” dataset for fast environment creation but even with code reviews it hasn’t been successful at catching the O(n^2) unindexed query or accidental O(scary) at the app tier caused by a refactor. Or the data migration which wasn’t batched but needs to be. Developer turnover is only accelerating, with predictable results. At this point we are looking into volume snapshots as a potential speedup for creation of a full-sized per-release staging environments but that still leaves the problem of generating a realistic multi-tenant customer load to solve and maintain over time.
- nprateem 5y ago
  Catch the 90% of stuff that works with your cut down dataset. The other 10% includes things you might only catch not only with O(n^2) queries over a 10TB, but also when under prod load like you say. Just either gradually roll out those high-risk changes with feature flags, etc. or make sure your monitoring is up-to-scratch so you can catch issues and fix them.
  - tharkun__ 5y ago
    Totally seconding the 80/20 rule here. It's probably less frustrating and faster for everyone if you can work on compartmentalizing failures that do slip through or only happen for those two customers with very special data sets that you most probably wouldn't catch in testing either because your test suite won't execise the full 10TB data set anyway (assuming here because we don't know your product)