2 ms·
> were the logs really the best option for getting that "seed" data? It seemed to us like they were. The database under test belongs to a large monolithic app
by lawrjone 8y ago
> were the logs really the best option for getting that "seed" data?
It seemed to us like they were. The database under test belongs to a large monolithic app that has long reads, heavy write batch jobs and is serving API traffic. My team and I struggled to find an alternative that we didn't feel was too fake, that we could really have confidence in.
Even though the preparation of the benchmark loses some realism, it's still close enough to real life that it gives you great signal on how things might degrade. Running this replay against a new cluster and simultaneously verifying the performance of a critical endpoint was the gold standard for verifying we were safe to move to the new hardware.
Whether this is all true for others, I'm really not sure. We were willing to invest the time because this was so critical for us, but it might not make sense for other people to take the same approach.
> I wonder if some of the work done facilitating logical replication
Perhaps? You could capture writes, but the majority of our traffic were reads which would have been unreported by logical replication.
> getting a load testing data set
Anecdotally, capturing the logs seemed quite easy. Just turn Postgres logging up to max and shove them on a different disk. That said, if we weren't in GCP with quick disk snapshotting then things would be a lot more awkward!
- sbuttgereit 8y ago> Perhaps? You could capture writes, but the majority of our traffic were reads which would have been unreported by logical replication. Ah yes... of course.... forget I mentioned it... I do a lot of ERP type work so am in a very similar boat most of the time, but I'm currently doing the information architecture for a green field development project and am very focused on some other priorities, so a little work induced myopia on my part.