3 ms·
"Nobody needs IOPS until they need IOPS" I've had to scrub a multi terabyte database of PII before moving to a staging environment, it hurts. With modern data
by bearjaws 2y ago
"Nobody needs IOPS until they need IOPS"
I've had to scrub a multi terabyte database of PII before moving to a staging environment, it hurts.
With modern data architectures, you may write the same data 4-5 times in its life cycle, staging data, data warehouses, marketing, PowerBI, Looker, etc...
Especially reporting solutions, where they may aggregate massive amounts of data and write it to temp tables.
It will require IOPS, and you will pay handsomely for it.
- bastawhiz 2y agoI don't know about your specific example, but doing an operation where you rewrite most of a multi terabyte database online is almost certainly not best accomplished with SELECT/UPDATE. Even if you need multiple passes, that's N terabytes times M passes times two. That's... not a lot of reads and writes. Dumping the database to files on blob storage, rewriting them, then reading them into your destination is almost certainly the fastest and cheapest way to go about that. And that's not an iops avoidance thing, that's a "this isn't what your database is built to do with the configuration your running it in" sort of thing.