4 ms·
Resharding in vitess is not quite as magical as this post implies. My infrastructure team has tried resharding the cluster powering my app several times and eac
by Misdicorl 5y ago
Resharding in vitess is not quite as magical as this post implies. My infrastructure team has tried resharding the cluster powering my app several times and each time it's failed.
It's been a while since the last attempt so I forget the details but I think the reshard replication lag was growing no matter what resources we threw at it. We've since scaled vertically instead and are working on architecture changes to reduce our MySQL load (which admittedly is extremely high).
- dkhenry 5y agoI would encourage you to join the vitess slack channel and let them know of any issues you run into there. Resharding has successfully been done at massive scale, so its unlikely that the system can't handle the load. Here is the blog post from when Github re-sharded live under full github load https://github.blog/2021-09-27-partitioning-githubs-relational-databases-scale/ https://github.blog/2021-09-27-partitioning-githubs-relation...
- samlambert 5y agoAgreed. Just to add to this we’ve seen reshards on live datasets that are over 10 petabytes in size.
- Misdicorl 5y agoThanks, I'll pass along to my infrastructure team. Glancing through the blog post, its quite light on details, and I wonder if we just have a very different data load pattern. Our dataset size is actually reasonably small (10s of TB), but our transaction throughput is very high. Glancing at our dashboard, baseline is 250k/s, with sustained daytime load in the 500k/s region.