3 ms·

Linked in the article is this other one, "Partitioning GitHub’s relational databases to handle scale" (https://github.blog/2021-09-27-partitioning-githubs-relat

Linked in the article is this other one, "Partitioning GitHub’s relational databases to handle scale" (https://github.blog/2021-09-27-partitioning-githubs-relational-databases-scale/ https://github.blog/2021-09-27-partitioning-githubs-relation...). That describes how there isn't just one "main primary" node; there are multiple clusters, of which `mysql1` is just one (the original one — since then, many others have been partitioned off).

throwusawayus 5y ago
from that article it sounds like they are mostly doing "functional partitioning" (moving tables off to other db primary/replica clusters) rather than true sharding (splitting up tables by ranges of data) functional partitioning is a band-aid. you do it when your main cluster is exploding but you need to buy time. it ultimately is a very bad thing, because generally your whole site is depenedent on every single functional partition being up. it moves you from 1 single point of failure to N single points of failure!
- wincent 5y ago
  Towards the end: > In addition to vertical partitioning to move database tables, we also use horizontal partitioning (aka sharding). This allows us to split database tables across multiple clusters, enabling more sustainable growth.
  - throwusawayus 5y ago
    yes i know, the fact that this is a small blurb at the bottom of the article (which is largely about functional partitioning) exactly proves my point
- dikei 5y ago
  >> functional partitioning is a band-aid. I disagree, functional partitioning is not a band-aid, but an architectural changes that in the end can reap much more benefit than simple data sharding. >> your whole site is dependent on every single functional partition being up. it moves you from 1 single point of failure to N single points of failure! Not necessarily, it can also be that only some parts of your site are dead while others work perfectly fine.
  - throwusawayus 5y ago
    too idealistic. invariably some team (or usually many teams) don't properly gate some critical-path logic, they depend on some functional partition always being online and then boom much larger blast radius than intended then they fix it in post-mortem but pattern just repeats. i have seen it so many times! used to be much worse in the earlier days of the cloud when VMs would go poof more often

throwusawayus 5y ago

from that article it sounds like they are mostly doing "functional partitioning" (moving tables off to other db primary/replica clusters) rather than true sharding (splitting up tables by ranges of data) functional partitioning is a band-aid. you do it when your main cluster is exploding but you need to buy time. it ultimately is a very bad thing, because generally your whole site is depenedent on every single functional partition being up. it moves you from 1 single point of failure to N single points of failure!

wincent 5y ago
Towards the end: > In addition to vertical partitioning to move database tables, we also use horizontal partitioning (aka sharding). This allows us to split database tables across multiple clusters, enabling more sustainable growth.
- throwusawayus 5y ago
  yes i know, the fact that this is a small blurb at the bottom of the article (which is largely about functional partitioning) exactly proves my point
dikei 5y ago
>> functional partitioning is a band-aid. I disagree, functional partitioning is not a band-aid, but an architectural changes that in the end can reap much more benefit than simple data sharding. >> your whole site is dependent on every single functional partition being up. it moves you from 1 single point of failure to N single points of failure! Not necessarily, it can also be that only some parts of your site are dead while others work perfectly fine.
- throwusawayus 5y ago
  too idealistic. invariably some team (or usually many teams) don't properly gate some critical-path logic, they depend on some functional partition always being online and then boom much larger blast radius than intended then they fix it in post-mortem but pattern just repeats. i have seen it so many times! used to be much worse in the earlier days of the cloud when VMs would go poof more often