4 ms·

My guess at what happened: They had a split brain when multiple masters were running. Then they were not able to choose a master to keep because the data in bo

My guess at what happened: They had a split brain when multiple masters were running. Then they were not able to choose a master to keep because the data in both masters was 'corrupted' so they are now restoring from a backup. So how do they get data corruption from multiple masters running: 1) Performing reads from slaves during an update operation. If you perform a read from a slave then you might get data from the other master. If you update data on one master based on data from another master then you get data corruption. Probably they don't do this much because if you had any slave lag then you would notice this problem during normal operation. However, they might do it for checking permissions. You can imagine because this data almost never changes it would never show up as a problem normally. 2) Data stored outside of the database. This would be the repositories themselves and cookie-session storage. Imagine repos have an incrementing id. Then if you have two masters a repo gets created with the same id on both masters. This is very bad because now two people can see each others data. You have the same problem with cookies. Imagine if you have user id as an incrementing id. The two masters create a user with the same id and an encrypted cookie or another storage system (redis) stores the user id. Now depending on which master you get routed to you appear as a different user. Weirdly enough this would always be a problem based on how their failover system works. The only safe way i know how to turn a master-slave system into a safe HA system is how joyent does it (https://github.com/joyent/manatee/blob/master/docs/user-guide.md https://github.com/joyent/manatee/blob/master/docs/user-guid... : you basically need a vote at commit time. if you have this then only 1 master can commit). However, I'm guessing most of the time they only have two masters running for < 1 minute but probably this time they had two masters running for a long time. EDIT: oh. they said it affected issues & pull requests [they have different dbs for different stuff] so repo and authentication stuff wouldn't apply. oh, they said no data was lost as well so presumably that would exclude a split brain running for a long period of time.

sudhirj 8y ago
Split brain can run for a while if the all identifiers are UUIDs and tables are used as append only. Restoration is complex, though.
- benmmurphy 8y ago
  depending on your application logic you can get external inconsistencies. like your typical CRUD app might dump the state of an entity in an edit form, then a user might edit one field, and the whole form is sent to the backend and written over whatever exists there. if you have two different masters then you can get a series of updates where you can't tell if the update is because the user meant to make the change or if it was because the change was incorrectly propagated from the other master. like even if you knew the user read from the other master you can't tell if the user intended the change or not. maybe they saw the value was the value they wanted so they didn't change it. i guess this is a 'bug' because when you do stuff like this you should include some kind of version identifier to catch a concurrent update in the normal case. this scenario is also 'storing stuff outside the DB' kind of similar to the user_id in the cookie getting out of sync.

sudhirj 8y ago

Split brain can run for a while if the all identifiers are UUIDs and tables are used as append only. Restoration is complex, though.

benmmurphy 8y ago
depending on your application logic you can get external inconsistencies. like your typical CRUD app might dump the state of an entity in an edit form, then a user might edit one field, and the whole form is sent to the backend and written over whatever exists there. if you have two different masters then you can get a series of updates where you can't tell if the update is because the user meant to make the change or if it was because the change was incorrectly propagated from the other master. like even if you knew the user read from the other master you can't tell if the user intended the change or not. maybe they saw the value was the value they wanted so they didn't change it. i guess this is a 'bug' because when you do stuff like this you should include some kind of version identifier to catch a concurrent update in the normal case. this scenario is also 'storing stuff outside the DB' kind of similar to the user_id in the cookie getting out of sync.