4 ms·

I'm the VP of Engineering from PingCAP. As a devoted customer and a big fan of GitHub, it’s devastating to see service interruptions and our business impacted

I'm the VP of Engineering from PingCAP. As a devoted customer and a big fan of GitHub, it’s devastating to see service interruptions and our business impacted by these problems; as the team behind TiDB, we (PingCAP) believe there is something we can do to help solve the database high availability and scalability problem. We would like to propose the database team in GitHub to consider the TiDB platform. TiDB and TiKV can work as a scale-out MySQL and it has been battle-tested in all kinds of hyper-scale scenarios. Additionally, just in case you didn’t do this, we also recommend GitHub considering the Chaos-Mesh project (https://github.com/pingcap/chaos-mesh https://github.com/pingcap/chaos-mesh) to do Chaos Engineering. Chaos Mesh was our internal chaos engineering platform and we open-sourced it on Dec. 31, 2019. It can be used for simulating different kinds of failures including network partition, flaky disk, node outage. You can easily use it to simulate failures in a test environment and confirm that the high availability works as expected.

ithkuil 6y ago
Can it simulate the overflow of a 32-bit autoincrementjng primary key? Should it?
- shenli3514 6y ago
  Do you mean Chaos-Mesh or TiDB? For Chaos-Mesh, it is not a suitable tool for deterministic unit test case (like inserting an overflow value and expect to get an error). For TiDB, it is easy to use `Alter Table` statement to enlarge the length of a integer column with zero impact on the online business.
  - capableweb 6y ago
    Not sure if you're purposefully missing the question here. ithkuil is not asking if you can change the length of the integer column but rather if you somehow would have prevented the issue GitHub saw here with hitting the limit. So something like automatically updating the length, or giving the developers warnings, or alerts of some sort. My guess would be no, as I haven't seen this in the wild. And if the answer is indeed "no", your comment seems again to plug something that wouldn't actually solve the problem GitHub saw here.
  - ithkuil 6y ago
    What capableweb said. Furthermore, I wonder how long does it take to enlarge a pk column using alter table on a database that has a significant fraction of 4 billion rows? EDIT: I'm not arguing that github would not benefit from TiDB; they might be genuinely better of with it on many fronts. I just wonder if it's fair to plug a product this way
    - shenli3514 6y ago
      Thanks for the clarification. There is not automatically way to enlarge the length. And I think that is not the things should be done by the database itself. That's my own opinion. But I think it would be helpful to show a warning to DBA when the allocated value is close to the upper limit. For the following question, the answer is in seconds. To enlarge the column length, TiDB just need to modify the meta data. "how long does it take to enlarge a pk column using alter table on a database that has a significant fraction of 4 billion rows?"
      - capableweb 6y ago
        > we believe there is something we can do to help solve the database high availability and scalability problem > There is not automatically way to enlarge the length > I think that is not the things should be done by the database itself > would be helpful to show a warning to DBA So, in the end, TiDB does nothing to actually solve the problem they experienced but you still feel comfortable offering them your own solution? Sounds like you're a VP of Marketing, not a VP of Engineering. Leaves a very sour taste when spamming a forum like this. I hope your bosses don't read HN and sees your comments plugging TiDB everywhere where it's not relevant.
        neilhan 6y ago
        well said, we will take note; We were just trying to help anything related to MySQL's scalability
      - ithkuil 6y ago
        > TiDB just need to modify the meta data. thanks, that's interesting. So existing data using the previous 32-bit encoded values happily coexist with the new data being written after the migration?
capableweb 6y ago
If you read the published report, you'll see that only one of the issues are actually related to scalability in any way (and most newly created projects learned their lesson of not using auto-incrementing IDs for many reasons). The same goes for many companies, the downtime is not because of any scaling issues but rather maintenance, code deploys or other things where humans are involved. So while the reward of you promoting your product on HN can be high, unless it's very specific and actually solves their need, it just looks spammy.
- jrockway 6y ago
  The big problem that I took away from the availability report was the one where they lost acknowledged writes. They might not need the scaling, but they do need the ability to lose the leader without losing writes. (It would be preferable to reject the writes and become unavailable rather than to acknowledge writes only to discard them minutes later. No external API user can handle the case where "Github acknowledged my writes, and I read them back, but now they're gone". But they can handle "Github returned 'service unavailable' when I made a write".)
- shenli3514 6y ago
  Thanks for your comment. Yes, there is only one of the issues are actually related to scalability. So I mentioned Chaos-Mash as well. Hope it would be helpful to test large scale distributed system. Both of them can not solve all the issues. But I hope they would be helpful for some of the issues.

ithkuil 6y ago

Can it simulate the overflow of a 32-bit autoincrementjng primary key? Should it?

shenli3514 6y ago
Do you mean Chaos-Mesh or TiDB? For Chaos-Mesh, it is not a suitable tool for deterministic unit test case (like inserting an overflow value and expect to get an error). For TiDB, it is easy to use `Alter Table` statement to enlarge the length of a integer column with zero impact on the online business.
- capableweb 6y ago
  Not sure if you're purposefully missing the question here. ithkuil is not asking if you can change the length of the integer column but rather if you somehow would have prevented the issue GitHub saw here with hitting the limit. So something like automatically updating the length, or giving the developers warnings, or alerts of some sort. My guess would be no, as I haven't seen this in the wild. And if the answer is indeed "no", your comment seems again to plug something that wouldn't actually solve the problem GitHub saw here.
- ithkuil 6y ago
  What capableweb said. Furthermore, I wonder how long does it take to enlarge a pk column using alter table on a database that has a significant fraction of 4 billion rows? EDIT: I'm not arguing that github would not benefit from TiDB; they might be genuinely better of with it on many fronts. I just wonder if it's fair to plug a product this way
  - shenli3514 6y ago
    Thanks for the clarification. There is not automatically way to enlarge the length. And I think that is not the things should be done by the database itself. That's my own opinion. But I think it would be helpful to show a warning to DBA when the allocated value is close to the upper limit. For the following question, the answer is in seconds. To enlarge the column length, TiDB just need to modify the meta data. "how long does it take to enlarge a pk column using alter table on a database that has a significant fraction of 4 billion rows?"
    - capableweb 6y ago
      > we believe there is something we can do to help solve the database high availability and scalability problem > There is not automatically way to enlarge the length > I think that is not the things should be done by the database itself > would be helpful to show a warning to DBA So, in the end, TiDB does nothing to actually solve the problem they experienced but you still feel comfortable offering them your own solution? Sounds like you're a VP of Marketing, not a VP of Engineering. Leaves a very sour taste when spamming a forum like this. I hope your bosses don't read HN and sees your comments plugging TiDB everywhere where it's not relevant.
      - neilhan 6y ago
        well said, we will take note; We were just trying to help anything related to MySQL's scalability
    - ithkuil 6y ago
      > TiDB just need to modify the meta data. thanks, that's interesting. So existing data using the previous 32-bit encoded values happily coexist with the new data being written after the migration?

capableweb 6y ago

If you read the published report, you'll see that only one of the issues are actually related to scalability in any way (and most newly created projects learned their lesson of not using auto-incrementing IDs for many reasons). The same goes for many companies, the downtime is not because of any scaling issues but rather maintenance, code deploys or other things where humans are involved. So while the reward of you promoting your product on HN can be high, unless it's very specific and actually solves their need, it just looks spammy.

jrockway 6y ago
The big problem that I took away from the availability report was the one where they lost acknowledged writes. They might not need the scaling, but they do need the ability to lose the leader without losing writes. (It would be preferable to reject the writes and become unavailable rather than to acknowledge writes only to discard them minutes later. No external API user can handle the case where "Github acknowledged my writes, and I read them back, but now they're gone". But they can handle "Github returned 'service unavailable' when I made a write".)
shenli3514 6y ago
Thanks for your comment. Yes, there is only one of the issues are actually related to scalability. So I mentioned Chaos-Mash as well. Hope it would be helpful to test large scale distributed system. Both of them can not solve all the issues. But I hope they would be helpful for some of the issues.