3 ms·

After this latest incident I brought it up to my company that we should start considering alternatives if this situation doesn't improve. From an outsiders pers

After this latest incident I brought it up to my company that we should start considering alternatives if this situation doesn't improve. From an outsiders perspective, it looks like all the new features they recently introduced seem to be crippling their databases. Microsoft needs to get a grip on this situation as the parent company. Their golden goose acquisition is about to become persona non grata if these outages continue.

anaccountexists 5y ago
My company uses the same DB sharding tech. It took us about a year of daily / weekly outages until we finally were able to fix our performance issues. 256 database splits, lots of cross-shard queries removed, etc before we finally reached a happy-ish state for a year. Now it’s scheduled to fall over in about 6 months and everyone is freaking out again. It’s not new features that are hurting us, it’s the existing core product line. All the new stuff is built on horizontally sharded DBs from the get-go.
- reasonabl_human 5y ago
  Wow. What kind of tech debt are you reckoning with / what better design decisions would you implement if redesigning today?
gtirloni 5y ago
How are you doing cost analysis between moving off of GitHub and working around the issues?
nijave 5y ago
Used to work at BigBank$ where they had an internal 8 node Bit bucket cluster. Down multiple times a week for 6+ months before they could finally get it under control (Jira + CI was apparently putting a lot of load on and apparently BitBucket doesn't scale past 8 nodes) Needless to say, self hosting can be more reliable but you'll probably end up dedicating a lot of resources to building that out and supporting it
- cyberpunk 5y ago
  Single VM (backed up) with gitlab, k8s for workers. Not difficult, not hard to maintain. Worked fine for our ~500 projects (each with pipelines/ci|cd) and ~100 devs.
- MrJohz 5y ago
  I've never tried to self-host something like that, but on the other hand, I've never worked at a place where the internal GitLab wasn't a shaky mess, particularly when CI or Pages or certificates is involved. I believe people when they tell me they've managed it without major problems, but clearly they're the 10x developers of the self-hosted world, because the people I work with seem to find it hard - and that is not a slight on them, because clearly even Github can't do so much better!
- bberenberg 5y ago
  Great talk on this topic from 2015 by the team from Netflix: https://www.youtube.com/watch?v=aWcKsX0_qAg https://www.youtube.com/watch?v=aWcKsX0_qAg

anaccountexists 5y ago

My company uses the same DB sharding tech. It took us about a year of daily / weekly outages until we finally were able to fix our performance issues. 256 database splits, lots of cross-shard queries removed, etc before we finally reached a happy-ish state for a year. Now it’s scheduled to fall over in about 6 months and everyone is freaking out again. It’s not new features that are hurting us, it’s the existing core product line. All the new stuff is built on horizontally sharded DBs from the get-go.

reasonabl_human 5y ago
Wow. What kind of tech debt are you reckoning with / what better design decisions would you implement if redesigning today?

gtirloni 5y ago

How are you doing cost analysis between moving off of GitHub and working around the issues?

nijave 5y ago

Used to work at BigBank$ where they had an internal 8 node Bit bucket cluster. Down multiple times a week for 6+ months before they could finally get it under control (Jira + CI was apparently putting a lot of load on and apparently BitBucket doesn't scale past 8 nodes) Needless to say, self hosting can be more reliable but you'll probably end up dedicating a lot of resources to building that out and supporting it

cyberpunk 5y ago
Single VM (backed up) with gitlab, k8s for workers. Not difficult, not hard to maintain. Worked fine for our ~500 projects (each with pipelines/ci|cd) and ~100 devs.
MrJohz 5y ago
I've never tried to self-host something like that, but on the other hand, I've never worked at a place where the internal GitLab wasn't a shaky mess, particularly when CI or Pages or certificates is involved. I believe people when they tell me they've managed it without major problems, but clearly they're the 10x developers of the self-hosted world, because the people I work with seem to find it hard - and that is not a slight on them, because clearly even Github can't do so much better!
bberenberg 5y ago
Great talk on this topic from 2015 by the team from Netflix: https://www.youtube.com/watch?v=aWcKsX0_qAg https://www.youtube.com/watch?v=aWcKsX0_qAg