5 ms·

This is the last straw for me. I'm gonna stop using them to host my code. They have been down way too many times in the last year. It's been six failures from t

This is the last straw for me. I'm gonna stop using them to host my code. They have been down way too many times in the last year. It's been six failures from them in the last two weeks alone. I'm gonna self host Gitea to fix my issues. I cannot believe that they fail so hard. Why does a failure mean that I cannot read AND write from BitBucket ? Why are those two things even related ?

bg4 9y ago
DDoS?
- simlevesque 9y ago
  My self hosted repos won't get DDoS. BitBucket is a large target and that's a problem for me.
mtgx 9y ago
It's likely a DDoS attack against Akamai (again). Github also saw a record-breaking 1.3 Tbps attack recently: https://www.wired.com/story/github-ddos-memcached https://www.wired.com/story/github-ddos-memcached
- simlevesque 9y ago
  That's something that is bound to happen more often. I cannot let that affect me. I cannot let their problems become mine.
  - zbentley 9y ago
    You use the internet. Their problems are already yours. The only situation in which self-hosting or ditching BitBucket/similarly large providers will help protect you from the fallout of historically-large, catastrophic attacks is if you do all of your development on the same local network as your hosted server (and don't rely on any internet services to access it). And even if you diligently self-host every part of your own services (not as easy as just plop a gitlab/gitea install on a host you own and start it up), you have to deal with the fallout from internet-breaking DDoS attacks and other malicious activity if you want to use the internet to run or use your code: from congestion caused by compromised devices in your network "neighborhood" (same/similar ISPs or last-mile providers) to DNS outages to BGP hacks, we have seen time and time again that, if not exactly centralized, the systems that comprise the usable internet are certainly highly interdependent. Large-scale attacks of many kinds compromise them. Instead of tantrums, it might behoove users to understand what kind of SLAs they can promise in order to operate their self-hosted services in such an interdependent environment. Some examples: Do you need local power? A local ISP to be up? More than one? If you have more than one internet link, how do you pair the connections--if it's via BGP, what happens if the central authority on that has issues? If local power is down, does your local ISP's connection stay up? How long does it stay up (is there a node/amp somewhere on the line that cut to battery)? Do you need to access internet services by hostname? If so, do you do local DNS caching? If so, how stale can it get in the event of a loss of external DNS? Most importantly: how much (it's a nonzero number unless you're developing for yourself, by yourself, on your LAN) dependence on external services are you comfortable with, and how much time are you willing to spend eliminating the long tail of such dependencies?
chrisan 9y ago
> Why does a failure mean that I cannot read AND write from BitBucket ? Why are those two things even related ? https://status.aws.amazon.com/ https://status.aws.amazon.com/ If there are network issues then it wont matter if you want to read or write to bitbucket
- 013a 9y ago
  There are ways to engineer on AWS to basically never go down barring massive, systemic, cross-region failure. Granted, there's no economical way you could self-host a Gittea instance to avoid this.
  - ComputerGuru 9y ago
    It’s laughably economical to do so, actually. Just not at GitHub/Bitbucket’s scale.
    - 013a 9y ago
      Github costs $7/mo. Is it more economical than that? Can you seriously say that you can achieve higher cross-region redundancy, for both your instances and storage, than Github can, at a price less than $7/mo? The only way I can think of accomplishing it is to make use of GCP's free f1.micro instance, so you can spin up two of those for pretty cheap in different regions ($3.88/mo). Have DNS hosted somewhere that can resolve to each of your two instances at random ($12/year? Lets's say $1/mo). Then you have instance storage; good luck finding globally redundant block storage for practically unlimited repositories, with backups, for $2.12/mo. Let's just leave network egress charges out of it, since those would be marginal. And let's go ahead and say I value my free time at a conservative $30/hr. It takes me an hour to set this thing up and maintain it every year; horribly conservative. That's an additional $2.50/mo. Maybe you can do it. It isn't laughably economical.
      - ComputerGuru 9y ago
        Economical does not mean “the cheapest option” or even “still cheaper than GitHub”. I have no argument with you.
BoorishBears 9y ago
I’ve been having issues with several services today: - Jira - Ring Central - Github I suspected AWS but don’t see anything on the status page. Isn’t part of the point of a DVCS to not be overly held up by an inability to access your server? I don’t know about your situation but I know that a company like Github can probably do better at SRE than I can with a self-hosted server
- simlevesque 9y ago
  > Github can probably do better at SRE than I can with a self-hosted server GitHub maybe but for BitBucket, I'm not sure at all that it's true.
- Siecje 9y ago
  You can self host JIRA.
  - BoorishBears 9y ago
    I’m aware, and I’d wager most self-hosted Jira instances are behind on updates and stability. One former workplace had to bring in multiple Jira consultants due to poor performance and stability
gtsteve 9y ago
Well if you can't read then you can't see what you're writing to right? But I agree, it's not ideal that Bitbucket has been experiencing issues. I am also looking at alternatives, most likely using AWS CodeCommit and Upsource. It's been at the back of my mind to move away for a while.
udia 9y ago
Relying solely on one platform for your organization's code repository needs is a bad thing in general. I have a workflow where I host my own Gogs instance on a raspberry pi, as well as on Github. Maybe this would be useful to you? I've configured my push such that it will deploy to both Github and on my Gogs meaning that I always have an up to date repository in two places. https://github.com/gogits/gogs https://github.com/gogits/gogs (if you're interested in setting up your own) https://stackoverflow.com/questions/14290113/git-pushing-code-to-two-remotes https://stackoverflow.com/questions/14290113/git-pushing-cod... (push to two remotes) You don't have to use just BitBucket, or rely entirely on your self hosted git service.
- simlevesque 9y ago
  That's what I wanted to do. Thank you for the guide !
- foxylion 9y ago
  I think hosting the repository is one thing. But integrating those two repositories with all the automated workflows is quite a headache. Jenkins won't automatically switch over to another git backend if one fails. Other automated tools like code review are mostly relying on a centralized repostiory. I don't see any simple solution to solve these issues. Bitbucket, GitHub, GitLab don't have any easy fallback solutions when you relying core workflows on their services. Self hosting is maybe one solution (we do this with bitbucket), but this requires major administrative effort to keep it running reliably (always available, no data losses on hardware/software failures).
  - udia 9y ago
    I agree. Having CI with multiple git repositories is still a painful thing. (I continue to rely primarily on GitHub here. My backup repos don't have CI enabled.) I think if your primary repo that's hooked into CI goes down, you're still SOL. Having your own repo just enables you to continue local development among your team. I don't see any great solution, other than making your app distributed to begin with and doing your build/deploy manually. As a complete aside, I've fantasized about deploying to all cloud vendors (Azure/GCE/AWS/Heroku/DigitalOcean/misc.) each with their own specific build/deploy and having persistent state shared with something like CockroachDB. Having some load balancer managing state between all instances. Taking advantage of the free/basic tiers provided by all of the vendors.

bg4 9y ago

DDoS?

simlevesque 9y ago
My self hosted repos won't get DDoS. BitBucket is a large target and that's a problem for me.

mtgx 9y ago

It's likely a DDoS attack against Akamai (again). Github also saw a record-breaking 1.3 Tbps attack recently: https://www.wired.com/story/github-ddos-memcached https://www.wired.com/story/github-ddos-memcached

simlevesque 9y ago
That's something that is bound to happen more often. I cannot let that affect me. I cannot let their problems become mine.
- zbentley 9y ago
  You use the internet. Their problems are already yours. The only situation in which self-hosting or ditching BitBucket/similarly large providers will help protect you from the fallout of historically-large, catastrophic attacks is if you do all of your development on the same local network as your hosted server (and don't rely on any internet services to access it). And even if you diligently self-host every part of your own services (not as easy as just plop a gitlab/gitea install on a host you own and start it up), you have to deal with the fallout from internet-breaking DDoS attacks and other malicious activity if you want to use the internet to run or use your code: from congestion caused by compromised devices in your network "neighborhood" (same/similar ISPs or last-mile providers) to DNS outages to BGP hacks, we have seen time and time again that, if not exactly centralized, the systems that comprise the usable internet are certainly highly interdependent. Large-scale attacks of many kinds compromise them. Instead of tantrums, it might behoove users to understand what kind of SLAs they can promise in order to operate their self-hosted services in such an interdependent environment. Some examples: Do you need local power? A local ISP to be up? More than one? If you have more than one internet link, how do you pair the connections--if it's via BGP, what happens if the central authority on that has issues? If local power is down, does your local ISP's connection stay up? How long does it stay up (is there a node/amp somewhere on the line that cut to battery)? Do you need to access internet services by hostname? If so, do you do local DNS caching? If so, how stale can it get in the event of a loss of external DNS? Most importantly: how much (it's a nonzero number unless you're developing for yourself, by yourself, on your LAN) dependence on external services are you comfortable with, and how much time are you willing to spend eliminating the long tail of such dependencies?

chrisan 9y ago

> Why does a failure mean that I cannot read AND write from BitBucket ? Why are those two things even related ? https://status.aws.amazon.com/ https://status.aws.amazon.com/ If there are network issues then it wont matter if you want to read or write to bitbucket

013a 9y ago
There are ways to engineer on AWS to basically never go down barring massive, systemic, cross-region failure. Granted, there's no economical way you could self-host a Gittea instance to avoid this.
- ComputerGuru 9y ago
  It’s laughably economical to do so, actually. Just not at GitHub/Bitbucket’s scale.
  - 013a 9y ago
    Github costs $7/mo. Is it more economical than that? Can you seriously say that you can achieve higher cross-region redundancy, for both your instances and storage, than Github can, at a price less than $7/mo? The only way I can think of accomplishing it is to make use of GCP's free f1.micro instance, so you can spin up two of those for pretty cheap in different regions ($3.88/mo). Have DNS hosted somewhere that can resolve to each of your two instances at random ($12/year? Lets's say $1/mo). Then you have instance storage; good luck finding globally redundant block storage for practically unlimited repositories, with backups, for $2.12/mo. Let's just leave network egress charges out of it, since those would be marginal. And let's go ahead and say I value my free time at a conservative $30/hr. It takes me an hour to set this thing up and maintain it every year; horribly conservative. That's an additional $2.50/mo. Maybe you can do it. It isn't laughably economical.
    - ComputerGuru 9y ago
      Economical does not mean “the cheapest option” or even “still cheaper than GitHub”. I have no argument with you.

BoorishBears 9y ago

I’ve been having issues with several services today: - Jira - Ring Central - Github I suspected AWS but don’t see anything on the status page. Isn’t part of the point of a DVCS to not be overly held up by an inability to access your server? I don’t know about your situation but I know that a company like Github can probably do better at SRE than I can with a self-hosted server

simlevesque 9y ago
> Github can probably do better at SRE than I can with a self-hosted server GitHub maybe but for BitBucket, I'm not sure at all that it's true.
Siecje 9y ago
You can self host JIRA.
- BoorishBears 9y ago
  I’m aware, and I’d wager most self-hosted Jira instances are behind on updates and stability. One former workplace had to bring in multiple Jira consultants due to poor performance and stability

gtsteve 9y ago

Well if you can't read then you can't see what you're writing to right? But I agree, it's not ideal that Bitbucket has been experiencing issues. I am also looking at alternatives, most likely using AWS CodeCommit and Upsource. It's been at the back of my mind to move away for a while.

udia 9y ago

Relying solely on one platform for your organization's code repository needs is a bad thing in general. I have a workflow where I host my own Gogs instance on a raspberry pi, as well as on Github. Maybe this would be useful to you? I've configured my push such that it will deploy to both Github and on my Gogs meaning that I always have an up to date repository in two places. https://github.com/gogits/gogs https://github.com/gogits/gogs (if you're interested in setting up your own) https://stackoverflow.com/questions/14290113/git-pushing-code-to-two-remotes https://stackoverflow.com/questions/14290113/git-pushing-cod... (push to two remotes) You don't have to use just BitBucket, or rely entirely on your self hosted git service.

simlevesque 9y ago
That's what I wanted to do. Thank you for the guide !
foxylion 9y ago
I think hosting the repository is one thing. But integrating those two repositories with all the automated workflows is quite a headache. Jenkins won't automatically switch over to another git backend if one fails. Other automated tools like code review are mostly relying on a centralized repostiory. I don't see any simple solution to solve these issues. Bitbucket, GitHub, GitLab don't have any easy fallback solutions when you relying core workflows on their services. Self hosting is maybe one solution (we do this with bitbucket), but this requires major administrative effort to keep it running reliably (always available, no data losses on hardware/software failures).
- udia 9y ago
  I agree. Having CI with multiple git repositories is still a painful thing. (I continue to rely primarily on GitHub here. My backup repos don't have CI enabled.) I think if your primary repo that's hooked into CI goes down, you're still SOL. Having your own repo just enables you to continue local development among your team. I don't see any great solution, other than making your app distributed to begin with and doing your build/deploy manually. As a complete aside, I've fantasized about deploying to all cloud vendors (Azure/GCE/AWS/Heroku/DigitalOcean/misc.) each with their own specific build/deploy and having persistent state shared with something like CockroachDB. Having some load balancer managing state between all instances. Taking advantage of the free/basic tiers provided by all of the vendors.