9 ms·
We are investigating reports of degraded performance
- MuffinFlavored 3y agoWhy can't they just roll it back /s Who decided to deploy something risky on a Friday /s Why don't they have lower environments like QA to test it in before going to production /s
- __turbobrew__ 3y ago404 on all PRs -> degraded My definition of degradation is different.
- edgyquant 3y agoI would say it’s degraded. I still see I’m logged in, can still see the org screen and even the repo (on and off.). Don’t think degraded implies anything good. Just that something’s are working others aren’t
- SoftTalker 3y agoI like how "Every request returns a 404" == "degraded performance"
- CodeCompost 3y agoDegraded in relation to 99.90÷ spread across the year.
- akdor1154 3y agoHow on earth did you make that typo? I'm honestly intrigued, do you use an ime?
- CodeCompost 3y ago+×÷=/_<>[] ... Samsung Keyboard on my phone.
- CodeCompost 3y agoI'm a moron. I typed in the divide symbol. What an idiot I am. Obviously I should have typed %
- akdor1154 3y agoThey do kinda look the same if you tilt your head i guess :)
- agilob 3y agoIt should load faster if there's nothing to load
- methodical 3y agoThey should have called it upgraded performance. The 404 page loads blisteringly quick!
- flukus 3y agoIf nothing loads nothing gets cached and you can get in a cycle of very slow 404s. The current abomination I'm working on avoids this by caching the errors and serving them for several hours...
- agilob 3y agoOne of the worst outages I witnessed was due to negative DNS caching on the most outer router that took the company a few (3 or 5) working days to fix *after* issue was identified.
- lcnPylGDnU4H9OF 3y agoIs this a designed self-cache poisoning? Abomination indeed.
- layer8 3y agoAt least it did return a response.
- Animats 3y agoGithub is becoming too important. A routine backup system is needed, something like archive.org, that backs up important Github projects.
- shepherdjerred 3y agoI assume that there are some archives of public GitHub repositories somewhere.
- rizky05 3y ago[dead]
- hipadev23 3y agoThat exists on every dev’s machine.
- Animats 3y agoIn theory, the code is there, but putting a project back together in a hurry after trouble at Github is non-trivial. Especially if the build process uses proprietary stuff such as "GitHub actions". The issues and discussions are all Github-only, too.
- jrockway 3y agoThe source code of the actions is in your repo. This is more of the problem of relying on proprietary software; if the company that makes it dies, you die too!
- misnome 3y agoIt’s crazy that there is no common CI spec. Every single platform is completely different. Last time I tried Act it had some pretty severe limitations. Perhaps I should take it for another spin.
- 3y ago
- andrelaszlo 3y ago⮕ git push ERROR: Repository not found. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. That scared me for a minute. ':D
- misnome 3y agoYeah, retriggered our deploy pipeline a few times, checked downdetector and no reports, logged in to machine and verified external connection. Sighed, assumed I was on the first wave of an incident on top of their current slack incident, then logged off for the day.
- droptablemain 3y agoMy repos were coming up as 404s. I was wondering if I had been canned...
- deleted 3y ago[deleted]
- shepherdjerred 3y agoHah I had the same thought
- strongly-typed 3y ago[flagged]
- 0xDECEA5ED 3y ago[flagged]
- __MatrixMan__ 3y agoThe "nothing to see here" approach to access control has a lot of weird culture-consequences. I wish software would just address me like the peasant that I am, rather than trying to gaslight me into believing that my artificially limited world is the whole one.
- openthc 3y agoIf there's nothing wrong with me, maybe there's something wrong with the universe...The universe is a spheroid region, 705 meters in diameter.
- dilap 3y agoI think you don't want to reveal the presence of projects by name or not -- just the name of repo could be sensitive info, potentially.
- 3y ago
- Eji1700 3y agoOnce again proving that Monday is "whenever you push to production"
- methodical 3y agoPR really spun gold when they decided to label everything from every single one of their databases getting deleted and backups nuked to intermittent connectivity issues as "degraded". Who exactly are they making feel better by not calling a spade a spade?
- skeaker 3y agoShareholders, generally.
- brookst 3y agoPeople will take issue with any wording. Why try to find the exact right words each time when people will have exactly the same complaints?
- methodical 3y agoI take issue with clear PR-speak trying to make the issue lesser than it actually is. When you're having an outage- call it an outage. Having a feature completely unusable and labeling it as "degraded performance" is clearly twisting your words to lessen the outwardly perception of the scale of the problem.
- brookst 3y agoBecause they want to use the same language every time, and say something quickly, and litigating whether something is an outage or degradation or latency or intermittent or whatever is a distraction. People are going to jump on them no matter what. Having a engineers talking to PR wouldn’t help anyone.
- WolfeReader 3y agoI cannot wait for Gitea, Forgejo, and GitLab to start federating with each other via ActivityPub. Then we can all take one more step away from a corporate-controlled internet.
- rglullis 3y agoThat would surely be nice and helpful, but why do we need to wait for it? Even my open source projects in github are just mirrors from the "real place" of work: gitlab or my own gitea instance. If github is down, it is a minor inconvenience but I can still work.
- dgivney 3y agoI'm a little confused, Gitlab is a publically traded corporation? https://www.google.com/finance/quote/GTLB:NASDAQ https://www.google.com/finance/quote/GTLB:NASDAQ
- ironmagma 3y agoDoes that matter if it federates openly? Anyone can.
- WolfeReader 3y agoGitLab, Gitea, and Forgejo are applications that can be easily self-hosted. One of those also has a corporation associated with it, but that has minimal effect in this case.
- perihelions 3y ago- 'one more step away from a corporate-controlled internet" Downvote me to grey-world if you like, but I think everyone's crazy to put all their code infrastructure in the hands of fucking Microsoft. Especially literal free open-source software. Who do you think Microsoft is? What do you know of Microsoft's history and their core values (they're "embrace, extinguish & exsanguinate"). It's like giving fucking Sauron safekeeping of your power-rings in Mordor, oh we have great infrastructure for safe ring storage here, very secure, the orcs are really expert guards.
- rglullis 3y agoI need a macro template for those memes with Bart Simpson on the blackboard and make it say "I will always have a backup plan for third-party services". Seriously people: gitea exists. Gitlab self-hosted exists. Drone/Woodpecker CI exists. It's not that difficult to set up a project that does not depend on Github. I spent less time setting these up than the amount of down time that Github has had this year.
- waihtis 3y agoProblem is, this is one of 50 things that can go wrong for which you should have a robust backup plan for
- asdff 3y agoIt's amazing how many of these issues can be obviated by taking a step back at a given SAAS, asking can I self host this, then if you can and need redundancy to just buy two desktops and stick one in your place and another in your friends apartment in another town. With a lightweight static site, a modern desktop is probably more than powerful enough to deal with most any load you might realistically see for your given project. You are also extremely unlikely to have both of these desktops go down at once if they are on different local power grids and internet service providers, short of invasion of the continental U.S. perhaps.
- pixl97 3y agoSo how do you keep data synced between them in a manner that ensures the data is safe? Now you're playing security sysadmin, and you're playing backup administrator, and your playing hardware admin (did you use 2 SSDs bought at the same time?). You don't ever escape the payment cost of the issues of keeping these systems running. You're paying staff to do it, you're paying a cloud provider to do this, or you're paying out of your own time.
- asdff 3y agoIf these things were so insurmountable, the early internet would not have existed at all. Yet it did and still does in many corners, which stands to reason that you can do these things too.
- deleted 3y ago[deleted]
- yieldcrv 3y agowell this was a wakeup call, we don't use github but apparently a bunch of bash scripts for package managers do we couldn't onboard a new hire because they couldn't even run a curl command for some basic tooling I thought it was a url, but the url actually loads a bash script that tries to pull from their github repo
- kklisura 3y agoRegardless of this: does anyone experience general slowness of Github? I view a file (on web) and it takes time for page to be fully interactive - no buttons work, rest of the file cannot be viewed - just the top part is shown (above the fold, maybe). Honestly, it's so nerve wracking.
- rvba 3y agoQuestion. Recently github is pushing very agressively for two factor authenticartion. So I installed the authenticator app. But the authenticator does not work when the clock on my phone is not perfectly synchronized. But my phone's clock is intentionally sped up by +15 minutes? What to do?
- dingdong33 3y agoSynchronize your clock.
- rvba 3y agoI want my clock to be sped up by 15 minutes.
- 4hg4ufxhy 3y agoYou could fork some open source TOTP app and change the time generation. Or you can save your code and wait 15 minutes to login.
- Arbortheus 3y agoWhy?
- rvba 3y agoI watched a TV show about the Queen of England and I use her trick to never be late.
- tanseydavid 3y agoWrite an app to do the +15 for you. There must be at least 7-8 other people in the world would pay for this.
- louis-lau 3y agoTOTP usually only allows for about 30 seconds of time drift between the device and the server. If you really must set your phone to the wrong time you can use a cheap second device, or a hardware token. Technically if an authenticator app has an option for a time offset it should work, but I've never encountered one.