9 ms·
Incident with GitHub Actions and Codespaces
- dgb23 5y agoIs it just me or are the incidents, outtages, security breaches just raining down on major services since a couple years?
- jacquesm 5y agoNo, it's just you. This has been happening since the cloud became a thing. What is changing is that the severity is going up because more and more parties are now dependent on these cloud things. Wait until we have self driving cars that are somehow cloud dependent and in the middle of rush hour AWS or GCP decides to burp.
- yed 5y agoTo put these billing errors in context, our account suddenly showed ~$12,000,000 in usage fees. I saw some others posting bills high as 100s of millions. Pretty heart attack inducing.
- karussell 5y agoyeah, better than coffee ;) ... we had $78,610,108
- krab 5y agoThe amount was so ridiculous that it didn't replace coffee for me. :-) Anyway, it's already fixed for our account.
- fishnchips 5y agoOur account is still showing $188,536,934.07 overage.
- xrd 5y agoYou didn't read the blog post from GitHub saying they were switching to Zimbabwean dollars because the dollar is falling out of favor as the reserve currency?
- dx034 5y agoThat these were so scary shows the downside of usage-based fees. If you run on dedicated hardware, you always know what you'll have to pay, no matter how you utilize the system. The risk of surprise bills, even with the advantage of higher scalability, is one of the major factors to put me off many cloud services. I wouldn't want to run a business where you always have the risk of some bug or unforeseen behavior destroying your budget.
- bencollier49 5y agoI've said this over and over again. There should be laws in place enabling a person or company to set a hard limit on spend. It would be a pro-innovation law because people would then have the confidence to innovate.
- Dave3of5 5y agoI agree cloud providers shouldn't be able to bankrupt their users due to high usage. IMO it should be mandatory to set a limit it makes sense from a finance side. In my case my cards don't have $12 million of a limit so they could not actually charge me that amount of money. It makes sense if you use invoice type billing i.e. like a company but as an individual these insane bills can't actually run.
- blibble 5y agothey wouldn't be able to charge your CC but this isn't really relevant to what you owe them they can still attempt to come after you for that $12 million
- Dave3of5 5y agoOf course but I cannot ever pay that amount so what would be the point from their side. It would be more sensible to stop after they run out of money on my CC.
- OJFord 5y agoI also think it'd be good for companies. If you usually pay (say, numbers don't matter, except that AWS or whoever doesn't care until they get bigger I suppose) 1k-3k and set a 'don't surprise me too much' limit of 5k, then when something unexpected happens, instead of having a bad time pleading with them to let you off 12k or whatever and feeling cautious about using it again perhaps with some sour taste even if they do let you off, you think 'woah, glad I have the limit', and pretty much get on with your day; AWS or whoever gets an unexpected 2-4k instead of writing off 9-11k, and keeps the happy customer (at the customer's expense instead of its own).
- jotm 5y agoDamn. Well, when you owe $10,000 it's your problem, when you owe $10,000,000 it's their problem :D
- prepend 5y agoI think this code wasn’t supposed to go into effect until Microsoft moves beyond their friendly stage and moves into maximum revenue extract stage.
- rvz 5y agoLess than a month with another incident with GitHub Actions. Last time that happened was two weeks ago. [0] Not looking good for going 'all in' on GitHub products or 'centralizing everything' to GitHub as I predicted. [1] I expect them to go down at least once every month guaranteed. Not 5 times in two weeks. I don't think they can handle a month without a serious incident at all. [0] https://news.ycombinator.com/item?id=30841387 https://news.ycombinator.com/item?id=30841387 [1] https://news.ycombinator.com/item?id=22867803 https://news.ycombinator.com/item?id=22867803
- extheat 5y agoThis "incident" has no material effect beyond the number on the billing dashboard.
- noir_lord 5y agoActions fell over yesterday afternoon. In our case first sign was Jenkins not kicking off builds on push.
- atomicfiredoll 5y agoActions were unavailable yesterday for folks, as the GitHub incorrectly thought that people were out of credits.
- deleted 5y ago[deleted]
- flower-giraffe 5y agoI find it concerning that the incident is marked as resolved and the incident history suggests that have corrected the billing. We still have a $7m dollar balance on our account, so the issue clearly is not resolved. Providing incorrect information on status pages makes it hard to trust in the future.
- enw 5y agoWhat is the typical relative balance on your account? Does $7m mean 7 million USD? Over what period of time?
- a1371 5y agoSeven... Million? Though it says the incorrect billing is still not fixed there.
- dx034 5y agoI understand it differently. They said 7 hours ago it hadn't been fixed but afterwards they said the issue is resolved. I would understand that this means all issues have been resolved. Otherwise I don't get how they can close the incident.
- flower-giraffe 5y ago9 hours ago they posted "We have started a cleanup of incorrect billing information and expect that to be completed in a few hours." 2 hours later they marked the incident as resolved, but they clearly have not completed the "cleanup" yet. status page : Resolved This incident has been resolved. Posted 7 hours ago. Apr 14, 2022 - 01:28 UTC Update Actions and Codespaces are operating normally except for incorrect billing reports. We have started a cleanup of incorrect billing information and expect that to be completed in a few hours. Posted 9 hours ago. Apr 13, 2022 - 23:36 UTC Update We've applied a mitigation to unblock running Actions jobs and using Codespaces. We are currently working on addressing the inconsistencies in billing information. Posted 11 hours ago. Apr 13, 2022 - 21:07 UTC
- dx034 5y agoAnd here we are with our on prem Gitlab with CI/CD that hasn't been down for 2+ years (except for very short, planned maintenance) and has 100% predictable cost at much lower rates than cloud services. We can still scale within minutes if needed (thanks to Terraform). Dedicated servers are just so cheap, esp with sth like CI/CD that's perfect to host at cheaper providers. Seeing constant issues with Gitlab and Github products just reinforces that decision.
- qeternity 5y agoWe switched from self-hosted Gitlab to Gitlab.com in the name of reducing infra burden...my oh my do I regret that choice. Same experience to you - no downtime that wasn't planned when we self-hosted. Load was very predictable, and ran on k8s anyway. Much, much faster than the hosted version.
- dx034 5y agoAnd I don't get what infra burden there is. We run Gitlab, Mattermost, Databases etc and running/maintaining those is still just a very minor part in the work of some developers (who like and are good at infra). Put everything behind Tailscale/Cloudflare Tunnel and your attack surface is very low to start with. Most deployment and patching can be automated anyway, so there's not much left. In the end we save a lot of cost, avoid lock-in and keep much more knowledge in-house.
- qeternity 5y agoIt’s just one more thing to backup, test, monitor, etc. Upgrade migrations didn’t always succeed so then you need to go back and remember what you did last time to fix it. It’s a cognitive burden and distraction. In theory it’s cheaper to pay Gitlab to handle for us. Except that now we lose more time due to Gitlab.com outages than we spent maintaining on prem.
- Cthulhu_ 5y agoThat's exactly it, it adds a list of nodes to the "required knowledge" graph of your company, adding job positions, expenses, people, and (operational?) risk to your company that I'm sure already has its own list of things to worry about. There's so many companies out there whose infrastructure is run by this One Guy. Any company needs to assess whether they can risk losing their One Guy (see also: Bus Factor), or if they'll just have to swallow the cost and (some) risk of (some) downtime by using a service provider.
- donavanm 5y agoOh billing :( My own disappointing incident was metering customer usage in Bytes when billing was setup for Gigabytes. A couple of “concerned” customers noticed that super quick when the new functionality was turned on. Having to sort that out at 3am highlighted how many “business process” dependencies dont have the same level of operational response as our systems. On the plus side it highlighted some areas to improve continuous monitoring (and alerting) of what customers are metered and billed.
- Dave3of5 5y agoThis type of thing scares the shit out of me everytime using cloud. The possibility of something idiotic happening and me losing my life savings is just too high. That's why I use a cloud provider that doesn't charge for bandwidth usage and I use a fixed set of "stuff" such that I always know what my fee's should be. I understand for big companies with millions of users they can't operate in that way but for me risk vs reward is too high.
- Nextgrid 5y ago> I understand for big companies with millions of users they can't operate in that way but for me risk vs reward is too high. They absolutely can. If anything, big companies have more resources to build their own cloud based on on-prem or rented bare-metal than smaller companies. It's just all cargo-culting. The "cloud" is not some kind of unobtanium hardware you absolutely can't get anywhere. With rare exceptions, it's standard hardware you can buy directly from any server & network equipment manufacturer.
- Dave3of5 5y agoThat's an error on my part sorry I meant can vs can't.
- ricardobayes 5y agoMorning coffee, check Daily github incident, check
- deleted 5y ago[deleted]
- andrew_ 5y agoWe spent three hours trying to track down what happened, thinking that an account had been compromised, before GitHub acknowledged this. Our usage report said that the egregious minutes were run on deleted repositories so we were very concerned. These outages and incidents are getting very tiring, and costly.
- karimbellazrak 5y agoSuper cool !
- godzillafarts 5y agoAny idea if this tangentially impacted LFS bandwidth somehow? We're seeing a huge spike in LFS transfer starting yesterday afternoon. The spike doesn't seem to correspond to CI builds cloning the repo (because it was sustained over night after our devs had signed off). No notable changes to the repo, automated processes, etc. We've had to purchase additional data packs to unblock CI builds and folks pulling from the remote and assume we will need to purchase more. 11 days left in our billing cycle...
- deleted 5y ago[deleted]