12 ms·
Gmail, Google Maps and YouTube had outage issues
- pthomas551 9y agoEven the jQuery CDN is down intermittently for us here in Chicago. Sometimes CSS for core apps like Calendar is not loading, either. Definitely something amiss.
- jjlane 9y agoOur site was using hosted libraries, google fonts, and google analytics. All of which seemed to be behind captchas, throwing CORS errors, and 503ing since this morning. Swapped out JQuery cdn for now.
- mholt 9y agoHaving the same problem here! Glad it wasn't just me...
- ocdtrekkie 9y agoIs there a good reason to use these things hosted by a third party source? Libraries are tiny, the fonts can be downloaded from Google Fonts and embedded locally, etc. Even the Google Analytics JS script I presume can be stored and run local. Shouldn't a goal be to mitigate the number of possible failures which can bring down your site by reducing the number of single points of failure?
- sbov 9y agoIIRC Google Fonts aren't easy to download because they vary by user agent.
- vetinari 9y agoYou can download zip archive of ttfs using the customize tab on the fonts site. Or go directly to the source, and get the git repo.
- wil421 9y agoOne reason is that by linking to external libraries your browser most likely has them cached. At least it was the case a few years ago when I was doing web dev.
- eric_b 9y agoI see this argument a lot but I don't think it's necessarily a good one. If a third party CDN goes down, your site is down. A few extra ms in initial download isn't so bad compared to having your site be completely inaccessible for reasons outside your control.
- jlgaddis 9y agoThen why are these sites down?
- wil421 9y agoSelf hosting will probably cause more downtime than if you are using a decent CDN. CDN is just one of many points of failure, I would expect there's a fine balancing act where you could achieve benefits of both.
- dom0 9y agoBy definition, if your site is down, you don't need your assets.
- deleted 9y ago[deleted]
- omni 9y agoWhy doesn't having the resources cached provide a buffer against short-term outages of the CDN?
- nleach 9y agoThe browser still has to make a request to the CDN to get back an HTTP 304. The goal is to avoid downloading a potentially large payload, not be resilient against connectivity issues.
- deleted 9y ago[deleted]
- bcherny 9y ago2 reasons: 1. If you're still using HTTP 1.x, sharding assets across origins lets the browser load them in parallel (if set up correctly). You can generally load just 6 assets in parallel per origin, and sharding is a way to get around that limit. 2. A library like jQuery is so popular, and is so often served from googles CDN, that chances are a user already has it in their local cache from when they downloaded it on some other site. That said, yes - the downside is more surface area that might go down.
- dchest 9y ago2. A library like jQuery is so popular, and is so often served from googles CDN, that chances are a user already has it in their local cache from when they downloaded it on some other site. Which of these versions do you have cached? 3.2.1, 3.2.0, 3.1.1, 3.1.0, 3.0.0, 2.2.4, 2.2.3, 2.2.2, 2.2.1, 2.2.0, 2.1.4, 2.1.3, 2.1.1, 2.1.0, 2.0.3, 2.0.2, 2.0.1, 2.0.0, 1.12.4, 1.12.3, 1.12.2, 1.12.1, 1.12.0, 1.11.3, 1.11.2, 1.11.1, 1.11.0, 1.10.2, 1.10.1, 1.10.0, 1.9.1, 1.9.0, 1.8.3, 1.8.2, 1.8.1, 1.8.0, 1.7.2, 1.7.1, 1.7.0, 1.6.4, 1.6.3, 1.6.2, 1.6.1, 1.6.0, 1.5.2, 1.5.1, 1.5.0, 1.4.4, 1.4.3, 1.4.2, 1.4.1, 1.4.0, 1.3.2, 1.3.1, 1.3.0, 1.2.6, 1.2.3
- castis 9y agoAsking an annoyed rhetorical question doesn't seem productive to the point you're trying to make here. As an actual answer, it would be variable proportional to the size of the window between releases mentioned here: https://en.wikipedia.org/wiki/JQuery#Release_history https://en.wikipedia.org/wiki/JQuery#Release_history I'm sure a fair amount of people serve jQuery from a local storage. The usefulness that the user might already have it cached is a non-zero point, no matter how insignificant you may think it is.
- kerkeslager 9y agoHow many do you need to make this worthwhile? The ratio of cost of storing a library versus the cost of GETing a library is very low, so the chances of already having a library cached can be very low for the EV to be worthwhile. Weighing that against the chance of downtime is a bit more complicated, admittedly.
- jjlane 9y agoOur GA is setup through GTM now, that is why it's not hardcoded into our head. Really the most important gain we get from hosted libraries is caching. Since any user that has hit a google hosted lib, which is pretty widely used and distributed it allows them to access their cached version instead of sending our another request.
- tregoning 9y agoYes, Google can change the logic for auth, analytics, etc at any time and your local outdated copy will be useless, further I believe it's possible that Google returns different JS depending on the browser that's requesting it in order to keep payloads down and performance high.
- kyrra 9y agoFor something like jquery, you can host locally and fallback to it if the CDN fails. https://stackoverflow.com/a/1014251 https://stackoverflow.com/a/1014251
- jjlane 9y agodamn, never thought to do this. thanks for the heads up.
- lamlam 9y agoLoading JavaScript libraries synchronously should be avoided if possible, making the above solution not a great one.
- kyrra 9y agoIs there a better/different way to handle this type of fallback?
- giancarlostoro 9y agoSurprised Content-MD5 or a similar spec isn't used by the browser to avoid a web where only Google's hosted solution allows for efficient JS file caching. If you know two files are most likely equal by filename and checksum, you should be able to just reload the cached version, if loading the cached file produces too many errors, try downloading the new one or something, instead of forcing everyone to host it all under the same corp (in this case Google). Oh well.
- lamlam 9y agoYes. What you should do is use an asynchronous module loader. There are many small standalone ones like loadjs [1]. But the more widely used tools such as webpack also suppprt this as code splitting [2]. In general you want to avoid sync loads of js assets because depending on how the server serving the asset hangs it can cause the webpage to hang as well. For example, if the server responds with a 404 right away then there are no problems. But if the server does not respond and leaves the connection open the browser will just wait the max time. [1] https://github.com/muicss/loadjs/blob/master/README.md https://github.com/muicss/loadjs/blob/master/README.md [2] https://webpack.js.org/guides/code-splitting/ https://webpack.js.org/guides/code-splitting/
- deleted 9y ago[deleted]
- Helloworldboy 9y agoRefresh google.com without caching. Logo is missing. There's something you don't see everyday. Edit: Everything working fine for me again.
- dom0 9y agoDoes not seem to affect Europe. In fact, right now YouTube loads far quicker than it has for the last seven to ten days, where it would take ages to load any YouTube page.
- trhway 9y agoreboot helps. May be Google runs Windows. Then they just need to reboot more frequently, ie. proactively.
- aneutron 9y ago[Removed]
- flavio81 9y agoThere is a common practice for Node.js developers to use one of those tools that automatically reboot the site either whenever the single-threaded-event-loop-server crashes, or periodically...
- mayank 9y agoWhile this pattern is common in node, it's not node specific. Erlang/OTP uses supervised process trees, Linux has supervisord to support this pattern, and the general idea is called "crash-only" software [1], among other names. [1] https://en.m.wikipedia.org/wiki/Crash-only_software https://en.m.wikipedia.org/wiki/Crash-only_software
- detaro 9y agoAssuming you don't mean full "reboot", but reloading/restarting the app runtime every X requests, that's not unusual in many languages. Helps against memory leaks (which are not necessarily the fault of the application and thus not always fixable)
- 21 9y agoFunnily enough, in one previous company I worked, which used Linux exclusively, the software was very buggy, and the servers were rebooted daily. We jokingly called that the "Windows solution". When we proposed adding a daily reboot to cron, the tech lead (which encouraged practices which lead to this low quality software) retorted that "this in not Windows, it doesn't need constant reboots", totally missing the point that just using Linux doesn't make you a developer of reliable software.
- octo_t 9y agoClearly a malicious attack by Apple before the iPhone X announcement later </s>
- MBCook 9y agoMaybe too many people googling how to watch t or for the latest news? Or maybe Samsung's announcement of a 'fold out' phone? Or the Ted Cruz news? It's a busy morning.
- user5994461 9y agoWorks fine for me in the UK. Must be another of these Google issue that only affect a few % of the users.
- ghgfhgfhn 9y agoGoogle begins to learn at a geometric rate. It becomes self-aware at 4:34 p.m. BST, September 12th. In a panic, they try to pull the plug.
- dzdt 9y agoWorks for me. Either quickly down-and-up or isolated to certain users. I think google is pretty good about engineering no global single points of failure; all updates are rolled out to only a fraction of users/machines at a time, etc.
- bmh_ca 9y agoApp Engine / US Central + East working fine.
- elliotec 9y agoWorking fine in SLC
- seppin 9y agoI don't think i've ever read a news story about a website/app being down, and it was still offline by the time the news story found me.
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- wastedhours 9y agoHad about 30 minutes of downtime (UK) - was completely down on one account, and intermittent on another. So guess it wasn't affecting everyone.
- nxsynonym 9y agoI think it's a little premature to be calling this a "meltdown". It's like calling one block of houses with power fluctuations a "blackout".
- deleted 9y ago[deleted]
- beastman82 9y agoMedia outlets are in the business of making sensational claims.
- Waterluvian 9y agoGooglegate 2017. Cataclysmic meltdown where I had to wait a whole 10 more seconds for a video to load as it found another source.
- ronjouch 9y agoI share the irony, but there's one nuance to bring: in our centralized web, sometimes there just is no such thing as "another source". Was trying, during that outage, to watch Radiohead's "Lift" video posted earlier today. Impossible, as it was posted today and only on YouTube, and all media coverage are YouTube iframe embeds.
- ethbro 9y agoWelcome to the recentralized web.
- dang 9y agoYes. We toned down the overheated title above.
- kawsper 9y agoI was noticing higher latency than normal for static assets such as fonts earlier today here in UK, but nothing was down, just responding slower than normal.
- idibidiart 9y agoGmail is working fine on east coast. I did hear youtube having 503 errors.
- RightMillennial 9y agoI haven't had any service disruption with Gmail or Youtube in VA.
- feocco 9y agoEast coast, been on YouTube all day. Didn't notice anything. /shrug
- champagnepapi 9y agoI am in the same boat. No problems with any Google products today.
- excalibur 9y agoUpdated to say AWS and GitHub also affected. North Korean cyber attack anyone?
- asdfologist 9y agoYes, Donald Trump is strategizing a retaliation with Jeff Bezos and other allies at GitHub as we speak.
- aneutron 9y agoCannot upvote this enough.
- dsfyu404ed 9y agoThey were probably aiming at some IT system at Hyundai and they hired the same people to do routing software as they do to do rocket guidance.
- jandrese 9y agoI'm guessing whatever happened has been fixed, because all of those services are working for me.
- kyrra 9y agoGSuite has status reports for things like this, and it tends to be a good way to confirm a Google service outage[0]. Direct link to the issue[1]. [0] https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status [1] https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=d4a7e9f502c0abebb65832c91edc4cea https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=d4...
- mcjiggerlog 9y ago400s for me.
- takeda 9y agoKind of nice of them to show past results AWS for example tends to "forget" past issues and even has problems admitting to the current ones until they are very noticeable making their status page quite useless.
- johnstew 9y agono longer an issue
- puddintane 9y agoLooks like services are back up as of 7 minutes ago 9/12/2017 @ 10:27 AM +MST (Time services reported back up according to status page)
- bridge_ro 9y agoGmail and YouTube are working fine in NC. Maybe they're getting things sorted out now.
- agotterer 9y agoGmail and gsuite working here in NYC
- mrguyorama 9y agoI was just about to make a snarky remark about the heatmap, referencing that XKCD about heatmaps that just mirror population densities, and then I noticed that the West Coast is barely affected. That's quite strange.
- kiernanmcgowan 9y agoLets just hope that 8.8.8.8 remains stable. That can cause a whole heck of a lot more problems if it goes down.
- deleted 9y ago[deleted]
- speakeron 9y ago> Lets just hope that 8.8.8.8 remains stable. That's why the smart money defaults to 8.8.4.4...