23 ms·
This seems to be impacting a number of huge sites, including the UK government website[0]. [0] https://www.gov.uk/ https://www.gov.uk/ https://m.media-amazon.
by lpmitchell 5y ago
This seems to be impacting a number of huge sites, including the UK government website[0].
[0] https://www.gov.uk/ https://www.gov.uk/
https://m.media-amazon.com/ https://m.media-amazon.com/
https://pages.github.com/ https://pages.github.com/
https://www.paypal.com/ https://www.paypal.com/
https://stackoverflow.com/ https://stackoverflow.com/
https://nytimes.com/ https://nytimes.com/
Edit:
Fastly's incident report status page: https://status.fastly.com/incidents/vpk0ssybt3bj https://status.fastly.com/incidents/vpk0ssybt3bj
- c-fe 5y agoalso https://www.reddit.com https://www.reddit.com (at least in Netherlands) edit: 12:05 up again for me, no images or custom fonts loading though ... and down again 1 minute later edit: 13:01 reliably up again for me
- deleted 5y ago[deleted]
- secondcoming 5y agoDown in UK
- tacticalmook 5y agoDown in US. Also Imgur, which is closely related
- floating_panda 5y agoDown in india
- silviot 5y agoSame here in Germany: imgur and reddit are down, plus a bunch of other sites.
- tarsinge 5y agoSame in France
- daniaal 5y agoUp for me but no images mate.
- freebuju 5y agoDown in SA
- mondaygreens 5y agoQuora and reddit too
- amiga-workbench 5y agohttps://www.theverge.com/ https://www.theverge.com/ seems to be down too
- weird-eye-issue 5y agoYikes seeing just a "connection failure" on Paypal is something else. edit: PayPal looks be back up at least in US East but when I turn off my VPN and access from Asia I get "Fastly error: unknown domain: www.paypal.com." Now I'm seeing a 503
- thrdbndndn 5y agoInterestingly, Twitter only has its emoji SVGs down.
- oneeyedpigeon 5y agoAnd this is (one reason) why using images instead of actual emojis is such a stupid idea. Why, Twitter, WHY?
- bb010g 5y agoBranding! (Fun fact: Hacker News strips emoji.)
- busymom0 5y agoHow does it strip it? Test: Edit: You are right. It got rid of the emoji after Test.
- numpad0 5y agoerr, to make representations platform independent?
- oneeyedpigeon 5y agoThat sounds antithetical to the purpose of emojis.
- mbreese 5y agoSame reason why Gmail uses their own emojis rather than the system ones — (as said above) branding. When you send a tweet, Twitter wants it to look identical across all devices. The classic native UI vs cross-platform UI debate in a nutshell.
- gspr 5y agoCool, so instead of actually serving text, they could also just serve up little SVGs for each letter. Because god forbid the recipient chooses a different font than Gmail!
- black_puppydog 5y agoAll of these work from here in Grenoble, France...
- clydethefrog 5y agoInteresting. Here in the Netherlands they don't.
- agumonkey 5y agoNot for me
- bsaul 5y agodepends where in France, most people i know here are affected as well
- black_puppydog 5y agoyeah grenoble, updated.
- terramex 5y agoSame here in central Poland (Łódź area), no problem with any of linked websites. edit: My whole Twitter timeline is full of posts saying "Twitter outage? what outage?". Same on Reddit and Twitch chat, feels like for a short time I was invited into some exclusive circle lmao. StackOverflow and other StackExchange sites also work so I can look stuff up for you.
- ju-st 5y agoGermany here (n=1), everything works except reddit and ft.com
- magicturtle 5y agoreddit down aswell
- bartread 5y agoIt's OK though, because large swathes of this discussion seem to have turned HN into reddit, at least temporarily. Normal service will no doubt resume in due course.
- zhan_eg 5y agoBitwarden is also down (the Web Vault, not the website).
- brador 5y agoSearchable offline backup of stack anyone?
- sammygreen 5y agoSeems to be every site that runs varnish...
- meowface 5y agoFastly largely runs on Varnish, it seems: https://www.fastly.com/blog/benefits-using-varnish https://www.fastly.com/blog/benefits-using-varnish >At the core of Fastly is Varnish, an open source web accelerator that’s designed for high-performance content delivery. Varnish is the key to being able to accelerate dynamic content, APIs, and logic at the edge.
- busymom0 5y agoI think Fastly is the one having problems (they happen to use varnish but I haven't seen anything which says varnish is the root cause) - so all sites using it are down.
- abhiminator 5y agoPayPal seems to be working for me at the moment. Rest of the sites are 503s.
- PaywallBuster 5y agoand Imgix
- playpause 5y agoalso https://www.ft.com/ https://www.ft.com/
- threeseed 5y agoI wonder why Amazon is not using Cloudfront for their own website.
- schappim 5y agoYeah, this is what makes me feel this is more an AWS thing
- apexalpha 5y agoIt sure looks like a AWS error, even Amazon.com is mostly down.
- deleted 5y ago[deleted]
- lpmitchell 5y agoThe m.media-amazon.com domain (and a few other CDN'd domains that they use) are running through Fastly: nslookup m.media-amazon.com Name: media.amazon.map.fastly.net It is very interesting that they are not using CloudFront!
- tester34 5y agoWhy?
- jamespo 5y agoGood to have 3rd party redundancy, time to fail over to something else now I'd think though.
- longwave 5y agoThey already have: $ host m.media-amazon.com m.media-amazon.com is an alias for c.media-amazon.com. c.media-amazon.com has address 99.86.119.84 (which is a Cloudfront IP)
- xmdx 5y agoTerraform having issues and rubygems down too
- deleted 5y ago[deleted]
- meepmorp 5y agoThat explains the spotty container build failures over the last half hour. Good thing I decided to procrastinate instead of debugging the issue!
- max23_ 5y agoalso https://www.speedtest.net/ https://www.speedtest.net/
- toaway 5y agoalso https://www.theguardian.com/ https://www.theguardian.com/
- Hani1337 5y agoetsy.com too
- doublerabbit 5y agohttps://deb.debian.org https://deb.debian.org is down too which borked my installation.
- benrbray 5y agohttps://hackage.haskell.org/ https://hackage.haskell.org/ down as well
- fgaz 5y agoThe mirrors still work though, and cabal will just fall back to those
- rvz 5y agoCentralising everything™ and the whole internet goes down because of that.
- aembleton 5y agoand yet you're able to leave this comment.
- rvz 5y agoBecause HN and those who use less or those who use backup services are smart and those who are caught now have to panic and wait. Probably going to short the hell out of $FSLY.
- morelikeborelax 5y agoOver one issue that highlights they have an abundance of top level customers? Interesting strategy when it's already at a low.
- rvz 5y agoOne issue that should have been mitigated at least by Fastly; worse if the client has to do it. They proudly stated this from their own website to their customers: > "Fastly’s network has built-in redundancies and automatic failover routing to ensure optimal performance and uptime." If that isn't one huge lie, I don't know what is.
- aliasEli 5y agoPlease don't call it a lie. It means that they knowingly presented something they knew to be false as the truth. So far I have seen no evidence to support that.
- southerntofu 5y agoIt is definitely a lie, but it's the same lie sold by all cloud offerings. Can you name a single cloud/CDN operator without downtimes? It's normal to have downtimes but they are usually scheduled and quick (think <10 minutes per month for rebooting and/or hardware parts replacement). I'm pretty sure most non-profit hosts like disroot.org or globenet.org have similar or better 9's than all these fancy cloud services.
- deleted 5y ago[deleted]
- maest 5y agoAmusingly, the Stackoverflow 503 page has a typo: Error 503 Service Unavailable Service Unavailable Guru *Mediation*: Details: cache-lon4236-LON 1623146049 854282175 Varnish cache server
- Cyberspy 5y agoWe use Fastly (and our site is down too) but I asked them about this a couple of years ago. It is deliberate. They said it was so they can tell if it is their Varnish service or the customer's Varnish service that is down
- dormando 5y agoThis comment is correct. I made that change ages ago. Amused it's still there.
- lwhi 5y agoLooks like an error (typo) with Varnish .. it's the same on multiple sites. Maybe a good way to work out which versions are being used.
- nomilk 5y agoSame 503 error message on the actual http://fastly.com/ http://fastly.com/ website
- saagarjha 5y agoIt's interesting because Varnish gets it write in their docs: https://varnish-cache.org/docs/trunk/users-guide/troubleshooting.html#varnish-gives-me-guru-meditation https://varnish-cache.org/docs/trunk/users-guide/troubleshoo...
- qeternity 5y ago> It's interesting because Varnish gets it write in their docs Sorry I couldn't help myself...it's too funny to misspell "right" in a thread about spelling mistakes
- tus89 5y agoPaypal back, off fastly
- jslakro 5y agoWhy no other sites bypass the CDN and go directly?
- meibo 5y agoIf you aren't prepared to do CDN changes on a whim when something like this happens, it's often better to wait for the problem to be resolved instead of making things worse for yourself due to misconfigurations, revealing your origin IPs, etc. Can always improve the process for the next outage.
- tus89 5y agoYou need big infra and Crack teams of ops people, which paypal can't afford not to have.
- lucasverra 5y agoBut https://news.ycombinator.com/ https://news.ycombinator.com/ is UP! :) Prepare those HN servers for massive influx in 3...2..1..
- atymic 5y agoNew error now, hopefully fix in progress. Fastly error: unknown domain: www.fastly.com. Details: cache-syd10161-SYD
- deleted 5y ago[deleted]
- VMtest 5y agoI am already here
- unityByFreedom 5y agoWhile we're here.. I am a bit surprised to see how many sites use Fastly. As a dev I've always been happy with Cloudflare.
- zthxxx 5y agohttps://docs.npmjs.com https://docs.npmjs.com down as well
- huijzer 5y agoCSS/Javascript at https://github.com/ https://github.com/ appears to be down as well making GitHub quite unusable.
- oneeyedpigeon 5y agoGitHub Pages appears to be down too, taking an awful lot of sites offline
- busymom0 5y agoGithub is working fine for me in Canada but others aren't. Tried without browser cache too and it works okay. EDIT: Most sites seem fixed now here in Canada. Tested stackoverflow, reddit, GitHub, PayPal, gov.UK and all worked fine.
- oneeyedpigeon 5y agoYeah, things are mostly back now. Including my personal GitHub Pages site :)
- alvis 5y agohttps://www.reddit.com https://www.reddit.com as well
- tandav 5y agohttps://docs.python.org/ https://docs.python.org/ https://www.last.fm/ https://www.last.fm/
- treeshateorcs 5y agodeveloper.spotify.com
- dpacmittal 5y agoelastic.co down as well
- stevenwliao 5y agoSeems to affect Target ( https://www.target.com/ https://www.target.com/ ) and Reddit ( https://www.reddit.com/ https://www.reddit.com/ ) as well.
- maelito 5y agonature.com
- EmptyStatement 5y agohttps://repo.maven.apache.org/ https://repo.maven.apache.org/ is down too
- keithnz 5y agotwitch also, lots of other minor ish websites
- deleted 5y ago[deleted]
- zarker 5y agoSpotify is behaving strangely as well https://www.spotify.com/ https://www.spotify.com/
- conradfr 5y agohttps://getbootstrap.com/ https://getbootstrap.com/ as well
- antihero 5y agoI wonder why amazon.co.uk uses Fastly and not CloudFront?
- jonplackett 5y agoMust be more than fastly. Heroku is also down.
- abunuwas 5y agoThat's also because of fastly, I've got this response from the Heroku dashboard: Fastly error: unknown domain: dashboard.heroku.com.
- abunuwas 5y agoThat doesn't appear to be the case does it? Amazon sites are all working fine, at least for me
- danso 5y agoTheir CSS and JS were down for a few minutes. I was able to login to Amazon but the entire site was in Times New Roman, but was fixed a few minutes later
- georgekeesjones 5y agoI imagine they use a few different CDNs for things like this.
- toaway 5y agowww.gov.uk & bbc are back
- ramshanker 5y agotwitch.tv Too.
- madeofpalk 5y ago> [0] https://www.gov.uk/ https://www.gov.uk/ Just checked, thank god the NHS vaccine site is still available - vaccines just got rolled out for under 30s today.
- patentatt 5y agoEdit: I didn’t mean anything negative here! Just slightly shocked that as the UK is opening up under 30 vaccinations, the US is struggling to find any more willing takers. It’s really probably a sign that there’s fewer anti-vaxxers in the UK more than anything. And that universal healthcare is more efficient at distribution than an inherently for profit system. I don’t know, but I just didn’t realize it was so different in the UK
- bdsa 5y agoI think this may be because we've had much higher uptake as far as I know, so getting down the age ranges has been slower (by which I mean, yes, maybe the US has made it available to all adults, but how many (as a proportion) have taken it up)
- patentatt 5y agoThis is awesome. And also exposes how broken the US is with its anti-vaccination trend
- bdsa 5y agoI have seen the argument made that one of the reasons for high vaccine confidence in the UK is as a result of Andrew Wakefield's MMR fraud, which was perhaps debunked more effectively in the UK than the US. https://www.youtube.com/watch?v=8BIcAZxFfrc https://www.youtube.com/watch?v=8BIcAZxFfrc
- spzb 5y agoUK is ahead of the US https://ourworldindata.org/covid-vaccinations https://ourworldindata.org/covid-vaccinations
- jakub_g 5y agohttps://www.bbc.com/news/technology-57399628 https://www.bbc.com/news/technology-57399628 "A number of leading media websites are currently not working, including the Guardian, Financial Times, Independent and the New York Times."
- kevincox 5y ago> potential impact to performance So it is a "performance" issue when all pages give a 503.
- coolspot 5y agoDoes the 503 page load fast(ly) or slowly?
- jb1991 5y agoFor what its worth, I'm having these problems also with cnn.com, reddit and many others, however when I switch away from WiFi to use my cell provider network, they work fine.
- deleted 5y ago[deleted]
- cellover 5y agoIs the fact of looking at those links is like looking at a road accident with insistence instead of just passing by?
- collyw 5y agoClick the new tab. Lots of posts about sites being down. All flagged.
- deleted 5y ago[deleted]
- milliams 5y agoYes, because they're all just repeats of this one.
- deleted 5y ago[deleted]
- caseymarquis 5y agoFastly Engineer 1: Seems like a common error message. Can you check stackoverflow to see if there's an easy fix? Fastly Engineer 2: I have some very bad news...
- divs1210 5y agoHaha! That sounds highly plausible!
- amirhirsch 5y agoStack Overflow is down, can someone tell me how to declare a static multidimensional array in C++?
- vlovich123 5y agoGoogle and DDG surface SO results cached within their own page. Here’s the copied answer: int main() { int arr[100][200][100]; // allocate on the stack return 0; }
- runawaybottle 5y agoOh man, how do we keep a pocket copy of SO? All of our jobs depend on it.
- detaro 5y agono they don't, but if yours does you can download a complete datadump of SO from them.
- nephanth 5y agoYou can use kiwix (https://www.kiwix.org/en/ https://www.kiwix.org/en/). It is an open-source software that allows you to keep and read offline static versions of websites in a specialized archive format (zim-files) It was originally designed to allow you to read wikipedia offline, but there are also dumps of stackoverflow available on the relevant page : https://wiki.kiwix.org/wiki/Content_in_all_languages https://wiki.kiwix.org/wiki/Content_in_all_languages
- 1ncorrect 5y ago> Monitoring The issue has been identified and a fix has been applied. Customers may experience increased origin load as global services return. Posted 4 minutes ago. Jun 08, 2021 - 10:57 UTC Looks to be working again my end.
- rottc0dd 5y agoSSO and github are back online now
- Simran-B 5y agoPantheon was also affected: https://status.pantheon.io/incidents/n9sngt0q0mct https://status.pantheon.io/incidents/n9sngt0q0mct
- aliasEli 5y agoWhat's far worse than half of the internet being down was that Hacker News also had problems. If I waited long enough on a comments page I got an error message. I don't quite understand what happened there. The communication between my system and HN must have been working otherwise I would never have gotten an error message, so it must have been some internal HN problem. But since HN should only need its own internal "database" to generate comment pages, I don't understand why it should be impacted by the Fastly problems.
- dreamer7 5y agoFirebase hosting has been affected as well
- dan-robertson 5y agoIs their anything these big sites could do in this situation, or must they choose between running and maintaining all of their own infra or relying on a single CDN?
- darkr 5y agoIf you have absolutely vanilla CDN requirements, you can run multiple CDNs and fail-over or load balance between them using DNS. Quite a few Fastly customers have more than vanilla requirements though, and may have a lot of business logic performed within the CDN itself. That Fastly is "just Varnish" and you can perform powerful traffic manipulation is one of it's main selling points.
- dan-robertson 5y agoI suppose it’s still a bad experience for the user if some % of attempts to connect fail or if some % of scripts/styles/images fail to load. So I think that means dns information about failures needs to somehow be propagated quickly. Not sure how well that works in practice.
- fmajid 5y agoUse two CDNs and DNS providers for redundancy. Gets expensive, but at scale, probably doesn't make a huge difference. More complexity for the site operators to manage, however.
- jzer0cool 5y agoI could not tell from the fastly status page. What caused the fault? Could anyone point to any past stories which may be of similar nature other than DDos?
- bartimus 5y agoStill a bit vague: https://twitter.com/fastly/status/1402221348659814411 https://twitter.com/fastly/status/1402221348659814411
- linuxfan2021 5y agoYou would think that the UK GOVERNMENT would have their own private CDN or something...
- iso1210 5y agoWhy?
- deleted 5y ago[deleted]
- lkbm 5y agoGood thing we use Cloudfront and Cloudflare where I work. > Statuspage Automation updated third-party component Spreedly Core from Operational to Major Outage. > Statuspage Automation updated third-party component Filestack API from Operational to Degraded Performance. Oh, right. :-D Don't get me wrong, I love the proliferation of APIs and easily-integrated services over the past 20 years. We're all one interdependent family, for better and for worse.