13 ms·

Massive Dyn DNS outage

391 points by jtmarmon 10y ago

Sites down: - DYN - Twitter - Etsy - Github - soundcloud - spotify - heroku - pagerduty - shopify - intercom (app, not landing page) Note that if these sites seem to be up to you, it's likely that your machine has cached the DNS response for these sites. Some of these sites seem to work when using a UK VPN

ptrmcrthr 10y ago
Dyn seems down
scrollaway 10y ago
github.com itself isn't resolving in some parts of the world. status.github.com is a beautiful default nginx 404 page.
- deleted 10y ago
  [deleted]
cloudaphant 10y ago
DYN is experiencing a DDoS https://www.dynstatus.com/incidents/nlr4yrr162t8 https://www.dynstatus.com/incidents/nlr4yrr162t8
- darrenp278 10y ago
  Yep. Seems to be the root of the problem.
kchoudhu 10y ago
Reddit is out -- my productivity is up, and I'm not happy about it.
- deleted 10y ago
  [deleted]
- cloudaphant 10y ago
  Working here. Try a non-US VPN if you want to stop doing work again.
  - kchoudhu 10y ago
    Ran my VPN ansible script on a London-based VPS. I'm back to not working!
- zhemao 10y ago
  Reddit is up for me, but Github is down. Goodbye productivity.
num3thod 10y ago
No parts of Github are up for me (Ottawa). Over to you, Bitbucket.
mikemcquaid 10y ago
GitHub employee here. We're monitoring an incident with our upstream DNS provider: https://twitter.com/githubstatus/status/789433336083001344 https://twitter.com/githubstatus/status/789433336083001344
- jtmarmon 10y ago
  not very useful to post a status update on a site that is experiencing the same issues with the same DNS provider. maybe post a github gist? oh wait...
  - cylinder 10y ago
    Should there be a global internet status page at an easily memorizable "vanity" IP address?
    - planteen 10y ago
      Seriously, just like the 8.8.8.8 name server. I used to work at a place where the firewall blocked accessing pages by IP though.
- alfalfasprout 10y ago
  Hahahahaha you do realize Twitter is one of the affected sites, right?
  - mohsinr 10y ago
    Exactly, I see tweet in Google SERPS for GitHub , I try to access and twitter too is down like Github.
- mirekrusin 10y ago
  Can you CC on HN, twitter doesn't work for people as well. HN seems to be fine.
  - 10y ago
drinchev 10y ago
CircleCI is also down. As well as github hooks to slack.
emes001 10y ago
thenextweb.com seems to be down as well (connecting from DC)
tair 10y ago
Intercom as well :(
peterjancelis 10y ago
Heroku is down as well.
deleted 10y ago
[deleted]
gregmilani 10y ago
Airbn is also down :(
num3thod 10y ago
I just realised how online adult entertainment has the most redundancy of any Internet service category, bar none.
- deno 10y ago
  You’re right :) $ dig @8.8.8.8 ns +short pornhub.com ns2.p44.dynect.net. ns3.p44.dynect.net. ns4.p44.dynect.net. sdns3.ultradns.net. sdns3.ultradns.com. sdns3.ultradns.org. sdns3.ultradns.biz. ns1.p44.dynect.net. PornHub has better DNS redundancy than Github and Twitter.
  - drmunklung 10y ago
    I'm sure this is to circumvent blacklisting rather than for this scenario.
    - deno 10y ago
      I can’t think of any type of blacklisting this would prevent. Care you elaborate?
  - dtoombs 10y ago
    Amazon does too: C:\>nslookup > set type=ns > amazon.com Non-authoritative answer: amazon.com nameserver = ns3.p31.dynect.net amazon.com nameserver = pdns1.ultradns.net amazon.com nameserver = ns4.p31.dynect.net amazon.com nameserver = pdns6.ultradns.co.uk amazon.com nameserver = ns1.p31.dynect.net amazon.com nameserver = ns2.p31.dynect.net
gamache 10y ago
Important: PagerDuty.com seems affected by this outage. So keep a real good eye on your graphs today -- you might not receive the alert.
- sn 10y ago
  pagerduty has switched to AWS + 4 different root TLDS: $ host -t NS pagerduty.com pagerduty.com name server ns-219.awsdns-27.com. pagerduty.com name server ns-739.awsdns-28.net. pagerduty.com name server ns-1198.awsdns-21.org. pagerduty.com name server ns-1569.awsdns-04.co.uk. I think I'm satisfied.
  - gamache 10y ago
    I'm not trying to single out PagerDuty in being vulnerable. They run a particularly crucial service, and I'm sure they are doing everything they can to get out from under this. That said, I still can't load www.pagerduty.com in a browser right now. :/
ekalvi 10y ago
PubNub is down.
james_a_craig 10y ago
Seems to be fine from the UK, so those of you with suitable VPNs might like to try that.
liveoneggs 10y ago
a bunch of services using fastly are also impacted
CarVac 10y ago
launchpad.net seems to be down. edit: Using Google public DNS fixes things.
45h34jh53k4j 10y ago
This is going to be a fun day. This little DNS outage is likely to cause millions of lost revenue for many industries. Im dead in the water and I cant complain on twitter :-(
- 6d6b73 10y ago
  change your name server at your registrar to something else, add all necessary entries in the new DNS and be up before DDoS is stopped.
  - drinchev 10y ago
    Sadly we're too interconnected. Every company that relies on that DNS should do what you suggest, but the control is definitely not in our ( users ) hands.
    - 6d6b73 10y ago
      Yeah, users are screwed. Unless they have a little more experience with how unreliable cloud can be, and they made a local copy of everything* their work depends on, just in case. *Everything that can be local.
  - 45h34jh53k4j 10y ago
    nah our corp dns is fine, its all the cloud services we and everyone else uses. Thank Sergey and Larry their stuff still works
- exolymph 10y ago
  I found out Twitter was down because I wanted to go complain about PayPal, lol.
- justinzollars 10y ago
  git still works on your local! Get back to work
ljosa 10y ago
We switched from Dyn to Route53 a few weeks ago … lucky.
minznerjosh 10y ago
npmjs.com seems to be down as well.
6d6b73 10y ago
All this talk about redundancy, real-time apps, scalable architecture and and a "simple" DDOS against DNS architecture brings half of the internet down. Honestly did nobody think about having a spare dns at some other company? or even backup dns server exactly for a situation like that?
- ljosa 10y ago
  The TTL for the glue records of a .com domain is 48 hours, so even if you have Route53 set up and ready to go, it takes a long time to switch the zone away from Dyn. We switched from Dyn to Rout53 a few weeks ago. It took about 12 hours before half of the traffic had shifted over.
  - 6d6b73 10y ago
    That's the reason to have your DNS at at least two different companies, working in tandem. In a case where one is down, your Unicorn Corp doesn't go down with it.
    - St-Clock 10y ago
      How does that work in practice? Even if I set NS records pointing to two different DNS providers, I don't think a DNS client would automatically switch and retry if one is too slow to respond/times out.
      - 6d6b73 10y ago
        Why not? That's the whole idea behind having more than one NS server isn't it?
      - acdha 10y ago
        Most DNS resolvers will automatically try each NS record until they get a response. That might be your ISP rather than your iPhone but that's an old practice going back to when the internet was even less reliable because some random Sun box under someone's desk failed. Modern web browsers will also do the same thing if a query returns multiple A records and they get a connection error.
johansch 10y ago
Works fine from Asia. (When asking e.g. ns1.p34.dynect.net directly.)
vdfs 10y ago
Shopify too: https://status.shopify.com/incidents/z714f9fjg9q0 https://status.shopify.com/incidents/z714f9fjg9q0
bobjordan 10y ago
Must be the Russians.
mugsie 10y ago
From Ireland all of the above are resolving (with no cache).
leabdalla 10y ago
zendesk admin panel is down here in brazil too
deleted 10y ago
[deleted]
FullMtlAlcoholc 10y ago
Chiming in from sunny Los Angeles Up as of 5:30AM PST: Twitter, Etsy, Github, Soundcloud, Spotify, Dyn DNS Down: Heroku, Pagerduty Might want to use a VPN to another area
raffandi 10y ago
Sendgrid support page is down as well https://support.sendgrid.com/hc/en-us https://support.sendgrid.com/hc/en-us
- errrnis 10y ago
  SendGrid's support is hosted on Zendesk, which is (unfortunately) still experiencing the outage. My company's doc site (also on Zendesk) is down too.
ekalvi 10y ago
Sentry (at least web dashboard) is down.
fabiorogeriosj 10y ago
GitHub pretty slow in Brazil
- ShaneOG 10y ago
  A lot of the bigger sites are very slow from Brazil today :( And SP is also suffering from a lack of power in some areas. Happy Friday!
  - fernandotakai 10y ago
    yeah, i thought it was my internet but it seems like a ton of people are having slow internet issues.
- fernandotakai 10y ago
  same here. getting the "page taking way too long to load" page every time.
- jeanlucas 10y ago
  Brazucas unite :-) Alguém teve algum outro problema fora os citados aqui?
fabiorogeriosj 10y ago
GitHub pretty slow in Brazil
verytrivial 10y ago
Is this another IP webcam etc. attack? Does anyone know of a write-up from a researcher in possession of one of these currently exploited bits of kits?
tbarbugli 10y ago
Include all website that use Github to sign-in or sign-up
calgoo 10y ago
All sites look good for me in Europe using 8.8.8.8. Edit: It actually looks like most of the sites are loading faster then normal. Edit2: I have cleared the cache's on my machine and router. Still works.
freshyill 10y ago
Five Thirty Eight! Today was a very long train ride without Twitter or the poll tracker.
- Aoreias 10y ago
  They stayed up all day, they just blocked rendering until they loaded some twitter assets, causing it to take 20-30 seconds to load until those assets timed out.
sergc 10y ago
CNN and TheGuardian are down as well
- s_kilk 10y ago
  Yup, I'm seeing TheGuardian down, from Edinburgh/Scotland.
michaelmwangi 10y ago
Pulling on AWS using github.com as origin failed had to add the ip to /etc/hosts.
nmd 10y ago
CircleCI is also having issues viewing and running tests, viewing from Amsterdam: https://status.circleci.com/ https://status.circleci.com/
crawdingle 10y ago
box.com seems to be down too.
- expjpi 10y ago
  also seeing that here in mountain view
- szarecor 10y ago
  Down in the midwest as well
royalharsh95 10y ago
gitter as well (https://twitter.com/gitchat https://twitter.com/gitchat)
momchenr 10y ago
Paypal
deleted 10y ago
[deleted]
lukevers 10y ago
Braintree is also experiencing these problems.
deleted 10y ago
[deleted]
Veen 10y ago
Looks like unsplash.com is down too
Veen 10y ago
Looks like unsplash.com is down too
utefan001 10y ago
Okta seems to be down.
mariuscoto 10y ago
http://okta.com http://okta.com is not working
jrod2121 10y ago
bigcommerce, volusion, new relic, optimizely, wistia, volusion, aweber, cnn, campaign monitor, all down for me. The biggest thing is seeing that ALL shopify stores are offline, so much $$$ being lost right now.
cafard 10y ago
python.org and cpan.org are down.
jssjr 10y ago
I'm a GitHub employee and want to let everyone know we're aware of the problems this incident is causing and are actively working to mitigate the impact. "A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769 https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well. [x-posted on https://news.ycombinator.com/item?id=12759697 https://news.ycombinator.com/item?id=12759697 as well]
- cynosurexy 10y ago
  how did you post on twitter which is down too?
  - eric_the_read 10y ago
    Cached DNS response.
  - SteveNuts 10y ago
    Twitter isn't down, just DNS resolution of twitter.com
lmpower 10y ago
just add the open DNS you are allset
lmpower 10y ago
use open DNS
- prefect42 10y ago
  Side Note: news (to me at least) Cisco had acquired Open DNS late Aug. Now I'll have to read all the doco to see what impact (if any) there is for personal usage.
celticpiping 10y ago
https://github.com/ https://github.com/ is up for me. North East
rotauserch 10y ago
Need round robin dns - geographically dispersed
- rpearl 10y ago
  That's not the part of the system that's under attack
constantskeptic 10y ago
If you add these sites ip addresses to your /etc/hosts file locally then you don't have to worry about this for the most part.
celticpiping 10y ago
https://github.com/ https://github.com/ is up for me.. NorthEast USA maybe I missed something
- jrod2121 10y ago
  looks like they are all back online now.
rotauserch 10y ago
Need to setup a round robin dns - geographically dispersed
shadowfacts 10y ago
All of the sites listed are back up for me, and still using Dyn DNS name servers.
- celticpiping 10y ago
  ditto for me
matiasb 10y ago
Mongo Cloud is having trouble too!
deleted 10y ago
[deleted]
timbl 10y ago
8.8.8.8 seems to know about github.com and github.io at least -- so I can work. (Maybe 8.8.8.8 built to resist censorship and therefor also attack?)
tarialys 10y ago
My Uber partner app crashed at 8am I was trying to complete a trip and it frozen my phone. It took about 5min for me to be able to enter back but it asked me for my SSN and permission to do a background check which is standard by Uber but I had already done so. Should I be concern that my personal data has been compromised? I contacted Uber but their idiots support people don't seem to have a clue and third fix is super basic like restart your phone, turn airplane mode or data on and off
gianarb 10y ago
http://downdetector.com/status/level3/map/ http://downdetector.com/status/level3/map/
FartyBumMcPoops 10y ago
i have highly available biscuits
FartyBumMcPoops 10y ago
i have highly available biscuits
danseasner 10y ago
Wiki leaks news held back all day for the most part the 20th. Clinton mafia attack. Had diplo,Ativan dinner in n y. Last night
danseasner 10y ago
Xclinton hack. News held back the 20th through 12 n 13th dump. Was a diplomatic dinner last night in n y c I live on east coast n e c
Avalanchee 10y ago
LinkedIn?
Avalanchee 10y ago
Linkedin doesn't seem to load page's contents as well.
danyork 10y ago
Many articles are now coming out about this outage this morning: http://www.techmeme.com/161021/p3#a161021p3 http://www.techmeme.com/161021/p3#a161021p3
dg123r 10y ago
And aws: 6:13 AM PDT [RESOLVED] Between 4:31 AM and 6:10 AM PDT, we experienced errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region. During the issue, customers may have experienced failures indicating "hostname unknown" or "unknown host exception" when attempting to resolve the hostnames for AWS services and EC2 instances. This issue has been resolved and the service is operating normally.
mszyndel 10y ago
For us here in Europe/Germany Paymill is also down!
yellowsir 10y ago
time to take a look at git-ssb p2p git+"github" https://git-ssb.celehner.com/%25n92DiQh7ietE%2BR%2BX%2FI403LQoyf2DtR3WQfCkDKlheQU%3D.sha256 https://git-ssb.celehner.com/%25n92DiQh7ietE%2BR%2BX%2FI403L...
mohsinr 10y ago
GitHub, Twitter down for me in Pakistan.
- victorbojica 10y ago
  down in Romania too
- camoby 10y ago
  Same here. Just now. In the UK.
  - Enxo 10y ago
    Can access Twitter just fine from the UK. Playstation network is down though.
camoby 10y ago
In the UK here and Twitter and Github just went dark.
- hujkikwhe 10y ago
  One hour ago, Wikileaks twitted: > PHOTO: Heavily armed 'police' appear outside Ecuadorian Embassy in London where Julian Assange has political asylum (photo, Tuesday morning) Hopefully just a coincidence -- seeing as Twitter is a key communication conduit for Wikileaks.
- Enxo 10y ago
  And playstation network. I can't play online and it's Friday :(
y-love 10y ago
Github still down for me in New York.
jbaviat 10y ago
Issue came from dynect.net: $ dig +trace github.com ; <<>> DiG 9.8.3-P1 <<>> +trace github.com [...] ;; Received 488 bytes from 192.228.79.201#53(192.228.79.201) in 154 ms github.com. 172800 IN NS ns1.p16.dynect.net. github.com. 172800 IN NS ns3.p16.dynect.net. github.com. 172800 IN NS ns2.p16.dynect.net. github.com. 172800 IN NS ns4.p16.dynect.net. ;; Received 178 bytes from 192.42.93.30#53(192.42.93.30) in 54 ms ;; connection timed out; no servers could be reached
- jorts 10y ago
  They're investigating now: https://www.dynstatus.com/incidents/nlr4yrr162t8 https://www.dynstatus.com/incidents/nlr4yrr162t8
- DonaldFoss 10y ago
  As of 2016-10-21T09:20:19Z, it's doing better: -------------------------------------------------------------------- <Commodore Pet 16 Unix>:$ dig +trace github.com ; <<>> DiG 9.8.3-P1 <<>> +trace github.com ;; global options: +cmd . 17430 IN NS i.root-servers.net. . 17430 IN NS a.root-servers.net. . 17430 IN NS b.root-servers.net. . 17430 IN NS l.root-servers.net. . 17430 IN NS d.root-servers.net. . 17430 IN NS g.root-servers.net. . 17430 IN NS h.root-servers.net. . 17430 IN NS c.root-servers.net. . 17430 IN NS e.root-servers.net. . 17430 IN NS j.root-servers.net. . 17430 IN NS k.root-servers.net. . 17430 IN NS m.root-servers.net. . 17430 IN NS f.root-servers.net. ;; Received 228 bytes from 8.8.8.8#53(8.8.8.8) in 382 ms com. 172800 IN NS m.gtld-servers.net. com. 172800 IN NS k.gtld-servers.net. com. 172800 IN NS a.gtld-servers.net. com. 172800 IN NS b.gtld-servers.net. com. 172800 IN NS f.gtld-servers.net. com. 172800 IN NS j.gtld-servers.net. com. 172800 IN NS e.gtld-servers.net. com. 172800 IN NS c.gtld-servers.net. com. 172800 IN NS h.gtld-servers.net. com. 172800 IN NS l.gtld-servers.net. com. 172800 IN NS d.gtld-servers.net. com. 172800 IN NS g.gtld-servers.net. com. 172800 IN NS i.gtld-servers.net. ;; Received 488 bytes from 192.228.79.201#53(192.228.79.201) in 363 ms github.com. 172800 IN NS ns1.p16.dynect.net. github.com. 172800 IN NS ns3.p16.dynect.net. github.com. 172800 IN NS ns2.p16.dynect.net. github.com. 172800 IN NS ns4.p16.dynect.net. ;; Received 178 bytes from 192.42.93.30#53(192.42.93.30) in 134 ms github.com. 300 IN A 192.30.253.113 github.com. 86400 IN NS ns4.p16.dynect.net. github.com. 86400 IN NS ns3.p16.dynect.net. github.com. 86400 IN NS ns1.p16.dynect.net. github.com. 86400 IN NS ns2.p16.dynect.net. ;; Received 130 bytes from 2001:500:90:1::16#53(2001:500:90:1::16) in 25 ms I ended up sending all my developers a patch to their hosts file... Hopefully they'll roll it back shortly. --Donald
crgwbr 10y ago
DNS came back for a while and just started flaking again. $ nslookup github.com ;; Got SERVFAIL reply from 8.8.8.8, trying next server Server: 10.0.0.1 Address: 10.0.0.1#53 Non-authoritative answer: Name: github.com Address: 192.30.253.112 $ nslookup twitter.com ;; Got SERVFAIL reply from 8.8.8.8, trying next server ;; Got SERVFAIL reply from 8.8.4.4, trying next server Server: 10.0.0.1 Address: 10.0.0.1#53 Non-authoritative answer: Name: twitter.com Address: 199.16.156.102 Name: twitter.com Address: 199.16.156.230 Name: twitter.com Address: 199.16.156.198 Name: twitter.com Address: 199.16.156.70
ZeroActual 10y ago
Twitter/GitHub/Soundcloud/etc all just went down for me (UK), also The Verge and The Next Web.
tokm 10y ago
twitter down again now....
tokm 10y ago
Twitter down again...
MattieTK 10y ago
deck.news also down
BlackGuyCoding 10y ago
On the West Coast, Twitter/SoundCloud/GitHub just went down for me...
- y-love 10y ago
  Github is working (slowly) through UK VPN for me right now.
- exolymph 10y ago
  Also West Coast. Twitter and PayPal are both down.
chromaton 10y ago
PayPal isn't working for me. Could cost a lot of people a lot of money.
- exolymph 10y ago
  I just need to invoice someone, but I would be pissed if I used them to process ecommerce or SaaS payments.
- apmoore 10y ago
  I just noticed it when I tried to do a bank transfer. Ebay was also loading slowly for a while but I didn't think anything of it. Hopefully this is resolved soon.
oelmekki 10y ago
Anyone has any idea which hosts desktop spotify app is trying to reach? `netstat -punta | grep spotify` tells me they are on those IP:port : * 151.101.120.246:80 * 194.132.198.50:4070 Problem is, I don't know what hosts this is supposed to be, and `dig -x` is not very helpful...
ericholiveira10 10y ago
Anyone with issues on AWS in North Virginia?
- onlydole 10y ago
  It has been affecting a few offerings all day...specifically WorkMail for me.
- paulddraper 10y ago
  We've seen zero problems (standard stuff: EC2, S3, RDS, Redshift). I expected that with 1/3 of the internet down something would be broken.
crobertsbmw 10y ago
It's Hillary's fault...
ericholiveira10 10y ago
Anyone experiencing problems with servers on AWS North Virginia?
komali2 10y ago
Add 1800 contacts, ynab. Maybe not important to some people but it was how I realized something was up.
apmoore 10y ago
In CA - Spotify starting going in and out around 9:15a, then completely shut down at 9:20a (no offline playlists were available). As of 9:49a my offline playlists are back up, but nothing else. Twitter was working from my mobile about 15 min ago, but not on my desktop. Now both are down. Can't believe Twitter is down. On my mobile I'm getting notifications, but can't view them. The people who are sending updates are mostly on the East Coast and Canada, so maybe it's back up for them.
- grav 10y ago
  I doubt that your offline playlists were affected ...
  - apmoore 10y ago
    Not sure why they were, but I tried 6 different offline playlists and each of them said "can't play the current song", then cited internet detection... which hadn't been an issue. Now (10:04a) all of my playlists are working again.
    - illumin8 10y ago
      Spotify has a terrible network connection algorithm that will sit there and spin forever trying to open the app if it thinks you're on the network but you're really not. Put your phone in airplane mode, then re-open Spotify, and magically, all of your offline playlists will play.
- y-love 10y ago
  On the East Coast, Twitter still down for me on mobile. All the Twitter media resources (twimg.com) are also not appearing.
  - apmoore 10y ago
    Twitter is still not loading for me, but just checked Hootsuite and it is up and running now. Hootsuite was down for a while also.
lukeredpath 10y ago
Lots of sites down for me in the UK. I was using my ISPs DNS (Sky), switched to 8.8.8.8 and not much better.
maaaats 10y ago
I had issues with netflix as well. Netflix.com was resolving but lot of the resources from sub/other domains didn't load.
y-love 10y ago
Now even the DYN Status page is inaccessible from the Eastern US https://www.dynstatus.com/incidents/nlr4yrr162t8 https://www.dynstatus.com/incidents/nlr4yrr162t8 (Github still down from the Eastern US.)
Mingso 10y ago
It's 9:55am PST California and Twitter is down.
- drcoopster 10y ago
  Did you change your clocks early?
Mingso 10y ago
It's 9:56am PST California and Twitter is down.
- jeff303 10y ago
  Github and Twitter are both unreachable for me still. Chicago area.
deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
Mingso 10y ago
It's 9:58am PST in California and Twitter is still down.
deleted 10y ago
[deleted]
Soulo_Guero 10y ago
10:02 PST in CA and Netflix (and, yes, still Twitter) is down.
- Soulo_Guero 10y ago
  Correction: Netflix is "half" loading.
deleted 10y ago
[deleted]
ewams 10y ago
Amazon is having an outage. Okta is also down because of this.
kyler887 10y ago
Runescape is also down
kyler887 10y ago
Runescape is also down
dfpr 10y ago
I managed to access twitter from Ecuador using Opera, activating the built-in VPN and setting it to Singapure.
mgalgs 10y ago
Same thing happened with hover last week... Same attacker perhaps? Maybe hover was just a test drive of their DNS DDoS machinery? https://twitter.com/HoverStatus/status/786296893013766144 https://twitter.com/HoverStatus/status/786296893013766144
Sami_Lehtinen 10y ago
Namecheap is also down.
moonlightsky 10y ago
https://ease.apperian.com https://ease.apperian.com is down over here
deleted 10y ago
[deleted]
paulddraper 10y ago
I thought DNS (particularly public) was basically immune to DDoS? If one DNS server is down, use the cached result or another server. DNS is some of the most distributable, cachable data I can imagine.
- facorreia 10y ago
  > DNS is some of the most distributable, cachable data I can imagine. Indeed. How can it be so broken? And, it seem that it would make sense to use the last known value if upstream is unavailable.
  - nzp 10y ago
    That would be a horrible idea from a security standpoint. Although not easy by any means for an ordinary attacker, at least one exploit scenario is obvious.
- pepon 10y ago
  Seriously, please someone explain this. I thought DNS info is propagated among DNS servers, and it is cached on all of them... How can an attack to a single DNS company cause this outage??
  - atom_enger 10y ago
    Explained above for ya
- 274below 10y ago
  True, but DNS records have TTLs that expire, which forces you to reach back out to the authoritative server for the zone. A very decent number of records will have a TTL of 1hr, which means that a total DNS server outage lasting longer than an hour = 100% failure rate. You could always increase the TTL to a day (or whatever other arbitrary value), but that means that you'll be waiting upwards of a day for clients to see any updates to your DNS records. It's a tradeoff between different types of resiliency. This is a simple case of a DNS provider evidently not having their crap together when it comes to a DDoS of any decent size. There are plenty of other DNS providers out there who are working fine currently, this DDoS is limited to just the one. It's all of that company's clients that are impacted. I'm sure that DynETC will post something afterwards about how this was the largest DDoS they've ever encountered by many orders of magnitude and that there was no way for anyone to ever be prepared to take so much traffic... but at this point I think they're inept. edit: just came back up. The TTLs on a few common records explain why this was so obviously a problem... - www.reddit.com: 300 seconds (pointer to a fastly.net address, which is hosted via dynect) - prod.reddit.map.fastlylb.net: 30 seconds - api.twitter.com: 300 seconds - herokussl.com: 3600 seconds So, an outage of five minutes is enough to take out access to all of reddit and twitter. An outage of an hour is enough to take out heroku. Of course, these are best case scenarios: in reality, 50% of your users would lose access to reddit after 15 seconds (the fastly record), and so on.
nodesocket 10y ago
Pagerduty down, which is not very helpful.
- wolf31o2 10y ago
  Critical infrastructure was outsourced and now they've gone down! Yay, the "cloud" and all it's "glory" where some joker with an API thinks that magically means always available.
koryteg 10y ago
Rollbar!!!! noooooo!!!!!
Enzo40Dog 10y ago
Ticketfly.com. Damn I really wanted to buy die Antwoord tickets
MomoHo 10y ago
add moz.com
frjalex 10y ago
GitHub seems fine for me.
4GLyle32 10y ago
When will outage be restored??
4GLyle32 10y ago
When will this outage be repaired??
GilsonSoares 10y ago
www.indeed.com too
GilsonSoares 10y ago
www.indeed.com too.
fidla 10y ago
Still down. 2:17pm EST
fidla 10y ago
You can still access these sites using a VPN like Tunnel Bear, or a VPN browser like Opera
parkerproject 10y ago
npm is down as well
murmanda 10y ago
www.sonatype.com also down (from Germany)
AlexRazor 10y ago
I think that many of the IP addresses that are involved in this DDoS attack can be found in the BlackList database. As example https://cleantalk.org/blacklists https://cleantalk.org/blacklists There are 2 084 161 IP's blacklisted. https://cleantalk.org/help/fraud-prevention https://cleantalk.org/help/fraud-prevention
murmanda 10y ago
www.sonatype.com also down (at least from Germany)
AlexRazor 10y ago
I think that many of the IP addresses that are involved in this DDoS attack can be found in the BlackList databases. As example https://cleantalk.org/blacklists https://cleantalk.org/blacklists There are 2 084 161 IP's blacklisted.
Davezilla 10y ago
my.wpengine.com is also down. The public site is still up.
thesmart 10y ago
It seems incredibly misguided to plug an IP address into hosts from a forum post. Is there an official source from GitHub?
- deleted 10y ago
  [deleted]
temp1231231 10y ago
Easy way to get around it for Github: https://www.apptic.me/blog/adding-github-to-your-hosts-file.php https://www.apptic.me/blog/adding-github-to-your-hosts-file.... Same idea, just add the IPs to your hosts file.
deleted 10y ago
[deleted]
videoshooter 10y ago
Zoho.com, one of my email providers is down as well
koyao 10y ago
Amazon DNS must be getting loads of new customers today :)
- mej10 10y ago
  Even though their status page says everything is basically fine -- oh just a blip with API dns this morning no biggie -- they have had major problems today too. Like, can't connect to dynamodb, sqs, kinesis. Unfortunately they lie through their status page pretty regularly. I am not sure what it would take for them to admit they had a major problem.
  - shuumai 10y ago
    Apparently us-east-1 was the only region that exclusively used dyndns (all other regions have failover strategies to other providers and that's why there wasn't downtime anywhere else but us-east and eu-west). They resolved the problem pretty quickly, but it drove our NOC insane...fortunately downtime for us was only about half an hour or so.
    - mej10 10y ago
      Yeah, thats about a half hour of down time for every AWS East based company... who knows how much lost revenue -- and yet they just say the service is green with a little info tag. If our service is down for any customer (much less an entire segment of customers) we at least publish that we had a partial outage.
E1Kenobi 10y ago
I cant get into the PlayStation Network either, I guess its time to go outside.
adhir8 10y ago
Also down: sie.sps.columbia.edu/enroll
stephb 10y ago
groupme.com is down in nyc... i should memorize friends' phone numbers...
realtorjohnf 10y ago
AthenaNet.athenahealth.com , one of Americas largest health care medical software providers is down, effecting 80,000 physicians and their patients.
michaelxia 10y ago
Does anyone have heroku's ip's?
jontodd 10y ago
It looks like Verizon customers continue to have issues resolving Dyn DNS. Wondering if Verizon has actually blocked Dyn requests
- ted_dunning 10y ago
  Dyn's status page still says they are working the problem. Why would Verizon block access to a downed service?
  - jontodd 10y ago
    Not entirely sure. Perhaps some of the DDOS was flowing through to them but that's complete speculation. I mostly wanted to throw this out there for others who are trying to segment the impact of the issue.
dan0009 10y ago
roblox.com is offline too.
dan0009 10y ago
roblox.com is offline aswell.
hcngo 10y ago
Quora.com cannot be resolved either.
deleted 10y ago
[deleted]
stef25 10y ago
Github down, can't deploy code!
GilsonSoares 10y ago
Whatsapp too: # host -tns whatsapp.com whatsapp.com name server ns2.p13.dynect.net. whatsapp.com name server ns1.p13.dynect.net. whatsapp.com name server ns4.p13.dynect.net. whatsapp.com name server ns3.p13.dynect.net.
- meyoulit 10y ago
  Thanks
yachtycane 10y ago
Why did they have to ddos the EA servers too I just wanna play battlefield 1
meyoulit 10y ago
dot.
machty 10y ago
poop.bike is still up
semartin 10y ago
U.S DDOS to block Assange cable drop
apmoore 10y ago
Is anyone noticing significant delays visiting CNN, ebay, orbitz? It took 15 seconds for the Orbitz page to load (tried from Google Chrome and Safari). It seems like outside of the core group of sites effected (GitHub, Twitter, PayPal), other major websites are having serious delays loading..
webandtech 10y ago
Anyone with an app affected by this because you use Heroku's SSL endpoint can fix this by switching to SNI. Unfortunately Heroku's API is also affected. There is a workaround... just edit your hosts file to point to the proper ip addresses temporarily. add to /etc/hosts 23.21.149.112 api.heroku.com 107.21.99.123 ssl-doctor.heroku.com Then use Heroku CLI to switch your app to the new SNI endpoint heroku certs:add --type sni server.crt server.key -a YOUR_APP That will spit out a new host name that you can point your DNS to and still be online.
- cstavish 10y ago
  Thanks a million, from me and the people who invested in my startup. edit: apparently SNI isn't supported on legacy dynos.
jemini 10y ago
We are seeing inconsistent resolutions on fppaco.org. sometimes it resolves to a 4.x address and sometimes a 50.x address. According to whois.net, their DNS servers are DYNect
photogrammetry 10y ago
Twitter, Github, Quora, Soundcloud, Etsy, PayPal, Heroku, The Guardian, WhatsApp and numerous others still down as of 1:00 pm PT. Amazon and Spotify are still up.
- Enxo 10y ago
  Playstation network down last 4 hours. This is in the UK don't know about the rest of the world. Status: https://status.playstation.com/en-gb/ https://status.playstation.com/en-gb/
clavallee 10y ago
Twitter still down Ohio
clavallee 10y ago
Twitter still down - Ohio
lawlorslaw 10y ago
Change your DNS settings and some of these websites will now work. http://www.dailydot.com/layer8/how-to-change-dns-settings-mac-windows/ http://www.dailydot.com/layer8/how-to-change-dns-settings-ma...
Andres102116 10y ago
I have a shop on etsy. It is functional now.
Andres102116 10y ago
I have a shop on etsy, and it is functional now.
Andres102116 10y ago
I have a shop on etsy.com. It is functional at this time (1418 PDT).
steppnav 10y ago
Remind me to populate my /etc/hosts file after this blows over.
aledalgrande 10y ago
hosts file settings: - to deploy to Heroku 174.129.22.147 git.heroku.com - to enable Twitter from Web 104.244.42.65 twitter.com 93.184.220.70 pbs.twimg.com 104.244.43.7 o.twimg.com - to access Github (as another poster wrote): 192.30.253.113 github.com 151.101.4.133 assets-cdn.github.com You can find any other IPs you need on sites like http://www.hcidata.info/host2ip.cgi http://www.hcidata.info/host2ip.cgi Just remember to rollback your hosts file after the outage finishes.
matobago 10y ago
This is not a dyn error, this is all the companies failed to prevent this to happen, is an old practice having different NS Servers for your domain, but it can prevent this to happen.
deleted 10y ago
[deleted]
jasonfb 10y ago
It appears that people (domain name administrators) are switching away from Dyn as their DNS provider, as per https://status.heroku.com/incidents/965 https://status.heroku.com/incidents/965 (issued 19:21 UTC). Does this mean that other DNS providers (Amazon Route 53, CDNetworks, CloudFlare, DNSimple, easyDNS, Google DNS, Verisign, OpenDNS) could potentially become targets? -Jason (5:13 PM EDT; 21:13 UTC)
jasonfb 10y ago
confirmed three (3) coordinated attacks on Dyn's infrastructure today: 11:10 UTC, 13:52 GTM, and the third time is unknown -- Dyn has not specified the time of the 3rd attack today on their dynstatus.com page (likely because the whole thing can be considered ongoing). Can anyone speak to when the 3rd attack began today?
2048-Bit 10y ago
Dyn can't figure out how to fix it because they can't get to stack overflow! :D
clavallee 10y ago
The outage is attributed to routing 576 Autonomous systems and 3431 prefixes.
latentsurfer 10y ago
The DDOS also hit small business websites in North America. I know atleast three companies who experienced the DDOS attacks. The IPs that were port scanning prior to the attack are 173.254.216.66, 128.52.128.105 (MIT Edu), 172.98.67.39. I hope there is some action against these hackers.
DyslexicAtheist 10y ago
you can watch BGP routes changing (as we speak) here https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16 https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16
rcummings 10y ago
Sounds like IETF needs to update the DNS standards with 'intelligent' features, so that we can have some consistency.
Bentsie 10y ago
More likely this idiot is doing it. Look at the location spots. Ben Callis. He's a little assbot programmer that likes hacking bank accounts and stealing people's shit. Runs around in a Roman Catholic Church. Pure Hoodlum and ugly disgusting rat. He's stealing networks and fuck-flashing it probably prior to the elections. If you want to know who's been hacking away at your banks and your stuff disappearing it's this asshole. He smiles and pretends but he's a career criminal. You see this maggot report him to FBI. Absolute Fiend and a pathetic piece of rotten crap. Dangerous too got ahold of some weapons starts acting up. BenCallis-CohhCarnage is a FIEND. He hassles and hurts women and children is a human-trafficker and disgusting piece of morbid crap.
deleted 10y ago
[deleted]

391 points by jtmarmon 10y ago

ptrmcrthr 10y ago

Dyn seems down

scrollaway 10y ago

github.com itself isn't resolving in some parts of the world. status.github.com is a beautiful default nginx 404 page.

deleted 10y ago
[deleted]

cloudaphant 10y ago

DYN is experiencing a DDoS https://www.dynstatus.com/incidents/nlr4yrr162t8 https://www.dynstatus.com/incidents/nlr4yrr162t8

darrenp278 10y ago
Yep. Seems to be the root of the problem.

kchoudhu 10y ago

Reddit is out -- my productivity is up, and I'm not happy about it.

deleted 10y ago
[deleted]
cloudaphant 10y ago
Working here. Try a non-US VPN if you want to stop doing work again.
- kchoudhu 10y ago
  Ran my VPN ansible script on a London-based VPS. I'm back to not working!
zhemao 10y ago
Reddit is up for me, but Github is down. Goodbye productivity.

num3thod 10y ago

No parts of Github are up for me (Ottawa). Over to you, Bitbucket.

mikemcquaid 10y ago

GitHub employee here. We're monitoring an incident with our upstream DNS provider: https://twitter.com/githubstatus/status/789433336083001344 https://twitter.com/githubstatus/status/789433336083001344

jtmarmon 10y ago
not very useful to post a status update on a site that is experiencing the same issues with the same DNS provider. maybe post a github gist? oh wait...
- cylinder 10y ago
  Should there be a global internet status page at an easily memorizable "vanity" IP address?
  - planteen 10y ago
    Seriously, just like the 8.8.8.8 name server. I used to work at a place where the firewall blocked accessing pages by IP though.
alfalfasprout 10y ago
Hahahahaha you do realize Twitter is one of the affected sites, right?
- mohsinr 10y ago
  Exactly, I see tweet in Google SERPS for GitHub , I try to access and twitter too is down like Github.
mirekrusin 10y ago
Can you CC on HN, twitter doesn't work for people as well. HN seems to be fine.
- 10y ago

drinchev 10y ago

CircleCI is also down. As well as github hooks to slack.

emes001 10y ago

thenextweb.com seems to be down as well (connecting from DC)

tair 10y ago

Intercom as well :(

peterjancelis 10y ago

Heroku is down as well.

deleted 10y ago

[deleted]

gregmilani 10y ago

Airbn is also down :(

num3thod 10y ago

I just realised how online adult entertainment has the most redundancy of any Internet service category, bar none.

deno 10y ago
You’re right :) $ dig @8.8.8.8 ns +short pornhub.com ns2.p44.dynect.net. ns3.p44.dynect.net. ns4.p44.dynect.net. sdns3.ultradns.net. sdns3.ultradns.com. sdns3.ultradns.org. sdns3.ultradns.biz. ns1.p44.dynect.net. PornHub has better DNS redundancy than Github and Twitter.
- drmunklung 10y ago
  I'm sure this is to circumvent blacklisting rather than for this scenario.
  - deno 10y ago
    I can’t think of any type of blacklisting this would prevent. Care you elaborate?
- dtoombs 10y ago
  Amazon does too: C:\>nslookup > set type=ns > amazon.com Non-authoritative answer: amazon.com nameserver = ns3.p31.dynect.net amazon.com nameserver = pdns1.ultradns.net amazon.com nameserver = ns4.p31.dynect.net amazon.com nameserver = pdns6.ultradns.co.uk amazon.com nameserver = ns1.p31.dynect.net amazon.com nameserver = ns2.p31.dynect.net

gamache 10y ago

Important: PagerDuty.com seems affected by this outage. So keep a real good eye on your graphs today -- you might not receive the alert.

sn 10y ago
pagerduty has switched to AWS + 4 different root TLDS: $ host -t NS pagerduty.com pagerduty.com name server ns-219.awsdns-27.com. pagerduty.com name server ns-739.awsdns-28.net. pagerduty.com name server ns-1198.awsdns-21.org. pagerduty.com name server ns-1569.awsdns-04.co.uk. I think I'm satisfied.
- gamache 10y ago
  I'm not trying to single out PagerDuty in being vulnerable. They run a particularly crucial service, and I'm sure they are doing everything they can to get out from under this. That said, I still can't load www.pagerduty.com in a browser right now. :/

ekalvi 10y ago

PubNub is down.

james_a_craig 10y ago

Seems to be fine from the UK, so those of you with suitable VPNs might like to try that.

liveoneggs 10y ago

a bunch of services using fastly are also impacted

CarVac 10y ago

launchpad.net seems to be down. edit: Using Google public DNS fixes things.

45h34jh53k4j 10y ago

This is going to be a fun day. This little DNS outage is likely to cause millions of lost revenue for many industries. Im dead in the water and I cant complain on twitter :-(

6d6b73 10y ago
change your name server at your registrar to something else, add all necessary entries in the new DNS and be up before DDoS is stopped.
- drinchev 10y ago
  Sadly we're too interconnected. Every company that relies on that DNS should do what you suggest, but the control is definitely not in our ( users ) hands.
  - 6d6b73 10y ago
    Yeah, users are screwed. Unless they have a little more experience with how unreliable cloud can be, and they made a local copy of everything* their work depends on, just in case. *Everything that can be local.
- 45h34jh53k4j 10y ago
  nah our corp dns is fine, its all the cloud services we and everyone else uses. Thank Sergey and Larry their stuff still works
exolymph 10y ago
I found out Twitter was down because I wanted to go complain about PayPal, lol.
justinzollars 10y ago
git still works on your local! Get back to work

ljosa 10y ago

We switched from Dyn to Route53 a few weeks ago … lucky.

minznerjosh 10y ago

npmjs.com seems to be down as well.

6d6b73 10y ago

All this talk about redundancy, real-time apps, scalable architecture and and a "simple" DDOS against DNS architecture brings half of the internet down. Honestly did nobody think about having a spare dns at some other company? or even backup dns server exactly for a situation like that?

ljosa 10y ago
The TTL for the glue records of a .com domain is 48 hours, so even if you have Route53 set up and ready to go, it takes a long time to switch the zone away from Dyn. We switched from Dyn to Rout53 a few weeks ago. It took about 12 hours before half of the traffic had shifted over.
- 6d6b73 10y ago
  That's the reason to have your DNS at at least two different companies, working in tandem. In a case where one is down, your Unicorn Corp doesn't go down with it.
  - St-Clock 10y ago
    How does that work in practice? Even if I set NS records pointing to two different DNS providers, I don't think a DNS client would automatically switch and retry if one is too slow to respond/times out.
    - 6d6b73 10y ago
      Why not? That's the whole idea behind having more than one NS server isn't it?
    - acdha 10y ago
      Most DNS resolvers will automatically try each NS record until they get a response. That might be your ISP rather than your iPhone but that's an old practice going back to when the internet was even less reliable because some random Sun box under someone's desk failed. Modern web browsers will also do the same thing if a query returns multiple A records and they get a connection error.

johansch 10y ago

Works fine from Asia. (When asking e.g. ns1.p34.dynect.net directly.)

vdfs 10y ago

Shopify too: https://status.shopify.com/incidents/z714f9fjg9q0 https://status.shopify.com/incidents/z714f9fjg9q0

bobjordan 10y ago

Must be the Russians.

mugsie 10y ago

From Ireland all of the above are resolving (with no cache).

leabdalla 10y ago

zendesk admin panel is down here in brazil too

deleted 10y ago

[deleted]

FullMtlAlcoholc 10y ago

Chiming in from sunny Los Angeles Up as of 5:30AM PST: Twitter, Etsy, Github, Soundcloud, Spotify, Dyn DNS Down: Heroku, Pagerduty Might want to use a VPN to another area

raffandi 10y ago

Sendgrid support page is down as well https://support.sendgrid.com/hc/en-us https://support.sendgrid.com/hc/en-us

errrnis 10y ago
SendGrid's support is hosted on Zendesk, which is (unfortunately) still experiencing the outage. My company's doc site (also on Zendesk) is down too.

ekalvi 10y ago

Sentry (at least web dashboard) is down.

fabiorogeriosj 10y ago

GitHub pretty slow in Brazil

ShaneOG 10y ago
A lot of the bigger sites are very slow from Brazil today :( And SP is also suffering from a lack of power in some areas. Happy Friday!
- fernandotakai 10y ago
  yeah, i thought it was my internet but it seems like a ton of people are having slow internet issues.
fernandotakai 10y ago
same here. getting the "page taking way too long to load" page every time.
jeanlucas 10y ago
Brazucas unite :-) Alguém teve algum outro problema fora os citados aqui?

fabiorogeriosj 10y ago

GitHub pretty slow in Brazil

verytrivial 10y ago

Is this another IP webcam etc. attack? Does anyone know of a write-up from a researcher in possession of one of these currently exploited bits of kits?

tbarbugli 10y ago

Include all website that use Github to sign-in or sign-up

calgoo 10y ago

All sites look good for me in Europe using 8.8.8.8. Edit: It actually looks like most of the sites are loading faster then normal. Edit2: I have cleared the cache's on my machine and router. Still works.

freshyill 10y ago

Five Thirty Eight! Today was a very long train ride without Twitter or the poll tracker.

Aoreias 10y ago
They stayed up all day, they just blocked rendering until they loaded some twitter assets, causing it to take 20-30 seconds to load until those assets timed out.

sergc 10y ago

CNN and TheGuardian are down as well

s_kilk 10y ago
Yup, I'm seeing TheGuardian down, from Edinburgh/Scotland.

michaelmwangi 10y ago

Pulling on AWS using github.com as origin failed had to add the ip to /etc/hosts.

nmd 10y ago

CircleCI is also having issues viewing and running tests, viewing from Amsterdam: https://status.circleci.com/ https://status.circleci.com/

crawdingle 10y ago

box.com seems to be down too.

expjpi 10y ago
also seeing that here in mountain view
szarecor 10y ago
Down in the midwest as well

royalharsh95 10y ago

gitter as well (https://twitter.com/gitchat https://twitter.com/gitchat)

momchenr 10y ago

Paypal

deleted 10y ago

[deleted]

lukevers 10y ago

Braintree is also experiencing these problems.

deleted 10y ago

[deleted]

Veen 10y ago

Looks like unsplash.com is down too

Veen 10y ago

Looks like unsplash.com is down too

utefan001 10y ago

Okta seems to be down.

mariuscoto 10y ago

http://okta.com http://okta.com is not working

jrod2121 10y ago

bigcommerce, volusion, new relic, optimizely, wistia, volusion, aweber, cnn, campaign monitor, all down for me. The biggest thing is seeing that ALL shopify stores are offline, so much $$$ being lost right now.

cafard 10y ago

python.org and cpan.org are down.

jssjr 10y ago

I'm a GitHub employee and want to let everyone know we're aware of the problems this incident is causing and are actively working to mitigate the impact. "A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769 https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well. [x-posted on https://news.ycombinator.com/item?id=12759697 https://news.ycombinator.com/item?id=12759697 as well]

cynosurexy 10y ago
how did you post on twitter which is down too?
- eric_the_read 10y ago
  Cached DNS response.
- SteveNuts 10y ago
  Twitter isn't down, just DNS resolution of twitter.com

lmpower 10y ago

just add the open DNS you are allset

lmpower 10y ago

use open DNS

prefect42 10y ago
Side Note: news (to me at least) Cisco had acquired Open DNS late Aug. Now I'll have to read all the doco to see what impact (if any) there is for personal usage.

celticpiping 10y ago

https://github.com/ https://github.com/ is up for me. North East

rotauserch 10y ago

Need round robin dns - geographically dispersed

rpearl 10y ago
That's not the part of the system that's under attack

constantskeptic 10y ago

If you add these sites ip addresses to your /etc/hosts file locally then you don't have to worry about this for the most part.

celticpiping 10y ago

https://github.com/ https://github.com/ is up for me.. NorthEast USA maybe I missed something

jrod2121 10y ago
looks like they are all back online now.

rotauserch 10y ago

Need to setup a round robin dns - geographically dispersed

shadowfacts 10y ago

All of the sites listed are back up for me, and still using Dyn DNS name servers.

celticpiping 10y ago
ditto for me

matiasb 10y ago

Mongo Cloud is having trouble too!

deleted 10y ago

[deleted]

timbl 10y ago

8.8.8.8 seems to know about github.com and github.io at least -- so I can work. (Maybe 8.8.8.8 built to resist censorship and therefor also attack?)

tarialys 10y ago

My Uber partner app crashed at 8am I was trying to complete a trip and it frozen my phone. It took about 5min for me to be able to enter back but it asked me for my SSN and permission to do a background check which is standard by Uber but I had already done so. Should I be concern that my personal data has been compromised? I contacted Uber but their idiots support people don't seem to have a clue and third fix is super basic like restart your phone, turn airplane mode or data on and off

gianarb 10y ago

http://downdetector.com/status/level3/map/ http://downdetector.com/status/level3/map/

FartyBumMcPoops 10y ago

i have highly available biscuits

FartyBumMcPoops 10y ago

i have highly available biscuits

danseasner 10y ago

Wiki leaks news held back all day for the most part the 20th. Clinton mafia attack. Had diplo,Ativan dinner in n y. Last night

danseasner 10y ago

Xclinton hack. News held back the 20th through 12 n 13th dump. Was a diplomatic dinner last night in n y c I live on east coast n e c

Avalanchee 10y ago

LinkedIn?

Avalanchee 10y ago

Linkedin doesn't seem to load page's contents as well.

danyork 10y ago

Many articles are now coming out about this outage this morning: http://www.techmeme.com/161021/p3#a161021p3 http://www.techmeme.com/161021/p3#a161021p3

dg123r 10y ago

And aws: 6:13 AM PDT [RESOLVED] Between 4:31 AM and 6:10 AM PDT, we experienced errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region. During the issue, customers may have experienced failures indicating "hostname unknown" or "unknown host exception" when attempting to resolve the hostnames for AWS services and EC2 instances. This issue has been resolved and the service is operating normally.

mszyndel 10y ago

For us here in Europe/Germany Paymill is also down!

yellowsir 10y ago

time to take a look at git-ssb p2p git+"github" https://git-ssb.celehner.com/%25n92DiQh7ietE%2BR%2BX%2FI403LQoyf2DtR3WQfCkDKlheQU%3D.sha256 https://git-ssb.celehner.com/%25n92DiQh7ietE%2BR%2BX%2FI403L...

mohsinr 10y ago

GitHub, Twitter down for me in Pakistan.

victorbojica 10y ago
down in Romania too
camoby 10y ago
Same here. Just now. In the UK.
- Enxo 10y ago
  Can access Twitter just fine from the UK. Playstation network is down though.

camoby 10y ago

In the UK here and Twitter and Github just went dark.

hujkikwhe 10y ago
One hour ago, Wikileaks twitted: > PHOTO: Heavily armed 'police' appear outside Ecuadorian Embassy in London where Julian Assange has political asylum (photo, Tuesday morning) Hopefully just a coincidence -- seeing as Twitter is a key communication conduit for Wikileaks.
Enxo 10y ago
And playstation network. I can't play online and it's Friday :(

y-love 10y ago

Github still down for me in New York.

jbaviat 10y ago

Issue came from dynect.net: $ dig +trace github.com ; <<>> DiG 9.8.3-P1 <<>> +trace github.com [...] ;; Received 488 bytes from 192.228.79.201#53(192.228.79.201) in 154 ms github.com. 172800 IN NS ns1.p16.dynect.net. github.com. 172800 IN NS ns3.p16.dynect.net. github.com. 172800 IN NS ns2.p16.dynect.net. github.com. 172800 IN NS ns4.p16.dynect.net. ;; Received 178 bytes from 192.42.93.30#53(192.42.93.30) in 54 ms ;; connection timed out; no servers could be reached

jorts 10y ago
They're investigating now: https://www.dynstatus.com/incidents/nlr4yrr162t8 https://www.dynstatus.com/incidents/nlr4yrr162t8
DonaldFoss 10y ago
As of 2016-10-21T09:20:19Z, it's doing better: -------------------------------------------------------------------- <Commodore Pet 16 Unix>:$ dig +trace github.com ; <<>> DiG 9.8.3-P1 <<>> +trace github.com ;; global options: +cmd . 17430 IN NS i.root-servers.net. . 17430 IN NS a.root-servers.net. . 17430 IN NS b.root-servers.net. . 17430 IN NS l.root-servers.net. . 17430 IN NS d.root-servers.net. . 17430 IN NS g.root-servers.net. . 17430 IN NS h.root-servers.net. . 17430 IN NS c.root-servers.net. . 17430 IN NS e.root-servers.net. . 17430 IN NS j.root-servers.net. . 17430 IN NS k.root-servers.net. . 17430 IN NS m.root-servers.net. . 17430 IN NS f.root-servers.net. ;; Received 228 bytes from 8.8.8.8#53(8.8.8.8) in 382 ms com. 172800 IN NS m.gtld-servers.net. com. 172800 IN NS k.gtld-servers.net. com. 172800 IN NS a.gtld-servers.net. com. 172800 IN NS b.gtld-servers.net. com. 172800 IN NS f.gtld-servers.net. com. 172800 IN NS j.gtld-servers.net. com. 172800 IN NS e.gtld-servers.net. com. 172800 IN NS c.gtld-servers.net. com. 172800 IN NS h.gtld-servers.net. com. 172800 IN NS l.gtld-servers.net. com. 172800 IN NS d.gtld-servers.net. com. 172800 IN NS g.gtld-servers.net. com. 172800 IN NS i.gtld-servers.net. ;; Received 488 bytes from 192.228.79.201#53(192.228.79.201) in 363 ms github.com. 172800 IN NS ns1.p16.dynect.net. github.com. 172800 IN NS ns3.p16.dynect.net. github.com. 172800 IN NS ns2.p16.dynect.net. github.com. 172800 IN NS ns4.p16.dynect.net. ;; Received 178 bytes from 192.42.93.30#53(192.42.93.30) in 134 ms github.com. 300 IN A 192.30.253.113 github.com. 86400 IN NS ns4.p16.dynect.net. github.com. 86400 IN NS ns3.p16.dynect.net. github.com. 86400 IN NS ns1.p16.dynect.net. github.com. 86400 IN NS ns2.p16.dynect.net. ;; Received 130 bytes from 2001:500:90:1::16#53(2001:500:90:1::16) in 25 ms I ended up sending all my developers a patch to their hosts file... Hopefully they'll roll it back shortly. --Donald

crgwbr 10y ago

DNS came back for a while and just started flaking again. $ nslookup github.com ;; Got SERVFAIL reply from 8.8.8.8, trying next server Server: 10.0.0.1 Address: 10.0.0.1#53 Non-authoritative answer: Name: github.com Address: 192.30.253.112 $ nslookup twitter.com ;; Got SERVFAIL reply from 8.8.8.8, trying next server ;; Got SERVFAIL reply from 8.8.4.4, trying next server Server: 10.0.0.1 Address: 10.0.0.1#53 Non-authoritative answer: Name: twitter.com Address: 199.16.156.102 Name: twitter.com Address: 199.16.156.230 Name: twitter.com Address: 199.16.156.198 Name: twitter.com Address: 199.16.156.70

ZeroActual 10y ago

Twitter/GitHub/Soundcloud/etc all just went down for me (UK), also The Verge and The Next Web.

tokm 10y ago

twitter down again now....

tokm 10y ago

Twitter down again...

MattieTK 10y ago

deck.news also down

BlackGuyCoding 10y ago

On the West Coast, Twitter/SoundCloud/GitHub just went down for me...

y-love 10y ago
Github is working (slowly) through UK VPN for me right now.
exolymph 10y ago
Also West Coast. Twitter and PayPal are both down.

chromaton 10y ago

PayPal isn't working for me. Could cost a lot of people a lot of money.

exolymph 10y ago
I just need to invoice someone, but I would be pissed if I used them to process ecommerce or SaaS payments.
apmoore 10y ago
I just noticed it when I tried to do a bank transfer. Ebay was also loading slowly for a while but I didn't think anything of it. Hopefully this is resolved soon.

oelmekki 10y ago

Anyone has any idea which hosts desktop spotify app is trying to reach? `netstat -punta | grep spotify` tells me they are on those IP:port : * 151.101.120.246:80 * 194.132.198.50:4070 Problem is, I don't know what hosts this is supposed to be, and `dig -x` is not very helpful...

ericholiveira10 10y ago

Anyone with issues on AWS in North Virginia?

onlydole 10y ago
It has been affecting a few offerings all day...specifically WorkMail for me.
paulddraper 10y ago
We've seen zero problems (standard stuff: EC2, S3, RDS, Redshift). I expected that with 1/3 of the internet down something would be broken.

crobertsbmw 10y ago

It's Hillary's fault...

ericholiveira10 10y ago

Anyone experiencing problems with servers on AWS North Virginia?

komali2 10y ago

Add 1800 contacts, ynab. Maybe not important to some people but it was how I realized something was up.

apmoore 10y ago

In CA - Spotify starting going in and out around 9:15a, then completely shut down at 9:20a (no offline playlists were available). As of 9:49a my offline playlists are back up, but nothing else. Twitter was working from my mobile about 15 min ago, but not on my desktop. Now both are down. Can't believe Twitter is down. On my mobile I'm getting notifications, but can't view them. The people who are sending updates are mostly on the East Coast and Canada, so maybe it's back up for them.

grav 10y ago
I doubt that your offline playlists were affected ...
- apmoore 10y ago
  Not sure why they were, but I tried 6 different offline playlists and each of them said "can't play the current song", then cited internet detection... which hadn't been an issue. Now (10:04a) all of my playlists are working again.
  - illumin8 10y ago
    Spotify has a terrible network connection algorithm that will sit there and spin forever trying to open the app if it thinks you're on the network but you're really not. Put your phone in airplane mode, then re-open Spotify, and magically, all of your offline playlists will play.
y-love 10y ago
On the East Coast, Twitter still down for me on mobile. All the Twitter media resources (twimg.com) are also not appearing.
- apmoore 10y ago
  Twitter is still not loading for me, but just checked Hootsuite and it is up and running now. Hootsuite was down for a while also.

lukeredpath 10y ago

Lots of sites down for me in the UK. I was using my ISPs DNS (Sky), switched to 8.8.8.8 and not much better.

maaaats 10y ago

I had issues with netflix as well. Netflix.com was resolving but lot of the resources from sub/other domains didn't load.

y-love 10y ago

Now even the DYN Status page is inaccessible from the Eastern US https://www.dynstatus.com/incidents/nlr4yrr162t8 https://www.dynstatus.com/incidents/nlr4yrr162t8 (Github still down from the Eastern US.)

Mingso 10y ago

It's 9:55am PST California and Twitter is down.

drcoopster 10y ago
Did you change your clocks early?

Mingso 10y ago

It's 9:56am PST California and Twitter is down.

jeff303 10y ago
Github and Twitter are both unreachable for me still. Chicago area.

deleted 10y ago

[deleted]

deleted 10y ago

[deleted]

Mingso 10y ago

It's 9:58am PST in California and Twitter is still down.

deleted 10y ago

[deleted]

Soulo_Guero 10y ago

10:02 PST in CA and Netflix (and, yes, still Twitter) is down.

Soulo_Guero 10y ago
Correction: Netflix is "half" loading.

deleted 10y ago

[deleted]

ewams 10y ago

Amazon is having an outage. Okta is also down because of this.

kyler887 10y ago

Runescape is also down

kyler887 10y ago

Runescape is also down

dfpr 10y ago

I managed to access twitter from Ecuador using Opera, activating the built-in VPN and setting it to Singapure.

mgalgs 10y ago

Same thing happened with hover last week... Same attacker perhaps? Maybe hover was just a test drive of their DNS DDoS machinery? https://twitter.com/HoverStatus/status/786296893013766144 https://twitter.com/HoverStatus/status/786296893013766144

Sami_Lehtinen 10y ago

Namecheap is also down.

moonlightsky 10y ago

https://ease.apperian.com https://ease.apperian.com is down over here

deleted 10y ago

[deleted]

paulddraper 10y ago

I thought DNS (particularly public) was basically immune to DDoS? If one DNS server is down, use the cached result or another server. DNS is some of the most distributable, cachable data I can imagine.

facorreia 10y ago
> DNS is some of the most distributable, cachable data I can imagine. Indeed. How can it be so broken? And, it seem that it would make sense to use the last known value if upstream is unavailable.
- nzp 10y ago
  That would be a horrible idea from a security standpoint. Although not easy by any means for an ordinary attacker, at least one exploit scenario is obvious.
pepon 10y ago
Seriously, please someone explain this. I thought DNS info is propagated among DNS servers, and it is cached on all of them... How can an attack to a single DNS company cause this outage??
- atom_enger 10y ago
  Explained above for ya
274below 10y ago
True, but DNS records have TTLs that expire, which forces you to reach back out to the authoritative server for the zone. A very decent number of records will have a TTL of 1hr, which means that a total DNS server outage lasting longer than an hour = 100% failure rate. You could always increase the TTL to a day (or whatever other arbitrary value), but that means that you'll be waiting upwards of a day for clients to see any updates to your DNS records. It's a tradeoff between different types of resiliency. This is a simple case of a DNS provider evidently not having their crap together when it comes to a DDoS of any decent size. There are plenty of other DNS providers out there who are working fine currently, this DDoS is limited to just the one. It's all of that company's clients that are impacted. I'm sure that DynETC will post something afterwards about how this was the largest DDoS they've ever encountered by many orders of magnitude and that there was no way for anyone to ever be prepared to take so much traffic... but at this point I think they're inept. edit: just came back up. The TTLs on a few common records explain why this was so obviously a problem... - www.reddit.com: 300 seconds (pointer to a fastly.net address, which is hosted via dynect) - prod.reddit.map.fastlylb.net: 30 seconds - api.twitter.com: 300 seconds - herokussl.com: 3600 seconds So, an outage of five minutes is enough to take out access to all of reddit and twitter. An outage of an hour is enough to take out heroku. Of course, these are best case scenarios: in reality, 50% of your users would lose access to reddit after 15 seconds (the fastly record), and so on.

nodesocket 10y ago

Pagerduty down, which is not very helpful.

wolf31o2 10y ago
Critical infrastructure was outsourced and now they've gone down! Yay, the "cloud" and all it's "glory" where some joker with an API thinks that magically means always available.

koryteg 10y ago

Rollbar!!!! noooooo!!!!!

Enzo40Dog 10y ago

Ticketfly.com. Damn I really wanted to buy die Antwoord tickets

MomoHo 10y ago

add moz.com

frjalex 10y ago

GitHub seems fine for me.

4GLyle32 10y ago

When will outage be restored??

4GLyle32 10y ago

When will this outage be repaired??

GilsonSoares 10y ago

www.indeed.com too

GilsonSoares 10y ago

www.indeed.com too.

fidla 10y ago

Still down. 2:17pm EST

fidla 10y ago

You can still access these sites using a VPN like Tunnel Bear, or a VPN browser like Opera

parkerproject 10y ago

npm is down as well

murmanda 10y ago

www.sonatype.com also down (from Germany)

AlexRazor 10y ago

I think that many of the IP addresses that are involved in this DDoS attack can be found in the BlackList database. As example https://cleantalk.org/blacklists https://cleantalk.org/blacklists There are 2 084 161 IP's blacklisted. https://cleantalk.org/help/fraud-prevention https://cleantalk.org/help/fraud-prevention

murmanda 10y ago

www.sonatype.com also down (at least from Germany)

AlexRazor 10y ago

I think that many of the IP addresses that are involved in this DDoS attack can be found in the BlackList databases. As example https://cleantalk.org/blacklists https://cleantalk.org/blacklists There are 2 084 161 IP's blacklisted.

Davezilla 10y ago

my.wpengine.com is also down. The public site is still up.

thesmart 10y ago

It seems incredibly misguided to plug an IP address into hosts from a forum post. Is there an official source from GitHub?

deleted 10y ago
[deleted]

temp1231231 10y ago

Easy way to get around it for Github: https://www.apptic.me/blog/adding-github-to-your-hosts-file.php https://www.apptic.me/blog/adding-github-to-your-hosts-file.... Same idea, just add the IPs to your hosts file.

deleted 10y ago

[deleted]

videoshooter 10y ago

Zoho.com, one of my email providers is down as well

koyao 10y ago

Amazon DNS must be getting loads of new customers today :)

mej10 10y ago
Even though their status page says everything is basically fine -- oh just a blip with API dns this morning no biggie -- they have had major problems today too. Like, can't connect to dynamodb, sqs, kinesis. Unfortunately they lie through their status page pretty regularly. I am not sure what it would take for them to admit they had a major problem.
- shuumai 10y ago
  Apparently us-east-1 was the only region that exclusively used dyndns (all other regions have failover strategies to other providers and that's why there wasn't downtime anywhere else but us-east and eu-west). They resolved the problem pretty quickly, but it drove our NOC insane...fortunately downtime for us was only about half an hour or so.
  - mej10 10y ago
    Yeah, thats about a half hour of down time for every AWS East based company... who knows how much lost revenue -- and yet they just say the service is green with a little info tag. If our service is down for any customer (much less an entire segment of customers) we at least publish that we had a partial outage.

E1Kenobi 10y ago

I cant get into the PlayStation Network either, I guess its time to go outside.

adhir8 10y ago

Also down: sie.sps.columbia.edu/enroll

stephb 10y ago

groupme.com is down in nyc... i should memorize friends' phone numbers...

realtorjohnf 10y ago

AthenaNet.athenahealth.com , one of Americas largest health care medical software providers is down, effecting 80,000 physicians and their patients.

michaelxia 10y ago

Does anyone have heroku's ip's?

jontodd 10y ago

It looks like Verizon customers continue to have issues resolving Dyn DNS. Wondering if Verizon has actually blocked Dyn requests

ted_dunning 10y ago
Dyn's status page still says they are working the problem. Why would Verizon block access to a downed service?
- jontodd 10y ago
  Not entirely sure. Perhaps some of the DDOS was flowing through to them but that's complete speculation. I mostly wanted to throw this out there for others who are trying to segment the impact of the issue.

dan0009 10y ago

roblox.com is offline too.

dan0009 10y ago

roblox.com is offline aswell.

hcngo 10y ago

Quora.com cannot be resolved either.

deleted 10y ago

[deleted]

stef25 10y ago

Github down, can't deploy code!

GilsonSoares 10y ago

Whatsapp too: # host -tns whatsapp.com whatsapp.com name server ns2.p13.dynect.net. whatsapp.com name server ns1.p13.dynect.net. whatsapp.com name server ns4.p13.dynect.net. whatsapp.com name server ns3.p13.dynect.net.

meyoulit 10y ago
Thanks

yachtycane 10y ago

Why did they have to ddos the EA servers too I just wanna play battlefield 1

meyoulit 10y ago

dot.

machty 10y ago

poop.bike is still up

semartin 10y ago

U.S DDOS to block Assange cable drop

apmoore 10y ago

Is anyone noticing significant delays visiting CNN, ebay, orbitz? It took 15 seconds for the Orbitz page to load (tried from Google Chrome and Safari). It seems like outside of the core group of sites effected (GitHub, Twitter, PayPal), other major websites are having serious delays loading..

webandtech 10y ago

Anyone with an app affected by this because you use Heroku's SSL endpoint can fix this by switching to SNI. Unfortunately Heroku's API is also affected. There is a workaround... just edit your hosts file to point to the proper ip addresses temporarily. add to /etc/hosts 23.21.149.112 api.heroku.com 107.21.99.123 ssl-doctor.heroku.com Then use Heroku CLI to switch your app to the new SNI endpoint heroku certs:add --type sni server.crt server.key -a YOUR_APP That will spit out a new host name that you can point your DNS to and still be online.

cstavish 10y ago
Thanks a million, from me and the people who invested in my startup. edit: apparently SNI isn't supported on legacy dynos.

jemini 10y ago

We are seeing inconsistent resolutions on fppaco.org. sometimes it resolves to a 4.x address and sometimes a 50.x address. According to whois.net, their DNS servers are DYNect

photogrammetry 10y ago

Twitter, Github, Quora, Soundcloud, Etsy, PayPal, Heroku, The Guardian, WhatsApp and numerous others still down as of 1:00 pm PT. Amazon and Spotify are still up.

Enxo 10y ago
Playstation network down last 4 hours. This is in the UK don't know about the rest of the world. Status: https://status.playstation.com/en-gb/ https://status.playstation.com/en-gb/

clavallee 10y ago

Twitter still down Ohio

clavallee 10y ago

Twitter still down - Ohio

lawlorslaw 10y ago

Change your DNS settings and some of these websites will now work. http://www.dailydot.com/layer8/how-to-change-dns-settings-mac-windows/ http://www.dailydot.com/layer8/how-to-change-dns-settings-ma...

Andres102116 10y ago

I have a shop on etsy. It is functional now.

Andres102116 10y ago

I have a shop on etsy, and it is functional now.

Andres102116 10y ago

I have a shop on etsy.com. It is functional at this time (1418 PDT).

steppnav 10y ago

Remind me to populate my /etc/hosts file after this blows over.

aledalgrande 10y ago

hosts file settings: - to deploy to Heroku 174.129.22.147 git.heroku.com - to enable Twitter from Web 104.244.42.65 twitter.com 93.184.220.70 pbs.twimg.com 104.244.43.7 o.twimg.com - to access Github (as another poster wrote): 192.30.253.113 github.com 151.101.4.133 assets-cdn.github.com You can find any other IPs you need on sites like http://www.hcidata.info/host2ip.cgi http://www.hcidata.info/host2ip.cgi Just remember to rollback your hosts file after the outage finishes.

matobago 10y ago

This is not a dyn error, this is all the companies failed to prevent this to happen, is an old practice having different NS Servers for your domain, but it can prevent this to happen.

deleted 10y ago

[deleted]

jasonfb 10y ago

It appears that people (domain name administrators) are switching away from Dyn as their DNS provider, as per https://status.heroku.com/incidents/965 https://status.heroku.com/incidents/965 (issued 19:21 UTC). Does this mean that other DNS providers (Amazon Route 53, CDNetworks, CloudFlare, DNSimple, easyDNS, Google DNS, Verisign, OpenDNS) could potentially become targets? -Jason (5:13 PM EDT; 21:13 UTC)

jasonfb 10y ago

confirmed three (3) coordinated attacks on Dyn's infrastructure today: 11:10 UTC, 13:52 GTM, and the third time is unknown -- Dyn has not specified the time of the 3rd attack today on their dynstatus.com page (likely because the whole thing can be considered ongoing). Can anyone speak to when the 3rd attack began today?

2048-Bit 10y ago

Dyn can't figure out how to fix it because they can't get to stack overflow! :D

clavallee 10y ago

The outage is attributed to routing 576 Autonomous systems and 3431 prefixes.

latentsurfer 10y ago

The DDOS also hit small business websites in North America. I know atleast three companies who experienced the DDOS attacks. The IPs that were port scanning prior to the attack are 173.254.216.66, 128.52.128.105 (MIT Edu), 172.98.67.39. I hope there is some action against these hackers.

DyslexicAtheist 10y ago

you can watch BGP routes changing (as we speak) here https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16 https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16

rcummings 10y ago

Sounds like IETF needs to update the DNS standards with 'intelligent' features, so that we can have some consistency.

Bentsie 10y ago

More likely this idiot is doing it. Look at the location spots. Ben Callis. He's a little assbot programmer that likes hacking bank accounts and stealing people's shit. Runs around in a Roman Catholic Church. Pure Hoodlum and ugly disgusting rat. He's stealing networks and fuck-flashing it probably prior to the elections. If you want to know who's been hacking away at your banks and your stuff disappearing it's this asshole. He smiles and pretends but he's a career criminal. You see this maggot report him to FBI. Absolute Fiend and a pathetic piece of rotten crap. Dangerous too got ahold of some weapons starts acting up. BenCallis-CohhCarnage is a FIEND. He hassles and hurts women and children is a human-trafficker and disgusting piece of morbid crap.

deleted 10y ago

[deleted]