10 ms·
Issues with upstream DNS provider
- tortilla 4y agoYes...all my apps. https://status.heroku.com https://status.heroku.com
- terracatta 4y agoI feel silly for posting now. Thanks for sharing.
- pardner 4y agoYes, cant even access https://dashboard-next.heroku.com/ https://dashboard-next.heroku.com/ so the problem seems broader than what they describe on their status page which seems to imply only issues related to updating DNS settings.
- mr90210 4y agoHeroku Herokuing.
- deleted 4y ago[deleted]
- hamhamed 4y agoImmediately came to HN, best status server
- daveguy 4y agoSame. If I am having a problem with a site but nothing else, next step is HN.
- reaperducer 4y agoI wonder if some clever person could build a routine to grep HN headlines and build a multi-service outage dashboard from what suddenly gets popular.
- mbesto 4y agoI'm building a service to create a forum, a la HN style, for service incidents. If you're interested in getting early access - get in touch here: hello@awareops.com
- btown 4y agoOur entire suite of sites is down, both ones proxied via Cloudflare and ones not proxied.
- soperj 4y agoHaven't made any updates, or changes in the last few days, and still won't show my site. So don't think it has to do with new apps or updates.
- jrochkind1 4y agoSame. The heroku incident report says "increased failure of insert/delete/changes to DNS." Haven't changed DNS in months. Site is down. Heroku is looking increasingly like nobody is minding the store these days. I still know of no competitor which can match the DX when it's working, but what good does that do you if it breaks all the time. We're still waiting on them fixing our ability to restore from backups without manual hack -- which is effecting many customers, but which they don't even have an active published incident on.
- mijoharas 4y ago+1 prod outage here too. No changes to DNS on our part.
- rajamaka 4y agoProduction completely inaccessible, on the plus side I might finally now get backing to migrate away from the Heroku cesspit.
- tortilla 4y agoYep...it's time (me too).
- bobsingor 4y agoYes, i'm also done with Heroku. What alternative do you recommend?
- tortilla 4y agoI'm looking at fly.io and render.com.
- camjohnson26 4y agoWe’re using Porter to simplify the migration to AWS
- fourstar 4y agoI moved from Heroku to Render. Can't recommend them enough. Smaller operation, of course so keep that in mind if you need support, but for a hobbyist like myself, I'm running 2 postgres dbs, 1 redis instance, and 4 web services (2 websites).
- anurag 4y ago(Render founder) thanks for the rec and feel free to get in touch with me anytime (email in profile). For those who need them, we offer enterprise support plans with response and uptime SLAs.
- dogweather 4y agoI just completed the move of my web app to render, and for now connecting back to Heroku for the postgres instance. I'll move that over eventually.
- deleted 4y ago[deleted]
- booi 4y agoUpstream DNS provider? I would have thought Heroku would run their own DNS servers? Is it just turtles all the way down?
- md224 4y agoTIL Substack runs on Heroku.
- antod 4y agoThey might just have the same upstream DNS provider?
- zinekeller 4y agoSeems to be (https://substack.statuspage.io/incidents/7ct4mgnd5dhn https://substack.statuspage.io/incidents/7ct4mgnd5dhn)
- jeffbee 4y agoOther Cloudflare-backed services like Observable are offline right now. Is it a Cloudflare outage?
- zinekeller 4y agoMore likely that Cloudflare is unable to access downed servers.
- jeffbee 4y agoMaybe. I'm just trying to figure out what's common between Steam, Observable, and Heroku, and Cloudflare itself which are all down according to downdetector.
- zinekeller 4y agoThat's blaming the wrong entity - again. No-one seems to read error pages nowadays, not shocking. Also, Steam never used CF services, and user reports about CF are that - they looked at their favorite Substack (https://substack.com https://substack.com) and just seeing a CF page they assumed it's CF's fault.
- mr90210 4y agohttps://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/
- a13n 4y agoMissive, Substack, and Steam appear to be down because of this
- samcheng 4y ago> We will be providing updates on a 1 hour cadence. Ouch. Not a great look, Heroku!
- swrobel 4y agolast update at this point: over an hour ago
- mscarborough 4y agoAnd no update after 2 hours. How/why did Salesforce run this platform into the ground?
- kennysmoothx 4y agoProduction outage here as well. Is this isolated to Heroku?
- ab-dm 4y agoComplete outage on all of my sites. I have a migration plan in place but it's been put on hold to launch a new Product.
- jarym 4y agoAnd the new product might have to be put on hold until Heroku is back… catch-22 - you’ll never migrate away
- ab-dm 4y agoWell, the new product is hosted on AWS, so it's much more stable, and we've made our first lot of pre-sales so we're on a tight deadline. Not sure why you're being so negative for no reason.
- bobby_growth 4y agoHeroku. DansGame.
- bfirsh 4y agoLooks like just DNS for the CNAME is broken. $ dig @1.1.1.1 stark-wisteria-rnbgkawldfk6gq7m8308ytts.herokudns.com A ... ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; OPT=15: 00 09 6e 6f 20 53 45 50 20 6d 61 74 63 68 69 6e 67 20 74 68 65 20 44 53 20 66 6f 75 6e 64 20 66 6f 72 20 68 65 72 6f 6b 75 64 6e 73 2e 63 6f 6d 2e ("..no SEP matching the DS found for herokudns.com.") ;; QUESTION SECTION: ;stark-wisteria-rnbgkawldfk6gq7m8308ytts.herokudns.com. IN A I wonder if there is any way to get out an IP address of the Heroku router we were assigned to that we can use in place of the CNAME. Might be in the logs somewhere, or in Cloudflare somewhere?
- john37386 4y agono SEP matching is a dnssec error. The DS key in the .com. zone file are there but no longer available in herokudns.com. Hopefully someone has a backup of those keys. If not, I think they have to contact .com. to replace the keys. It can take several hours to come back.
- mcint 4y agoSeeing issues with hackerweb.app and substack.com too. They don't share upstream DNS (and I'm not sure heroku's homepage has the same DNS provider as customer domains). NS matches SOA for each of these domains. SOA heroku.com. 1h00m00s "dns1.p04.nsone.net." "hostmaster.nsone.net." SOA hackerweb.app. 1h00m00s "olga.ns.cloudflare.com." "dns.cloudflare.com." SOA substack.com. 1h00m00s "ali.ns.cloudflare.com." "dns.cloudflare.com."
- ezekg 4y agoAll services are completely down for my business. Looks like they nuked their DNS.
- zinekeller 4y agoIt seems that NS1 (Heroku's provider) was having problems, but according to them it has been fixed: https://ns1status.com/#!/incident/365716 https://ns1status.com/#!/incident/365716
- deleted 4y ago[deleted]
- rsweeney21 4y agoWe just deployed a workaround that brought our site back up. 1. Find your DNS Target in heroku. It should end with .herokudns.com 2. Lookup the historical DNS record to get the IP addresses. You can find historical DNS records here: https://securitytrails.com/dns-trails https://securitytrails.com/dns-trails 3. Replace your CNAME record in your DNS provider with A records that point to the IP addresses you just found. Your site should come back up shortly. We plan to revert back to CNAME records once Heroku gets their DNS issues sorted.
- rajamaka 4y agoNo historical DNS records on that site for us unfortunately, any other possible sites to check?
- cyral 4y agoTry this command: nslookup DOMAIN 212.230.255.1 That is a random public DNS server I found on a DNS checker site that works (as it seems to be down from all the common ones like 8.8.8.8, 1.1.1.1, etc). DOMAIN can be your actual domain or the long heroku alias.
- iancarroll 4y agoThanks, for some reason this worked!
- rajamaka 4y agoYou bloody legend, thanks!
- yangcheng 4y agoyou saved my users! thanks so much
- danra 4y agoThis one worked for me: https://dnshistory.org/ https://dnshistory.org/
- zzh8829 4y agoLooks like Heroku uses NS1 as the upstream DNS provider. you can find the information like this dig NS @1.1.1.1 test.herokuapp.com -> fail dig @1.1.1.1 test.herokuapp.com -> fail dig NS @dns1.p03.nsone.net test.herokuapp.com -> works dig @dns1.p03.nsone.net test.herokuapp.com -> works So my conclusion is that NS1 is having issue responding DNS queries from other DNS servers. Interestingly, there is no public information on heroku being dependent on NS1 or any current outages from NS1 status page.
- ab-dm 4y agoHeroku dashboard was at least working before, but now trying to view details of an app it just hangs
- SamHoustonCM 4y agoWhy does Heroku have so many uptime issues? Seems to be happening every few months. Last week there was downtime, now again... Is Salesforce committed to Heroku?
- ab-dm 4y agoHeroku has brought our site down 5 times in the last 6 months. It's infuriating, but I only have myself to blame for not migrating.
- mhale 4y agoThis does not appear to be a total outage. I cannot reach any of our sites, and Pingdom also reports we are down, however, I can see normal looking traffic reaching our servers (via heroku logs --tail). In addition, members of our team are reporting via Slack that some can reach our Heroku-hosted sites, others cannot. It seems to be ISP-related. Two people within 1 block of each other on different ISPs see different results. We proxy some services through Cloudflare to gain IPv6 support, and all of those are down, which suggests the Cloudflare -> Heroku network route is broken.
- deleted 4y ago[deleted]
- exitnode_ttl0 4y agoI am now seeing one of my services coming back online. For this service I have not replaced the CNAME. Anyone else seeing some service restoration as well?
- seoneehills 4y agoSame boat for me. How long do you think you'll wait to go back to CNAME's?
- exitnode_ttl0 4y agoI went back to the CNAME now and everything is up and running. Fingers crossed that Heroku reports to have the issues resolved.
- hluska 4y agoOur service started coming back up around twenty minutes ago too.
- mijoharas 4y agoOur site just came back a few minutes ago, and we're showing up in 8.8.8.8 again, so it seems that heroku have sorted out their dns issues.
- klardotsh 4y ago17:57 PDT, at least a few of our sites are starting to wake up. Decidedly not all of them yet. sigh
- deleted 4y ago[deleted]
- nevster 4y agoSeems to be resolved now
- iuliu 4y agoIt seems that their service is back up
- nightpool 4y agoWeirdly, dig +trace works fine, but public resolvers like Google and Cloudflare refuse to return the DNS records. This has to be a DNSSEC issue, right? paging tptacek :p
- ab-dm 4y agoI've swapped the A records back to the CNAME and all is looking good. The Heroku Dashboard still isn't loading app details though.
- rafaparadela 4y agoIt worked for us too. Thanks. Note: If you used to have a CNAME record for yourdomain.com to www.yourdomain.com, then you have to add two A records per each IP (one whose name is yourdomain.com pointing to the IP, and another whose name is www pointing to the same IP)
- deleted 4y ago[deleted]
- sp24oct 4y ago
- johnklos 4y agoWow. Imagine being so out of touch that you have to have another entity run your DNS services. Email I get, because there has been a hard push for decades to force everyone on to big providers, but DNS can literally be run by anyone, anywhere. Did the primary servers push bad data, making the secondary / tertiary ones break, too? If not, why not extend the cache lifetime and run off of them until the primary are fixed? Sigh. This is rather ridiculous, and is rather embarrassing for Heroku.