7 ms·
We just deployed a workaround that brought our site back up. 1. Find your DNS Target in heroku. It should end with .herokudns.com 2. Lookup the historical DNS
by rsweeney21 4y ago
We just deployed a workaround that brought our site back up.
1. Find your DNS Target in heroku. It should end with .herokudns.com
2. Lookup the historical DNS record to get the IP addresses. You can find historical DNS records here: https://securitytrails.com/dns-trails https://securitytrails.com/dns-trails
3. Replace your CNAME record in your DNS provider with A records that point to the IP addresses you just found.
Your site should come back up shortly. We plan to revert back to CNAME records once Heroku gets their DNS issues sorted.
- rajamaka 4y agoNo historical DNS records on that site for us unfortunately, any other possible sites to check?
- cyral 4y agoTry this command: nslookup DOMAIN 212.230.255.1 That is a random public DNS server I found on a DNS checker site that works (as it seems to be down from all the common ones like 8.8.8.8, 1.1.1.1, etc). DOMAIN can be your actual domain or the long heroku alias.
- iancarroll 4y agoThanks, for some reason this worked!
- rajamaka 4y agoYou bloody legend, thanks!
- yangcheng 4y agoyou saved my users! thanks so much
- danra 4y agoThis one worked for me: https://dnshistory.org/ https://dnshistory.org/
- alexpoof 4y agoUsed this site https://www.whatsmydns.net https://www.whatsmydns.net, and it worked. Unfortunately the Heroku issue lost us a few thousand $ alongside a few customers.
- bfirsh 4y agoThis got us back online. Thank you so much.
- theunixbeard 4y agoPresumably if your site is behind Cloudflare then this strategy won't work, right? Since the IP addresses that Security Trails sees are just of Cloudflare rather than your actual Heroku IPs...? Or is it possible in the Cloudflare dashboard there is somewhere to see your Heroku server's IP address?
- bfirsh 4y agoYou need to put in the herokudns.com address that the CNAME is pointing at – e.g. stark-wisteria-rnbgkawldfk6gq7m8308ytts.herokudns.com in our case.
- iuliu 4y agoFor me the securitytrails.com website just crashes. I put my DNS target: "stark-horse-mrp4jeowu9yvwpnnma32x6hd.herokudns.com", clicked "Run Check" and it seems to redirect to a failed (no CSS) webpage. Anyone else experiencing this? EDIT: I MANAGED to make it work with this: https://www.nslookup.io/ https://www.nslookup.io/ Make sure to go through all the tabs at the top (Cloudflare DNS, Google DNS) - for me they were all "no A records found". Only "Authoritative" gave me 3 A records which I successfully managed to use.
- jonathanreinink 4y agoYeah securitytrails.com was working for me, but went down about 10 mins ago.
- eappleby 4y agoThat seems to happen when you put in a DNS target that doesn't have any records in SecurityTrails. In that case, it is best to use nslookup
- jplehmann 4y agoThanks so much, it worked for us.
- ladybro 4y ago
- deleted 4y ago[deleted]
- ______ 4y agoBless you! Thank you for taking the time out of your day to help us internet people :)
- bobsingor 4y agoWow this worked! Your amazing! Thank you so much
- exitnode_ttl0 4y agoAnybody know what the risks are for doing this? We have not one but 13 heroku domains running where we might do this. What are the chances that their current work on the DNS issues might replace the underlying IP-addresses and hence we are making it harder on ourselves?
- ab-dm 4y agoI thought about this, and it can't really get any worse than 'total outage', so I thought it was worth taking a shot. It can't go any more down. It worked for me, so once Heroku has sorted their s*t out and their DNS system is back up it's just a matter of deleting the A records and replacing it with the CName. Worst case scenario is the underlying Amazon IP Addresses change and it goes back down.
- Trasmatta 4y agoWouldn't the worst case be the underlying IP address changes, and your site now points to some other service? Since Heroku doesn't offer static IPs, I don't think.
- cyral 4y agoHeroku seems to use the Host header on the HTTP request to determine what customer the request is sent to, so I don't think it can end up somewhere it shouldn't be unless the IP leaves control of Heroku entirely and is picked up by another AWS customer.
- rsweeney21 4y agoHeroku uses the host name to route requests. So you wouldn't serve another site, it would just give you an error that your site wasn't found or something.
- exitnode_ttl0 4y agoThere is one more problem I can think of (but yes you're right, it can't get worse than 'total outage'): Heroku might detect that the CNAME is missing and will deactivate some stuff from their automatic domain management / TLS certificate issuing.
- kennysmoothx 4y agoThis worked for us, thank you. And for those using Cloudflare, this method works.
- Maarius 4y agoThanks so much, worked for us as well. I created A records for all three of the listed IP addresses but it started working for the first one already.
- tdehnke 4y agoWoot! Thank you - that seems to work, just used the first IP that came up on ST (there were 4 listed).
- ab-dm 4y agoOh my god, thank you so much. You're a lifesaver. For those on Cloudflare, this will still work. I just deleted the existing CNAME, added the 4 IP addresses as seperate A records and it came back up instantly.
- ladybro 4y agoWorked for me as well! Thank you!
- frameworkzach 4y agoThis worked for us too - thank you! Edit: It also works for subdomains.
- samcheng 4y agoHeroku - if you're listening. Please do NOT migrate IPs / load balancers here for a little while. You might also do some work to detect those of us who have bypassed your DNS, and give us some nudges individually to move back to your DNS when the time is right. You just caused a very painful outage for your entire customer base, and a number of us used this hack to get back online. If you don't recognize this, and cause ANOTHER outage (the third in the past few weeks...) we're going to flee even faster to more stable platforms.
- craigkerstiens 4y agoThose IPs tend to stay pretty stable and are shared across the fleet. While you should change them back once things are stable, it's reasonably likely these won't change out from under you in a few hrs.
- kwentakill 4y agoWorking well !
- bjoveski 4y agoThank you a lot!!
- dogweather 4y agoNice. We got our test site online by migrating just the dyno to render.com. (And connecting back to Heroku for the database.) We might just switch our DNS there and call it a permanent move.
- csjh 4y agoprobably worth staying at render considering how many issues heroku's had recently
- seoneehills 4y agoI think I love you. We're back online.
- dmbrooking 4y agoAnyone know how to make this work for a subdomain?
- nbarbettini 4y agoDo you have a CNAME record pointing to a herokudns.com domain? Replace it with an A record that points to the IP instead. You can find the IP by going to https://securitytrails.com/domain/<something-1234.herokudns.com>/dns https://securitytrails.com/domain/<something-1234.herokudns....
- nightpool 4y agoHi all, deleting a root CNAME record and adding a new A record on Cloudflare was too risky for us, with its potential to impact external DNS responses, so we took the following steps instead: 1. Create a new temporary subdomain in the Cloudflare DNS panel, e.g. "temp-heroku-resolve.your-domain.com", with IPs from the responses you get by querying Heroku's upstream provider directly (`dig @1.1.1.1 +trace your-app.herokuapp.com`). Make them unproxied (grey cloud) A records with a 30s TTL. 2. Change your root CNAME to point at "temp-heroku-resolve.your-domain.com" To undo, just reverse these steps and point your root CNAME back at Heroku.
- exitnode_ttl0 4y agoThis is essentially doing the same thing right? What is the advantage of this method? Less DNS changes? (not trying to be snarky, trying to understand)
- anayar 4y agoNah bc you're not swapping from CNAME > A records, which could have other random networking related impacts, especially if you're running a public API.
- nightpool 4y agoThe advantage is that you can do the CNAME swap atomically in the Cloudflare interface, instead of deleting a CNAME record and adding a new A record, which would lead to a few seconds when your DNS is entirely removed. Those few seconds may not sound like a lot, but they could be cached by Cloudflare or other external DNS resolvers, trigger "clean-up" functions within Cloudflare to remove resources associated with those domains, etc. In our case, we have substantial logic running on the edge in Cloudflare Workers, as well as lots of other Cloudflare configuration options, and I don't even want to know what it could potentially do to our zone to remove the root DNS record that everything's tied to.
- exitnode_ttl0 4y agoGood points, thanks!
- deleted 4y ago[deleted]
- anayar 4y agoWorked for us on Cloudflare using the .dns Heroku target!
- jamesr101 4y agoCan someone clarify what to do if your app doesn't use a custom domain, and running `heroku domains -a` just returns `my_app.herokuapp.com`, and then entering that into https://securitytrails.com/dns-trails https://securitytrails.com/dns-trails returns no results?
- iuliu 4y agoDo you use Cloudflare?
- jamesr101 4y agoNo we just run our API on heroku and use no custom domain names. For some of us requests are working as normal, but for others every request is failing.
- iuliu 4y agoWhat error are you getting? Because for us it seemed that the Cloudflare <-> HerokuDNS link was down. But even then I was able to access my app from the dashboard (like https://my-app.herokuapp.com/ https://my-app.herokuapp.com/)
- johnnymetz 4y agoThis worked for us. We're using Heroku + Cloudflare.
- 0xChain 4y agoThis saved our bacon and got me a pat on the back from our VP. Thanks!
- Justsignedup 4y agoYou and your team are brilliant. Thank you for this post! Much love.
- fusionflo 4y agoThank you! You saved our ass.
- rafaparadela 4y agoIt worked for us too. Thanks. Note: If you used to have a CNAME record for yourdomain.com to www.yourdomain.com, then you have to add two A records per each IP (one whose name is yourdomain.com pointing to the IP, and another whose name is www pointing to the same IP)
- tuxninja 4y agoYes, this worked for me...I was trying to do this right away, but I was missing a way to find historical DNS, thank you!!
- muvb00 4y agoWorked... thanks a bunch. Does anyone know if it's safe to go back to cname now or should I leave till the A record till the morning? Would prefer to get some sleep.
- newalexandria 4y agoHeroku reports the outage is over - but I wasn't able to resolve our Heroku-based domain name until recently today. Should be able to go back to CNAME now. Regardless, recheck yours