14 ms·
Tell HN: HN Moved from M5 to AWS
After many years of remaining static, HN's IP address changed.^1
Old: 209.216.230.240
New: 50.112.136.166
Perhaps this is temporary.
Little known fact: HN is also available through Cloudflare. Unlike CF, AWS does not support TLS1.3.^2 This is not working while HN uses the AWS IP.
1. Years ago someone on HN tried to argue with me that IP addresses will never stay the same for very long. I used HN as an example of an address that does not change very often. I have been waiting for years. I collect historical DNS data. When I remind HN readers that most site addresses are more static than dynamic, I am basing that statement on evidence i have collected.
2. Across the board, so to speak. Every CF-hosted site I have encountered supports TLS1.3. Not true for AWS. Many (most?^3) only offer TLS1.2.
3. Perhaps a survey is in order.
- metadat 4y agoI was surprised they only had one backup server, especially given the competitive price of rackmount hardware these days. More replicas needed. Although this was a fun exercise to learn how lost I feel without HN. Damn.
- dang 4y agoOur thinking was: (1) keep a hot standby to fail over to when we need it—that keeps downtime to seconds in routine cases (like pre-planned maintenance) and minutes or an hour in most failure cases—for example, when our primary server died last night, HN was down for about an hour while we brought up the standby; and (2) In the unlikely event that both the primary and standby servers fail at the same time, be able to bring up a fresh server from backup within hours, not days. The latter case is what happened today, and in the end we were down for just under 8 hours. (Assuming we don't sink back into the pit of hell overnight.) Assuming things don't fail again in the next day or two, since we still have a lot to take care of (fingers crossed—definitely not gloating), I feel like this was pretty reasonable. We don't have a lot of dev or ops resources—few people work on HN, and only me full-time these days. The more complex one's replica architecture, the higher the maintenance costs. The simplicity of our setup has served us well in the 9 years that we've been running it, and I feel like the tradeoff of "several hours downtime once a decade" is worth it if you draw one of those risk/cost managerial whiteboard things.
- metadat 4y ago@Dang, the state of affairs around here is already more than reasonable. In fact, it's incredible HN is almost never down, and the occasional 8-24 hour interruption every 5-7 years is actually a Good Thing for HN whores (like me) to reflect on how insanely much we are hooked on and love this stupid time sink technomancer site. Cheers, you are the true and literal soul of the machine embodying the best spirit of the oftentimes beautiful thing that is Post-Paul-Graham HackerNews. Please just promise to never die.
- toast0 4y agoIt might be worth considering a way to get a we're working on it notice up quickly. (HN status on twitter worked, but it's kind of nicer when something loads at the main address), but an 8 hour outage once a decade for something that's not really critical is pretty good; no need to increase complexity, although try to get some storage diversity for the future, now that you've learned about that.
- tannhaeuser 4y agoI'm guessing that several-hours-outage figure from restoring a full backup could be reduced by restoring the most up-to-date discussions first, then gradually restoring older discussions, then re-indexing for full-text search, all the while running in degraded mode but still having a front page. But tbh I'm just glad it wasn't an attack against free speech in tough times.
- andreareina 4y agoOne backup server is apparently sufficient, given the primary held up for 4.5 years. The issue was correlation between the primary and the spare which wouldn't have been solved by more replicas anyway.
- furyofantares 4y agoThat argument really got to you, did it?
- undoware 4y agoMy first thought as well -- but it was phrased as, "of course there is some specific person here who, due to a long-running disagreement over facts, has been passively gathering relevant data over a period of years." It's sort of like Rule 34, but for HN. "There is data of it"
- furyofantares 4y agoI'm certainly pleased by the result
- midislack 4y agoYou could run this site on a Pentium 3. Why even bother with virtual hosting?
- ethanwillis 4y agoIs this why there was downtime?
- sillysaurusx 4y agoNah, the root cause was a double disk failure. Their primary server’s disk failed, and then their failover server failed. https://twitter.com/hnstatus/status/1545409429113229312?s=21&t=IeCEvHjnnan8Bh-NH7ZVLQ https://twitter.com/hnstatus/status/1545409429113229312?s=21...
- sillysaurusx 4y agoI was wondering how they got the server up so quickly when their actual hardware failed. Dropping a bunch of cash on a big virtual server while they rebuild makes sense. It’s also a reminder that beefy virtual servers are pretty darn beefy nowadays. I wonder which tier they went with.
- jayd16 4y agoIt was down all morning, I half expected a post about a trip to Best Buy.
- sillysaurusx 4y agoYeah. Bare metal failing is approximately the worst case scenario behind data loss. Being down all morning is an impressive recovery time, because they had to provision an EC2 server and transfer all data to it.
- pmoriarty 4y ago> Being down all morning is an impressive recovery time, because they had to provision an EC2 server and transfer all data to it It looks like the data was already in S3: [1] The recovery probably would have been a lot faster had they had a fully provisioned EC2 image standing by... which I'd bet they will from now on. [1] - https://news.ycombinator.com/item?id=32032316 https://news.ycombinator.com/item?id=32032316
- fomine3 4y agoFor who unaware: M5 hosting https://www.m5hosting.com/ https://www.m5hosting.com/ It's not EC2 M5 instance.
- SomeBoolshit 4y agoIt wouldn't be a move to AWS otherwise.
- infogulch 4y agoIt's fair to clarify. To someone engrossed with AWS infra the phrase "moved from M5 to AWS" could be puzzling.
- benatkin 4y agoIt also isn't a move to an unreleased Apple processor. There's https://en.wikipedia.org/wiki/Apple_A5 https://en.wikipedia.org/wiki/Apple_A5 and https://en.wikipedia.org/wiki/Apple_M1 https://en.wikipedia.org/wiki/Apple_M1 https://en.wikipedia.org/wiki/Apple_M2 https://en.wikipedia.org/wiki/Apple_M2
- smn1234 4y agothanks for this! Interesting feature comparisons: https://www.m5hosting.com/iaas-cloud/ https://www.m5hosting.com/iaas-cloud/ so it's a private cloud, not m5 managed cloud environments across multi public cloud providers
- solardev 4y agoI'm surprised HN was on bare-metal to begin with. Is there a writeup about their infrastructure somewhere?
- sixothree 4y agoI vaguely remember hearing HN is a single file script.
- vorticalbox 4y agoHN is wrote in ARC I believe.
- deleted 4y ago[deleted]
- colechristensen 4y agoI don’t remember the exact details but basically one step above “running on that one computer in the closet”.
- efortis 4y agohttps://news.ycombinator.com/item?id=28479595 https://news.ycombinator.com/item?id=28479595
- matheussampaio 4y agoHave they disclosure how much traffic HN receives daily?
- booleandilemma 4y agohttps://news.ycombinator.com/item?id=16076041 https://news.ycombinator.com/item?id=16076041 4 million requests per day in 2018.
- herpderperator 4y ago6 million per day as of 10 months ago: https://news.ycombinator.com/item?id=28479595 https://news.ycombinator.com/item?id=28479595
- dralley 4y agoI hope this is only temporary. Where else will we discuss AWS outages when AWS goes down? Not even a joke.
- cherioo 4y agoThe IP appears to be us-west-2, so we will still be able to discuss us-east-1 outages alright!
- ignoramous 4y agobut... the s3 buckets are in us-east-1, and postgres in ap-southeast-2. More regions better than one, for maximum impact with minimum effort.
- pojzon 4y agoIs there at least one case when whole region went down ;)) ?
- withinboredom 4y agoAt least once, in 2012? I remember because just two days before we had fully switched over to AWS and hadn’t done multi-region yet. We got our first downtime in 10 years and there was nothing we could do, unlike when we had the servers in a colo. We were at the mercy of Amazon. After that, we moved everything back to real physical servers.
- samspenc 4y agoPer the spreadsheet here https://awsmaniac.com/aws-outages/ https://awsmaniac.com/aws-outages/ : There seem to have been multiple "full" outages in 2011-12 in AWS' us-east-1 region, which, granted, is the oldest AWS region and likely has a bunch of legacy stuff. By "full" outages I mean that a few core services fell over but the entire region become inaccessible due to those core failures.
- 4y ago
- wging 4y ago> Unlike CF, AWS does not support TLS1.3. This is not working while HN uses the AWS IP. This seemed implausible so I looked into it, and it's wrong as stated (at best, it needs to be made more precise to capture what you intended). First, you've mentioned Cloudflare, but the equivalent AWS product (CloudFront) does support TLS 1.3 (https://aws.amazon.com/about-aws/whats-new/2020/09/cloudfront-tlsv1-3-support/ https://aws.amazon.com/about-aws/whats-new/2020/09/cloudfron...). HN isn't behind CloudFront, though, so you probably mean their HTTP(s) load balancers (ALB) don't support TLS 1.3. Even that's an incomplete view of the load balancing picture, since the network load balancers (NLB) do support TLS 1.3, https://aws.amazon.com/about-aws/whats-new/2021/10/aws-network-load-balancer-supports-tls-1-3/ https://aws.amazon.com/about-aws/whats-new/2021/10/aws-netwo....
- Aeolun 4y agoNLB’s support everything that goes over TCP or UDP, that’s not exceptionally surprising.
- WatchDog 4y agoYeah but NLB can offload TLS from the app, which is what the parent commenter linked to. It’s not just passing through the TLS from the app(which is also possible).
- anarsdk 4y ago
- dang 4y agoCan you please not post like this to HN? I understand the fun of these sorts of swipes in a more intimate context, but on the open internet it's just guaranteed to make discussion sour and dumb. What you should do instead is explain what you know about NLBs in a neutral, respectful way that the rest of us can learn from. Either that, or it's always ok not to post. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
- deathanatos 4y ago> When I remind HN readers that most site addresses are more static than dynamic, I am basing that statement on evidence i have collected. Sure. But without seeing the other sides argument, I have to wonder if their point wasn't that they're not designed to be stable for the purpose of identifying a service/thing on the Internet; things can and do move and change. Hardware failure is a good example of that. Just like a house address, those too are normally stable but people can & do move. Just with software, it's like we look our friend up in the white pages¹ prior to every visit, which one might not do in real life. ¹oh God I'm dating myself here.
- deleted 4y ago[deleted]
- phailhaus 4y agoAside: How did you get the superscript? Is that supported by HN's formatter or is that just a literal superscript character?
- deathanatos 4y agoUnicode has code points for superscript/subscript digits. That one is U+00B9: https://www.compart.com/en/unicode/U+00B9 https://www.compart.com/en/unicode/U+00B9 (So it's "normal text", as far as HN is concerned. Note that HN does filter some things, like emoji.) I was on macOS when I typed it, there it's Control+Cmd+Space, and then search for "super" which gets close enough. On my Linux machine, I can either do Compose, ^, 1, or Super+e and then search for it. (But both of these require configuration; either setting a key to be Compose (I sacrifice RAlt), or setting up whatever it is the IME I have is for Super+e.)
- viraptor 4y agoNot only super/subscript, there's also some convenient fractions: ½ ⅔ ⅜, etc
- 1vuio0pswjnm7 4y agoThat was not the other side's "point". I routinely make the statement: Most sites submitted to HN have realtively static IP addresses, i.e., these addresses can change, but in fact they change only infrequently, if at all.^1 This is not an opinion. It is not a mindless regurgitation of something I read somewhere. I am looking at the data I have, not theorising. From where I sit, there is nothing to argue about. 1. Why do I state that. Because I kept reading about why DNS was created and always encountered the same parroted explanation, year after year. Something along the lines that IP addresses were constantly in flux. That may have been true when DNS was created and the www was young. But was it true today. I wanted to find out. I did experiments. I found I could use the same DNS data day after day, week after week, month after month, year after year. Why would I care. Because by eliminating remote DNS lookups I was able to speed up the time it takes me to retrieve data from the www.^2 Instead of making the assumption that every site is going to switch IP addresses every second, minute, day or week, I assume that only a few will do that and most will not. I want to know about those sites that are changing their IP address. I want to know the reasons. When a site changes its IP address, I am alerted, as you see with today's change to HN's address. Whereas when people assume every site is frequently changing its IP address, they perform unnecesary DNS lookups for the majority of sites. That wastes time among other things. And, it seems, people are unaware when sites change addresses. 2. Another benefit for me is that when some remote DNS service does down (this has happened several times), I can still use the www without interruption. I already have the DNS data I need. Meanwhile the self-proclaimed "experts" go into panic mode.
- betaby 4y agoConsistently without IPv6, even though AWS supports IPv6.
- pelagicAustral 4y agoI'm Ok with whatever. I almost had a anxiety breakdown today... And realised how hooked I was on this site. I do not have any other social network accounts and it was truly telling how much this has become my de facto 'twitter'/'facebook'
- olalonde 4y ago> Little known fact: HN is also available through Cloudflare. Unlike CF, AWS does not support TLS1.3. This is not working while HN uses the AWS IP. What does that mean? How do you access HN through CloudFlare and what do you mean by AWS not supporting TLS1.3? You can certainly run any https server on EC2, including one that supports TLS1.3.
- 1vuio0pswjnm7 4y agodrill news.ycombinator.com @108.162.192.195 Those are CF IP addresses. Before HN switched from M5 to AWS, CF was an alternative way to access HN. echo|openssl s_client -connect 50.112.136.166:443 -tls1_3
- jffry 4y agoI too have no idea what OP means by saying it's "available through Cloudflare" $ dig +noall +answer A news.ycombinator.com news.ycombinator.com. 0 IN A 50.112.136.166 $ nslookup 50.112.136.166 166.136.112.50.in-addr.arpa name = ec2-50-112-136-166.us-west-2.compute.amazonaws.com.
- tedmiston 4y ago> > Little known fact: HN is also available through Cloudflare. ... > What does that mean? How do you access HN through CloudFlare ... As written, it seems to suggest HN is in the Cloudflare cache. But, I don't think there's a way to access the cached version if a site's not down. I wasn't around during today's outage, so I can't speak to whether a generic Cloudflare cached version of HN was available during the downtime.
- dangus 4y agoThis sort of post-outage discussion and speculation is kind of like debating where Justin Bieber gets his tires changed and what brand he’s going to use after he ran over a nail. The only reason we are here doing this sort of thing is because we are “Justin Bieber” fans. We aren’t here because changing a tire is interesting, unique, nor will we learn anything from it – especially this particular tire (HN is like the Toyota Corolla of “vehicles” compared to the other complex mission-critical distributed systems that make up other popular web services).
- usrn 4y agoHN is written in a custom lisp dialect. I'd argue it's more like one of those designer cars than a corolla.
- loxias 4y ago> We aren’t here because changing a tire is interesting, unique, nor will we learn anything from it Speak for yourself, some of us (at least me) find "postmortum writeups" FASCINATING!! I read them every chance I get. Most of the time the root cause wouldn't have affected me, but, I still occasionally will read one, think "oh crap! that could have bit me too!", then add the fix to my "Standard Operating Procedures" mental model, or whatnot. Some of us are still trying to "finish the game" with zero losses. :)
- throwoutway 4y ago> I have been waiting for years. I collect historical DNS data. Why do you collect this? And in what format?
- Amfy 4y agothere’s commercial services that do this already: Securitytrails
- alberth 4y agoWhy the move? Hopefully it’s simply M5 didn’t have a server ready and they’ll migrate back. Vultr has a great assortment of bare metal servers. https://www.vultr.com/products/bare-metal/#pricing https://www.vultr.com/products/bare-metal/#pricing
- jacooper 4y agoThere is also Hetzner, their dedicated server pricing is very good.
- ta988 4y agoBut they don't have servers in US do they?
- Amfy 4y agothey do, in Ashburn. I would not host production with them.
- whitepoplar 4y agoWhy?
- ev1 4y agoI host a lot of production there, but always some layers underneath - like a high CPU or backing storage layer or database, worker nodes that pick up tasks. Never the front user facing. Do not let any user generated content be accessible from any hetzner IP or you are one email away from shutdown pretty much. Don't forget germany's laws on speech too; they are nothing remotely similar to US. I would host, for example, a corporate site just fine, but last thing ever would be a forum or image hosting site or w/e
- Amfy 4y agothis. matches my experience and the ones in my circle
- rhacker 4y agoIt is noticeably slower now.
- metadat 4y agoIt actually seems faster for me compared to the past two weeks. Not sure if I was being throttled or something, at times it felt like it.
- dang 4y agoWe don't throttle specific users*. At one time we did (it was called slowbanning) but that's been gone for close to a decade now. It's a bit disconcerting how often I say "close to a decade now" now. * Edit: we do rate-limit some accounts (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20rate%20limit&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...), which throttles how much they can post to the site, but we don't throttle how quickly they can view the site.
- metadat 4y agoThanks for the disclosure, if it gets crazy slow many I'll shoot you an email next time.
- dang 4y agoI'm worried about that too. Don't have good data or even consistent gut feelings yet. The big test will be when we hit peak load, perhaps on Monday or Tuesday.
- rhacker 4y agoI think overall the feeling of "slowness" last week may have been from everyone coming back in all at once or something. It seems just as fast as it was again.
- Tijdreiziger 4y agoWhere are you located? Ever since I read someone's comment that 'HN is the fastest site they regularly visit', I've wondered if that's because they're in the western US (where HN is hosted). I am in Western Europe and HN is decently fast, but tweakers.net and openstreetmap.org are faster.
- bushbaba 4y agoCurious, why not Digital Ocean? Which is a YC investment? I get why you choose AWS on a technical basis, but wouldn't it make sense to give the business to Digital Ocean to help their brand?
- dang 4y agoPretty sure Digital Ocean is not a YC-funded startup. We wouldn't make HN decisions on that basis anyhow, though, I don't think. Maybe if all other things were literally equal.
- Godel_unicode 4y ago> help their brand Who are the people that use HN and would notice it moved hosting, but haven’t heard of Digital Ocean?
- toast0 4y agoDigital Ocean recently ended their support for FreeBSD, and HN runs on FreeBSD (at least at M5), and while you don't really need to run an OS your vm provider supports, it's kind of nice.
- dang 4y agoIt's temporary.
- edanm 4y agoMaybe you should pin this comment? I almost lost it in the shuffle.
- dang 4y agoI thought about it, but it's fun to leave rewards for people who read entire threads.
- skywal_l 4y agoIn this kind of thread, I usually do CRTL+F "dang" ...
- boulos 4y agoI upvoted your top-level comment! But what happens if the comments on this topic get long enough for pagination to kick in? Very few people click the new next button for the linked list :).
- andreareina 4y agoThe "next" link is such a godsend, and a nice little surprise when I discovered it. Love it (and not getting an obnoxious "what's new" dialog box)
- porker 4y agoTIL what the "next" link does. Thanks!
- herpderperator 4y agoHow will you switch back to the new server once it's ready without losing database records?
- usrn 4y agoMy mail server's IP has been the same for 4 years now. Even the machine I'm typing this on only changes IPs every couple years or so and that's a residential IP that isn't supposed to be static. Of course you want everything in DNS and if your IP is supposed to be dynamic you should have provisions for automatically updating it (I have some shell script somewhere that calls nsupdate over ssh although I looked for it the other day and couldn't find it which is a bit disturbing.)
- Tijdreiziger 4y agoMany dynamic residential IPs seem to be 'your IP changes after the connection has been offline for some amount of hours', i.e. only in the rare event of some long-lasting outage.
- loxias 4y ago> someone on HN tried to argue with me that IP addresses will never stay the same for very long Deeply deeply agree with you. Not whoever was arguing. :) My "dialup" VM in the cloud I use for assorted things has had the same IP for at least 7 years, probably longer. (Thanks Linode!) After a few years, it's honestly not that hard to remember an arbitrary 32bit int. :) $ w3m -dump http://846235814 # ;)
- 1vuio0pswjnm7 4y agoThat's another myth that always iritated me: IPv4 addresses cannot be memorised. I have a few memorised, mainly some TLDs and Internic. With those I can "bootstrap" using DNS/FTP to get any other address. IPv4 are no more difficult to remember than phone numbers. Maybe people are not memorising phone numbers anymore because of mobile phone contacts storage but it does mean we are incapable of doing so. Thank you for stating the truth.
- loxias 4y agoYou're welcome. :) FWIW, I still memorize phone numbers too! I also avoid phone calls like the plague, so in reality it's just a handful of numbers per year.
- bawolff 4y agoLiterally nobody thinks its impossible to memorize 4 8-bit numbers. A bad user experience, sure. > I have a few memorised, mainly some TLDs and Internic. With those I can "bootstrap" using DNS/FTP to get any other address. Pretty sure that's true of 90% of everyone here, since all you really need to memorize is a public dns resolver, and 8.8.8.8 or 1.1.1 .1 is a particularly easy address to remember.
- andreareina 4y ago1.1 is even easier to type!
- 4y ago
- dubcee349 4y agoThe IP you resolve to for Hacker News may not have changed or remained static but that doesn't have to the case for the rest of the world. Depending on market there may be one more many IPs a specific site resolves to.
- ksec 4y agoI wonder if it is time for Server Upgrade. HN was running on Xeon(R) CPU E5-2637 v4 [1], that is SandyBridge era. A 2 Core CPU serving 6M request a day. If iPhone had more memory the whole of HN could be served from it. [1] https://www.intel.com/content/www/us/en/products/sku/64598/intel-xeon-processor-e52637-5m-cache-3-00-ghz-8-00-gts-intel-qpi/specifications.html https://www.intel.com/content/www/us/en/products/sku/64598/i...
- minimaul 4y agoThat's the wrong SKU, it's actually Haswell/Broadwell and 4c8t, not 2c4t: https://www.intel.com/content/www/us/en/products/sku/92983/intel-xeon-processor-e52637-v4-15m-cache-3-50-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/92983/i...
- Tijdreiziger 4y agoI mean, if it's capable of handling the traffic, is there any reason to upgrade?
- namibj 4y agoEnergy efficiency.