3 ms·
Ok a new server in US region is up. Unfortunately the disk filled up on the Europe server and Canadian server. I had to nuke all the Ephemeral sessions. Capabi
by keepamovin 3y ago
Ok a new server in US region is up.
Unfortunately the disk filled up on the Europe server and Canadian server. I had to nuke all the Ephemeral sessions. Capability should be back now. OS told me disk was 100.0% used when I logged in.
It was all tempfs mounted for each user. I never realized its importance. Will need bigger disks.
I'll add some timed updates to keep things organized below this point:
17:57:00 Z
Will add 2 new 32 core servers in Europe, and Asia, as both those regions are seeing lots of demand. Asia is at 100% right now.
18:03:00 Z
Just started creation of the Japan server. Once I get the install script started, I'll move to create the Europe one.
18:18:00 Z
Okay, Japan server should be online. Sorry about everybody in SEA that I just nuked your sessions, that must have sucked. The server was hard to manage at 100% utilization. Moving to create a new Europe server now.
18:26:00 Z
Okay Europe server is provisioned and setup is starting now. It should be live in about 10 minutes.
18:32:00 Z
London server is live. Stats update: since posting to HN 2 hours ago we had about 2000 sessions across (initially) 4 16core 32GB 128disk servers. So far have had to add 3 more 32/64/1TB servers, with disk being the unexpected cost to this type of load. BTW if anyone is reading this how was your experience? Was it slow/fast? Responsive? What region were you in?
18:42:00 Z
I'm monitoring the 7 servers now.
18:48:00 Z
Right now server selection is based on RTT/ping from client. I'm thinking it might be good to have ping endpoint include the current load. But I don't want to bork it.
19:31:00 Z
Okay, load average has been added to the ping check for fastest server. We now pick between the fastest two pings based on which has the lowest load. Should work well.
- deleted 3y ago[deleted]