6 ms·
Sorry about that! It's entirely my fault. I screwed up in a few ways, which is what you always see in something like this. Im on an overnight business trip fr
by e1ven 15y ago
Sorry about that!
It's entirely my fault. I screwed up in a few ways, which is what you always see in something like this.
Im on an overnight business trip from Boston to Dallas.. My Real Job wanted me to tour a new datacenter down here.
In any event, I've been running straight, without sleep since yesterday AM when I flew out on a Red Eye, so I.. I'll admit it.. I went to bed.
I was sleeping peacefully from 1AM to 4AM, cradling my old-school, screaming-loud pager secure in the knowledge that all my little servers would alert me if they had problems.. I have Pingdom, PagerDuty, Nagios, Munin, I've got monitoring 16 different ways..
When I woke up at 4AM (3 hours is enough. Why not?) to make my flight back to Boston, I see I've received several hundred text messages. Hrmm.. What is this?
I double-checked the work-servers, but those were fine, and have Other staff to help watch them, and escalate. Checking email, I see Pingdom is patiently explaining that RoboHash is down.
"But WAIT?!", I can hear your exclaim!
Why didn't your pager wake you up?
Well, it turns out that old-school pagers are.. Regional. Once I'm outside of NewEngland, it's a cool looking retro piece of trash.
The text messages arrived, but my sleeping brain was able to peacefully ignore them in bliss.
As to what actually happened? Linode migrated my machine to a new datacenter, and rebooted it.
I had a bug in my init script, so it didn't start up properly when rebooted.
Basically, I need to create a Ramdisk, then copy the code from the stable position on HDD. This is because a bazillion hits/second would overwhelm the disk if I loaded it manually from disk each time.
Anyway, Nginx was started (hence the error), but the init script was trying to start up my python code, without creating the ramdisk first, so.. RoboFail.
Sorry about the problem. Imagine my embarrassment.
On the plus side, I'm now in a TSA-approved boarding area, so I have a good 20 minutes to fix the init script before my flight back. ;)
- Roritharr 15y agoThank you for the explanation. A good example that something like this can always hit you, no matter of how prepared you are.
- e1ven 15y agoAlso, this gives me a good opportunity to note some new features- SSL support is in. Just use https://robohash.org/goldfish.png https://robohash.org/goldfish.png Also, We can use Gravatar instead of RoboHash, if a user has a Gravatar- Pass the gravatar-style hash in to RoboHash, and it'll return if it exists. http://robohash.org/620050a4db5104bae758cd75171d64ca?gravatar=hashed http://robohash.org/620050a4db5104bae758cd75171d64ca?gravata... You could also just pass the email into Robohash- http://robohash.org/colin@sq7.org?gravatar=yes http://robohash.org/colin@sq7.org?gravatar=yes But then your email is out on the public internet, so the hashed method is better if you can do it. With Either method, RH will return a Gravatar if one exists, and if not, it'll return a Sexy looking Robot, custom stamped just for you.
- dexen 15y agoPardon the (perhaps very silly) question, but... if you have enough RAM for the ramdisk, isn't that enough RAM to cache all the relevant files anyway? So the file open()/read() would not hit HDD itself time and time again; they'd be served (by the OS) from cache? Of course, cache warm-up would remain a possible problem source.
- simcop2387 15y agoI believe it's usually done (I've not had to do it) so that you can force some small set of data to always be in ram. This way even in the case of someone managing to cause something to leak memory the code will always be in ram. Whether this is really needed or not I can't say.
- e1ven 15y agoIn this case, it's because I don't want to assume that every dataset that will be used with this code stays in RAM, and it's simpler to handle things on the FS level by adding a new png to the stack, than to keep track of the images in RAM. This way the code can be "simpler"
- 0x12 15y agoWas the linode migration at your instigation or did they decide to do that all by themselves? (of course you should have configured the box properly and done a test reboot to see if it would all come up well but as you already wrote there are multiple factors at work here and I'm wondering if Linode is in the habit of taking machines down that are working just fine without their owners consent).
- leftnode 15y agoIf it was the migration I was told about, no, they give you about 2 weeks notice that they're going to do it on X date.