4 ms·
> I was able to sign in to the AWS console and resolve the issue Kids these days. I had a RAM stick fry in one of the physical machines sitting in a colo 1 ho
by apankrat 7y ago
> I was able to sign in to the AWS console and resolve the issue
Kids these days.
I had a RAM stick fry in one of the physical machines sitting in a colo 1 hour drive away. Not die, but just start flipping bits here and there, triggering most bizarre alerts you can imagine. On the night of December 24th. Now, that was fun.
--- To add ---
If you are a single founder - expect downtime and expect it to be stressful. Inhale, exhale, fix it, explain, apologize and then make changes to try and prevent it from happening again. Little by little, weak points will get fortified or eliminated and the risk of "incidents" will go down. There's no silver bullet, but with experience things become easier and less scary.
- quickthrower2 7y agoKids these days. Insert punchcard anecdote... Transistors! We had to replace valves in my day.
- thorwasdfasdf 7y agogeez... why do these things always need to happen right around the holidays. I mean the disaster couldn't wait a week or two later?
- jgimenez 7y agoReminds me of an int32 <> int64 mismatch that overflowed precisely also on the night of Dec 24th!
- idlewords 7y agoEffective apologizing is the #1 business skill of the one-person tech company.
- fanf2 7y agoThat reminds me of the time we had a DIMM actually melt on the 22nd December http://fanf2.user.srcf.net/hermes/doc/misc/orange-fire/ http://fanf2.user.srcf.net/hermes/doc/misc/orange-fire/
- quaquaqua1 7y agoAnd for everyone else who doesn't feel like doing that, Amazon Lambda and a server in your closet are two other options. If your workload doesn't fit into such a setup, then you simply need to take investment, optimize your code, or move closer to your colo
- gspetr 7y agoIsn't this exactly the case where you could have avoided this hassle entirely had you shelled out some cash for ECC memory?
- taf2 7y agoLive and learn is what I think the take away of this story is all about... I had a server fail dec 25 mid morning. It caused failures in away I had thought about before because instead appearing completely dead it was alive enough to not let go of any tcp connections. For the critical component in question, I didn’t have the correct timeouts in place... so as the single operator I was fortunate that my wife was also my co founder and so was a bit more understanding.
- deleted 7y ago[deleted]