TrogdorTheMan

56 ms·

1.
by TrogdorTheMan 5y ago
Yup ^^ this. That and some dang help with managing security audits for our security team -.-. Just can't afford the man power to do it all.
2.
by TrogdorTheMan 5y ago
From what I've seen on some internal email chains, there is a fail-over process for HA multi-region/site, but it didn't work right. Whomp whomp.
3.
by TrogdorTheMan 5y ago
I'm willing to bet we can get at least the 20% return, they're still "officially" down. I bet it'll be a few hours till they say we're good. Everyone make sure you've got detailed tickets into their servic
4.
by TrogdorTheMan 5y ago
Oh lawd its starting to come back up! Pingdom monitor is flapping and we're starting to get some successful, albeit slow, logins!
5.
by TrogdorTheMan 5y ago
I don't think so, I think that it was a combo of malicious intent and some indexes that never got run. I guess you might call it a bad migration since indexes didn't get run, but that seems more like a catalyst than a root. https
6.
by TrogdorTheMan 5y ago
heh, smells like its similar. thanks for the link.
7.
by TrogdorTheMan 5y ago
Do you have a link to that previous post-mortem?
8.
by TrogdorTheMan 5y ago
Yup, exactly. We're actively working on ripping them out of our software, this is ridiculous. I knew we should have taken the time to make a fail safe...sigh...
9.
by TrogdorTheMan 5y ago
They're fast approaching the 95% availability mark for the month! Enterprise customers are already guaranteed 50% return on fees, I suspect they'll be giving out MUCH more than that.
10.
by TrogdorTheMan 5y ago
We managed to get a reply from a C level. All we could get out of them was "something to do with our DB, but we don't know the root yet. Our fail-over process didn't work. This will never happen again". Also, it only too