3 ms·

Today's gentle reminder that there are things other than network or service outages that can and do occur that might necessitate an outside backup. What happen

Today's gentle reminder that there are things other than network or service outages that can and do occur that might necessitate an outside backup. What happens if AWS or [insert other megacloud] decides your account needs to be nuked from orbit due to a hack or some other confusion? We almost had this happen over the summer because of a problem with our bank's ability to process ACH payments. Very frustrating experience. Still isn't fully resolved. What happens if an admin account is taken over and your account gets screwed up? What happens if an admin loses his shit and blows up your account? What happens if your software has a bug that destroys a bunch of your data or fubars your account? There's a ton of cases where having at least a simple replica of your S3 buckets into a third-party cloud could prove highly valuable.

hinkley 5y ago
I would make a friendly wager that AWS user IDs don't contain check digits, let alone bullet proof ones (simple check digits don't guard against transposition errors). And that somewhere, someone can manually enter an account to delete, and that one of us will eventually have an account numbered XXX1234 and some idiot with account XXX1243 will legitimately earn an account deletion, but we'll be the ones who wake up to bad news.
btown 5y ago
Would you be able to expand at all about the ACH/AWS connection, obviously without identifying details? Was it just a miscommunication around AWS billing and them thinking you weren't paying? Or did AWS somehow put itself in the middle of, or react to, your use of ACH payment processing for *non-AWS* receivables or payables? If the latter, that's a business risk I'd never even thought about. I'm not even sure how they'd know. But I'm thoughtful that things like the MATCH list [0] exist, and how easily a merchant can accidentally wind up on these lists from either human error or a small amount of high-value chargebacks. If cloud providers are somehow paying attention to merchant services reputation, that would be very scary for many businesses! [0] https://www.merchantmaverick.com/learning-terminated-merchant-file-tmf-aka-match-list/ https://www.merchantmaverick.com/learning-terminated-merchan...
- numbsafari 5y ago
  Like most of these things, it was a series of unfortunate events. In our case {LargeCloud} acquired {SaaSVendor}. We were already using {LargeCloud}, with an existing billing arrangement. When {LargeCloud} got around to integrating the {SaaSVendor} into their billing system, it exposed multiple bugs in {LargeCloud}'s billing system, and ultimately limitations in our bank's internal systems--a well known establishment and it would blow your mind to learn how much manual crap they do. Traditionally, we received favor from {SaaSVendor} through Invoices. But when {SaasVendor} was subsumed by {LargeCloud}, we stopped receiving invoices. Our internal ops reached out to {LargeCloud} about this two days before we got our first "You will experience Dire Consequences" email from {LargeCloud}'s Robot Overlords. Our attempts to contact {LargeCloud} regarding this concerning message was always routed to a Robot Overlord who only spoke in tongues and could not solve our problems. Eventually, were able to get the Robot Overload to escalate us to a Robot Superlord that would only tell us to "follow the instructions in this handy dandy web page thing", except following the instructions always summoned a "Server 500" Demon, which {LargeVendor} claimed was impossible because their Robots are Divine and Holy. Finally circling back through random Human Actors we were able to avert the countdown to destruction. Some Robot Necromancer was able to resurrect our billing account from the "Server 500" Demon, but we would now need to setup automatic ACH payments, as whatever fix was implemented could only persist with regular monthly succor upon the alters of the Federal Reserve Automated Clearing WaffleHouse. Invoices, payments arranged through Our Lady of Visa and The Master Card would no longer suffice. We believed we had made the appropriate incantations before FratBoy 3000 at our local branch of the Federal Reserve Chapel. However, we eventually received another threat of Dire Consequences from {LargeCloud}, indicating that our prayers were not received. It took significant supplication in order to get FratBoy 3000 to confirm that our Federal Reserve Chapel had misrouted our prayers, deducting them from our account, but sending them to the wrong Demon, through no fault of our own. The whole time this was going on, we kept getting threats of Dire Consequences. We were told by Human Actors to have great faith, that the {LargeVendor} Robot Overlords had been placated through their secret prostrations. FratBoy 3000 was replaced by our Federal Reserve Chaplain, who informed us that they had no robots, this was all the result of Human Actor failures, but that, forthwith, all of our prayers could be answered if we moved all of our faith into a New Account which itself required additional monthly supplication, but would ensure divine routing of our prayers would always be successful. To this day, we continue to make our monthly pilgrimage to our local Federal Reserve Chapel, supplicating upon all necessary altars. The threats of Dire Consequences from {LargeCloud} have subsided. But we have cast ourselves out onto the trail, seeking refuge from a more receptive and responsive Federal Reserve Chapel. Everybody focuses on "what if us-east-X goes down", but, literally, sometimes it's a combination of billing and payment issues that can keep you up at night.

hinkley 5y ago

I would make a friendly wager that AWS user IDs don't contain check digits, let alone bullet proof ones (simple check digits don't guard against transposition errors). And that somewhere, someone can manually enter an account to delete, and that one of us will eventually have an account numbered XXX1234 and some idiot with account XXX1243 will legitimately earn an account deletion, but we'll be the ones who wake up to bad news.

btown 5y ago

Would you be able to expand at all about the ACH/AWS connection, obviously without identifying details? Was it just a miscommunication around AWS billing and them thinking you weren't paying? Or did AWS somehow put itself in the middle of, or react to, your use of ACH payment processing for *non-AWS* receivables or payables? If the latter, that's a business risk I'd never even thought about. I'm not even sure how they'd know. But I'm thoughtful that things like the MATCH list [0] exist, and how easily a merchant can accidentally wind up on these lists from either human error or a small amount of high-value chargebacks. If cloud providers are somehow paying attention to merchant services reputation, that would be very scary for many businesses! [0] https://www.merchantmaverick.com/learning-terminated-merchant-file-tmf-aka-match-list/ https://www.merchantmaverick.com/learning-terminated-merchan...

numbsafari 5y ago
Like most of these things, it was a series of unfortunate events. In our case {LargeCloud} acquired {SaaSVendor}. We were already using {LargeCloud}, with an existing billing arrangement. When {LargeCloud} got around to integrating the {SaaSVendor} into their billing system, it exposed multiple bugs in {LargeCloud}'s billing system, and ultimately limitations in our bank's internal systems--a well known establishment and it would blow your mind to learn how much manual crap they do. Traditionally, we received favor from {SaaSVendor} through Invoices. But when {SaasVendor} was subsumed by {LargeCloud}, we stopped receiving invoices. Our internal ops reached out to {LargeCloud} about this two days before we got our first "You will experience Dire Consequences" email from {LargeCloud}'s Robot Overlords. Our attempts to contact {LargeCloud} regarding this concerning message was always routed to a Robot Overlord who only spoke in tongues and could not solve our problems. Eventually, were able to get the Robot Overload to escalate us to a Robot Superlord that would only tell us to "follow the instructions in this handy dandy web page thing", except following the instructions always summoned a "Server 500" Demon, which {LargeVendor} claimed was impossible because their Robots are Divine and Holy. Finally circling back through random Human Actors we were able to avert the countdown to destruction. Some Robot Necromancer was able to resurrect our billing account from the "Server 500" Demon, but we would now need to setup automatic ACH payments, as whatever fix was implemented could only persist with regular monthly succor upon the alters of the Federal Reserve Automated Clearing WaffleHouse. Invoices, payments arranged through Our Lady of Visa and The Master Card would no longer suffice. We believed we had made the appropriate incantations before FratBoy 3000 at our local branch of the Federal Reserve Chapel. However, we eventually received another threat of Dire Consequences from {LargeCloud}, indicating that our prayers were not received. It took significant supplication in order to get FratBoy 3000 to confirm that our Federal Reserve Chapel had misrouted our prayers, deducting them from our account, but sending them to the wrong Demon, through no fault of our own. The whole time this was going on, we kept getting threats of Dire Consequences. We were told by Human Actors to have great faith, that the {LargeVendor} Robot Overlords had been placated through their secret prostrations. FratBoy 3000 was replaced by our Federal Reserve Chaplain, who informed us that they had no robots, this was all the result of Human Actor failures, but that, forthwith, all of our prayers could be answered if we moved all of our faith into a New Account which itself required additional monthly supplication, but would ensure divine routing of our prayers would always be successful. To this day, we continue to make our monthly pilgrimage to our local Federal Reserve Chapel, supplicating upon all necessary altars. The threats of Dire Consequences from {LargeCloud} have subsided. But we have cast ourselves out onto the trail, seeking refuge from a more receptive and responsive Federal Reserve Chapel. Everybody focuses on "what if us-east-X goes down", but, literally, sometimes it's a combination of billing and payment issues that can keep you up at night.