5 ms·
I've been in the other side of this... You build a new data backend for a service, you test it a million ways, and finally you migrate all the user's over. Ex
by londons_explore 5y ago
I've been in the other side of this...
You build a new data backend for a service, you test it a million ways, and finally you migrate all the user's over.
Except it goes wrong. 999,999,999 user accounts migrated. 1 account failed.
Leaving the old infrastructure up for that one user is costing millions of dollars a day, hurting performance reliability and failover and holding up hundreds of other engineers projects.
Turns out the failure is because the user managed to create a filename with 67000 byte order marks in and nothing else, and it trips up some parser in the new system. They probably did it using a custom client or some kind of exploit checking tool. It's probably some security vulnerability researchers test account, but we can't be sure because we can't look into the account for privacy reasons.
Yet fixing the bug for that 1 user account is going to take a while - a quick bodge solution breaks other user accounts, and the proper fix is part of Jane's project due next quarter.
The decision is made. We're going to treat the needs of the many over the few. We're going to disable this one user account and move on. That user will have to re-upload their data, and this time they won't manage to upload a file with 67000 byte order marks in the filename.
- himinlomax 5y agoAnd that's ok — as long as you tell that user that there is a problem and that you're working on it.
- rgj 5y agoI thought you said you tested it?
- londons_explore 5y agoNever thought to test a massive string of byte order marks... Extensive testing always misses something. Testing on real user data could have been done but has privacy issues (now we can't inspect logs to check stuff is working), but even so, the user could have put the offending data in after the testing was done.
- InitialLastName 5y agoIn that case, if your system leaves you with no way to contact that user and say "Hey, there's an issue with your account that is causing a problem that we need your help to solve, get back to us within 3 days or else we're going to need to reset your account", you've failed at customer service.
- nexuist 5y agoFrom the perspective of the customer to Apple - this is the opposite of customer service. It's not my problem if you built some new system that doesn't work with my data; you let me do whatever I wanted and didn't warn me that this wouldn't fly at some far-off point in the distant future. I'm not paying for the privilege to do QA for you. If you're having problems migrating my data you should let me continue using my account until you've fixed it on your end - I shouldn't have to know or be interrupted by an internal business process.
- brewdad 5y agoYou are correct. You will have your account disabled.
- 411111111111111 5y agoThat communication attempt would cause an even bigger shit storm... From a Publicity PoV, just disabling without communication is the best course of action.
- devoutsalsa 5y ago> you let me do whatever I wanted and didn't warn me that this wouldn't fly at some far-off point in the distant future It's not possible to anticipate every need, nor is it practical. You do the best you can to build something useful with the resources you have, ship it, and know that something will go wrong at some point. When something breaks, and it will, you fix it as best you can. If suspending your account for 5 days is the best you can do, it's the best you can do.
- echelon 5y agoCool, but it doesn't look like this guy doing anything out of the ordinary. He has a right to complain about his horrible experience and the way Apple customer support treated him.
- ska 5y ago> We're going to disable this one ... All pretty reasonable; The only thing wrong with your suggestion is it's missing the part where you contact that user and let them know what you are doing and why...
- moralestapia 5y ago>is costing millions of dollars a day VERY FEW (two?) companies in the world have operating expenses of 1B+/year on data infrastructure. I find your story quite hard to believe after reading that.
- SOLAR_FIELDS 5y agoI read it as the system being down costing the company millions of dollars through lost sales/revenue, not direct cost of infrastructure.
- londons_explore 5y agoIf your normal infra is 80,000 servers which you own, then running costs are quite low. But during the migration, you are renting in 80,000 servers from AWS too, so the two systems can run in parallel, which have a far higher running cost. As soon as the migration is over, workloads can return to cheap owned hardware.
- moralestapia 5y agoSo, you rented 80,000 AWS intances (!) at about (at least*) 12 bucks each one, per day? Also, for reference: https://www.ibtimes.com/apple-amazons-biggest-customer-spends-over-30m-month-aws-2787325 https://www.ibtimes.com/apple-amazons-biggest-customer-spend...*
- jjav 5y agoYou wouldn't need 80,000 servers to host the activity of one user.