10 ms·
This can be just calculated risk. Incredible damage for some users can be negligible damage to the company. In this case just 10TB of data was lost, so maybe t
by nabla9 6y ago
This can be just calculated risk.
Incredible damage for some users can be negligible damage to the company. In this case just 10TB of data was lost, so maybe thousands of users for the long term storage option.
Losing customer data has potential for big reputation or brand damage, but surprisingly often the damage is relatively small. Putting in too much effort when potential damage to business is minimal may not be worth of the cost.
- ChuckNorris89 6y agoThis. Outside of FAANG, Gov and finance, very few consumer facing companies have the budgets to engineer redundancies.
- nix23 6y ago>budgets to engineer redundancies You dont need that, but a backup ;)
- virgilp 6y agoThey obviously do have _some_ backups (since were able to restore all videos and all photos - but not all photos at the original resolution). In the end it's all a tradeoff... ignoring cost, we all know what to do to get pretty safe storage. But how much are you really willing to pay, per month, for that safety?
- nix23 6y ago>but not all photos at the original resolution Thats why you backup the original data and NOT the downscaled etc one. >But how much are you really willing to pay, per month, for that safety? You have to backup ALL your customers data, otherwise DONT go into file-hosting business.
- ChuckNorris89 6y agoWith your argument, Google photos should also not be in business since they also don't back up your original resolution unless you pay.
- virgilp 6y agoBut they did backup the original data - the article states that the problem was with deploying wrong cleanup code on the long-term storage (that was supposed to be only deployed for the short-term storage). How do you guard against that? You could store everything in backup storage & never delete it, but that has implications: - privacy - legal (GDPR) - cost etc. It's basically a "black swan" event, they designed their system without anticipating this particular failure mode. It's easy to claim in hindsight that "they should've or they should not go into that business" but seriously.... look in the mirror. Are you as tough on yourself when it comes to your customers? Can you honestly claim that there's absolutely no failure mode that could result in customer data loss?
- jefftk 6y ago> since were able to restore all videos and all photos - but not all photos at the original resolution "Canon has said that there is no technical measure to restore lost video files, but that photo files can be restored – albeit not at their original resolution."
- virgilp 6y agoAh I somehow misread that, thought they can restore videos but for photos they don't have the original resolution.
- classified 6y agoThat. Who in their right mind would expect IT professionals to take any responsibility whatsoever for the empty promises they make to their users?
- labster 6y agoI’m sure plenty of engineers will be made redundant by this fiasco.
- jacquesm 6y agoAnd even then, if you store your data there the responsibility is ultimately still yours.
- StillBored 6y agoI suspect this is more a view from the trenches of "modern" SAS companies hosting their offerings on cloud providers charging a premium for storage+transit. Because I can assure you, from the perspective of someone who worked for >10 years in a backup related part of the storage industry, our product penetration covered a huge portion of public companies, and untold numbers of non public businesses. Given that we were one of a half dozen or so similar products and our competitors also had non-insignificant footprints in places we didn't most companies were taking backup of their core systems very seriously. And even then, today, there are dozens of online backup services. There are plans which can backup a TB or two of data a year off a NAS/etc and provide a read only historical snapshot for the price of a couple starbucks coffees a month (and a reasonable internet connection). And then there are the dozens of mom & pop places which just use two or three ~$100 USB disks on a rotation with windows backup/time machine/whatever. So, no, lack of a proper backup (and recovery) strategy isn't really a question of cost. Its more a question of the competencies of the individuals charged with configuring/maintaining the IT systems. And sure, if you have .5PB+ of data that needs backing up, a tape library, or replicating dedupe appliance, software license, etc is going to cost $$$$ but overwhelmingly when compared with the overall IT budget for machines+storage+administrators those devices rarely crack double digit percentages of the yearly IT budgets. Further, there are ways to lower those costs even more with a bit of hunting on the resale markets where picking up a three year old piece of equipment for a tiny fraction of its original cost isn't unusual.
- kohtatsu 6y agoThis comment reads as "This is normal; nothing to see here move along". Which is more of an indictment on the state of things than on the comment itself. If you can't protect people's shit, don't offer to hold onto it for money!
- nabla9 6y agoI try to avoid normative statements because it leads to better discussion. Unfortunately this style of discussion leads to comments that project normative stance, or criticize me from not making my normative position clear. (positive statement is statement of what without indication of approval or disapproval, normative statement expresses a value judgment)
- TheNorthman 6y ago> In this case just 10TB of data was lost Seems like a value judgement to me. Anyhow, something about `innumerable, singular sites of suffering' makes the point rather moot.
- nabla9 6y agojust is positive assessment about relative size. I could be wrong about positive statement of course. But it seems very low amount for a cloud service. I wast trying to understand how business thinking work. Not judge or praise it.
- Aeolun 6y agoBut we’d make more money!
- plutonorm 6y agoYour comment is basically saying capitalism is bad. I agree.
- microcolonel 6y agoThen turn off the computer and cancel your internet.
- perlgeek 6y agoIt's easy for this calculated risk to misfire, if the platform gets a reputation for losing data early in its life.
- Uberphallus 6y ago> In this case just 10TB of data was lost, so maybe thousands of users for the long term storage option. I'm not even a professional photographer and have about 2TiB of photographs. This is barely data loss in the great scheme of things, but certainly a wake up call for Canon and their users.
- jiofih 6y agoA “calculated risk” doesn’t make it okay when the risk is all on the consumer losing their memories. This corporate view on business risk alone is unethical.
- galoisgirl 6y agoThere is no such thing as lack of risk in this world. When a restaurant takes your reservation for Friday night, there's a 0.00001% (or so) risk that they will burn down on Thursday or that half the staff will get heart attacks or something. Should they not do reservations at all?
- xnyan 6y agoIf they don't have a working smoke alarm, or are storing flammable materials over the burner, or don't have enough emergency exits then no, they should not do reservations at all. It seems Canon did not have a backup, they had replication of data. If you are storing data for money, then this is a professionally negligent behavior. Not being able to prepare for everything is not an valid excuse to not prepare for what you reasonably can.
- CydeWeys 6y agoWhat does adherence to fire code have anything to do with taking reservations? Also, reservations are free. It's not reasonable to expect restaurants to do anything and everything to honor 100% of reservations when many customers never show up for them and don't have any skin in the game anyway. Whatever point is trying to be made here is going out the window thanks to the "zero 9" nature of restaurant reservations. (Yes, not even 90% of restaurant reservations end up turning into actual paying tables.)
- jiofih 6y agoIndeed, it has nothing to do with taking reservations, but providing service which is the crux of the issue. If they accept customers while knowingly not following fire regulations, then yes, it’s wrong and unethical; similar thinking should apply to software services.
- agumonkey 6y agoI wonder if they discuss that and actually write down calculations
- vidarh 6y agoI've designed a storage system for an image storage app for a client, and I had an extensive spreadsheet with the cost and risks of various combinations of erasure coding and data centres, as well as other mitigations. So yes. But this data loss highlights that all too often it ends there, with calculating the risk of loss due to hardware or data centre failures. You need to also do the same in terms assessing how you design your software, and what processes you put in place to mitigate risks. In this case it was the software system design that failed: They allowed the system access to modify long term stored data, when really the system ought to have been designed to allow "write once unless the user has convinced us that they really, really want the data destroyed, in which case keep a backup for a limited time" type design. A lot of software fails in that respect. The moment your internal API's allow modification of data, you risk catastrophic loss, as Canon found out, and you really ought to have strategies for mitigation of that.
- agumonkey 6y agoBut was this framed as easy business from gambling with users artefacts ? or just the usual "shit happens" preliminary thinking ?
- vidarh 6y agoI'd guess in this case nobody had thought about it as a real risk. People are often woefully unaware of the risks of accidental effects of unnecessary privileged.
- deepstack 6y agoIn this case just 10TB of data was lost I don't know about you, but something doesn't seems right here. Just 10 TB of data? For me even 0.5TB is too much. Hech, 0.0001 is too much.
- deepstack 6y agoHow hard can it be to have a set of data, make 2 copies and store the in different locations.
- vidarh 6y agoIf you're storing a large enough set of data for long enough, 2 copies will be woefully insufficient to ensure you don't suffer a loss of a given subset of the data.
- jefftk 6y agoReading their description of the problem, it wasn't that they lost one of their redundant servers, it was that their code that cleans up temporary storage accidentally was also cleaning up long-term storage. Increasing the redundancy of their storage wouldn't have helped.
- chongli 6y agoIf you have regular (daily, hourly) tape backups that are shipped off-site then you greatly mitigate the potential damage of software bugs. It sounds like they weren’t making backups and just treating failover replica servers as a “backup” (which they clearly aren’t).
- StillBored 6y agoThis sounds like a case of a bug in their software uncovered a huge gaping hole in their architecture. Put another way, while they seem to have fixed the immediate problem, no word on whether they are putting real BACKUP plans in place to avoid a similar problem happening and whipping out everything.
- louwrentius 6y agoThe famous Fort Pinto story comes to mind. The car with the fuel thank in a bad spot, making the car a 'fire trap'. A risk analysis made by Ford was misinterpreted and seen as a calculated risk of recall vs settlements for injuries/death. But It turns out it was mostly a hoax. https://en.wikipedia.org/wiki/Ford_Pinto#Retrospective_safety_analysis https://en.wikipedia.org/wiki/Ford_Pinto#Retrospective_safet...
- XorNot 6y agoIntent tends to matter with the law. Ford knew there was a problem, knew they could fix it, and elected not to because "it's a small number of deaths". You can replace things, not people. That retrospective analysis utterly misses the point: for the area they were responsible for, if the company knew they could do better and elected not to, then they got treated exactly as they should have.
- throwaway0a5e 6y ago>Intent tends to matter with the law. Ford knew there was a problem, knew they could fix it, and elected not to because "it's a small number of deaths". You can replace things, not people. Per the link posted by the GP the Pinto did not have any problems relative to other cars of the same type and era. Ford's "intent" (if you can call it that since they didn't know the future at the time) was to say "err, we don't really think there's a problem here so we'll pay out money when this happens but it's not recall worth" and the investigation bore that out which is why they were not found guilty. I don't really want to defend BigCo (or any other large group with diffuse responsibility) but it's 2020 and here I am. >That retrospective analysis utterly misses the point: for the area they were responsible for, if the company knew they could do better and elected not to, then they got treated exactly as they should have. Everyone could always do better on any given metric but the question is at what cost and with what tradeoffs. People buy compacts and subcompacts because they are cheap and fuel efficient. You could build a subcompact that just bounces when rear-ended by a freight train but nobody would buy that because it would cost a ton and weigh a ton (several tons actually) defeating the value proposition of a compact car.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]