42 ms·
Canon's cloud platform has lost users' files and can't restore them
- klodolph 6y agoLesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost is usually due to software bugs, bad configurations, and operator errors. Design your system to protect against these things.
- _Microft 6y agoThis reminds me of this comment that was posted in the "medical data leak" discussion that is currently on the front page as well: https://news.ycombinator.com/item?id=24196041 https://news.ycombinator.com/item?id=24196041
- CalRobert 6y agoIf memory serves, a class of Glacier storage (AWS) isn't practical to use in the EU because the inability to delete makes it impossible to serve "Right to Erasure" GDPR requests.
- londons_explore 6y agoStore data and keys seperately. The set of keys for all your customers should be only a few megabytes, so is much much easier to back up more frequently and more copies of. It's also easier to have a "we do daily backups and delete any backups after 21 days" policy.
- lozenge 6y agoStoring names separately can help but the subset of data you have a legitimate legal reason to "refuse to forget" can change faster than you would want to rewrite your backups(well, nobody wants to rewrite backups, but you get what I mean)
- raverbashing 6y agoYou need to denormalize the non-forgettable information. This way you don't have to worry about what you can delete or not. Store Invoices, billing, etc, separately. These also usually have a fixed pre-determined retaining policy time. Past that time? Anonymize/aggregate or just delete it.
- soco 6y agoLegislation can change and what you once thought to be forgettable becomes suddenly not.
- aaronmdjones 6y agoIt'd be extremely unusual for such changes to be retrospective. For example, if you are currently required to keep certain kinds of financial records for 3 years, and you delete records older than that, you won't be held at fault when that law gets updated to say 5 years, and you then fail to produce a 4-year-old document; it has not yet been a year since the new law took effect. This extends to criminal law too, for example with statutes of limitation. If, 20 years and 1 day ago, there was a 20-year statute of limitations for murder charges, you cannot be charged for committing a murder on that day (barring extraneous circumstances, such as tolling); even if a law was passed the day after said murder removing this limitation.
- soco 6y agoWell, yes, but the topic here was data. And if at some time you thought it was forgettable and you scheduled it for deletion, once the law changes you should make damn sure all code involved in that deletion was checked or the data will be gone - against the new law. A big opportunity for human error here.
- syrgian 6y agoWouldn't you need to edit every single backup where the key is stored in order to comply?
- dodobirdlord 6y agoI assume they mean the standard practice of encrypting your backups with per-customer encryption keys. If you ever want to destroy all of the data for a particular customer you don't have to do any editing of backups, you just have to nuke every copy of the encryption key you use for that customer.
- Hamuko 6y agoNot necessarily. While it's not set in stone, GDPR right to erasure might not be in conflict with backup retention. >The GDPR is open to interpretation, so we asked an EU Member State supervisory authority (CNIL in France) for clarification. CNIL confirmed that you’ll have one month to answer to a removal request, and that you don’t need to delete a backup set in order to remove an individual from it. Organizations will have to clearly explain to the data subject (using clear and plain language) that his or her personal data has been removed from production systems, but a backup copy may remain, but will expire after a certain amount of time (indicate the retention time in your communication with the data subject). Backups should only be used for restoring a technical environment, and data subject personal data should not be processed again after restore (and deleted again). While this adds some complexity, it allows organizations to have some time to re-engineer their data protection processes. https://blog.quantum.com/2018/01/26/backup-administrators-the-1-advice-to-deal-with-gdpr-and-the-right-of-erasure/ https://blog.quantum.com/2018/01/26/backup-administrators-th...
- johnchristopher 6y agoAs I understood it at the time this is the difference between archives and backup.
- eru 6y agoI suspect you can square the circle, if you encrypt all the files, and store the keys in storage you can delete quickly.
- CalRobert 6y agoI see memory served me incorrectly! Apologies if I have spread misinformation. It was an offhand comment from an AWS rep a few years ago and things may have changed.
- klodolph 6y agoWhen GDPR passed there was a bit of a scramble to understand it, and people had all sorts of different interpretations that they were sharing.
- Sharlin 6y agoTo be fair, image.canon is definitely not a cold-storage or backup solution, never mind primary storage. Its main use case is acting as a buffer for auto-uploading photos from camera and then later downloading them to a computer. As an afterthought it also offers a tiny long-term storage space, 10 GB per user, smaller than most memory cards available today.
- SifJar 6y agoIt also appears that only images stored for >30 days were affected, so in theory most photos "should" have been downloaded and stored properly etc. if people are using the service as you describe
- stingraycharles 6y agoYeah, it seems some of the other commenters miss that point: it’s not as if some disk crashed or something, it’s a bug in a software upgrade that deleted long term storage in addition to short term storage. In my experience, the best way to mitigate these kind of things is to use PITR, which is always immutable. You make periodic snapshots, keep track of all changes since those snapshots. And then you use a policy to determine just how much data you want to keep here. The problem is that not supporting delete operations is often very much impractical. Consider for example the recent heat Flickr (or was it Instagram?) has gotten for not physically deleting files but just marking them as deleted/invisible. You sometimes have really good reasons that you must support delete operations, so you must just make sure that there’s a reasonable delay before they are actually completely gone.
- tonyedgecombe 6y agoIn case anybody else wonders what PITR is: https://en.wikipedia.org/wiki/Point-in-time_recovery https://en.wikipedia.org/wiki/Point-in-time_recovery
- mschuster91 6y agoThe problem with that is that under GDPR you have to support full deletion of an user's data.
- Hamuko 6y agoNot necessarily if you have a valid reason to storing the data. You might for example have to keep user's billing data around for legal reasons.
- NullPrefix 6y agoWas this billing data though?
- toong 6y agoStore customer data immutable and encrypted, with a unique AES encryption key per customer. Encrypt that AES key again with a single RSA key-pair and store the encrypted AES-key in a database. You can access your customer data, using the customer-specific AES key. You can access the customer-specific AES key using your private RSA key. When you need to delete the customer data under GDPR, you can delete the encrypted AES key for that customer from your database.
- chmod775 6y ago> Store customer data immutable and encrypted, with a unique AES encryption key per customer. Encrypt that AES key again with a single RSA key-pair and store the encrypted AES-key in a database. Now you have the worst of both worlds. You also now have 2 points of failure where data can get lost, because if either has a problem you lose data.
- Ensorceled 6y agoI can't think of a mechanism where you can lose the customer's database records but can still recover files associated with that customer ...
- mixedbit 6y agoBoth AWS S3 and Google cloud storage buckets have an option that makes it impossible to delete stored objects for some period of time. The option was added for some legal compliance reasons, but I find it useful as an extra safeguard that important service data is not accidentally or maliciously deleted.
- ramraj07 6y agoYou could always not pay the bill :)
- chrismatheson 6y agoI suppose that feature would be incompatible with user data in a GDPR world ??
- visarga 6y agoGit is also incompatible with GDPR, you can't simply delete a file from all history.
- nix23 6y agoOh man, all my WORM tapes are not GDPR compatible...thats terrible!!!
- icebraining 6y agoYou can, you just have to be willing to rewrite history. See git-filter-branch (and don't forget to read the fat WARNING).
- msh 6y agoGit is not really intended for personal data
- Hamuko 6y agoYeah, I don't really understand why you would like to stuff personal data into a git repository (unless we're strictly speaking about author data). It's really not the tool for it. Now for those who decided that a blockchain is the perfect solution for storing personal data however...
- lmilcin 6y agoThis is actually larger lesson. When I work on design I try to make catastrophic failure impossible. Not implementing deletion or implementing deletion with some unseparable constraint check is one way. For example, the files could have a flag set and from that point on the only delete function would check the flag and refuse to remove the image regardless of how it got invoked or where the image is placed (think like write protect notch on a floppy disk). Another way is to not actually remove the files but instead introduce grace period when they are marked as deleted, not accessible, to the user. The only real deletion would happen in a vacuuming system that will actually remove the files but with a hardcoded constraint next to filesystem delet that the file must have been marked as removed at least XX days ago. This would give some time after deletion to actually recover files. Yet another technique is to move objects marked as deleted to some other, cheaper form of storage. The function that moves objects must first confirm the object is available in new storage through its API and only then removes the original. You can also prevent damage that could result from failed modification using immutable objects and Copy on Write. The copies would be deleted using the above described mechanism. Yet another technique is to keep XX days of redo-log that could contain enough information to rebuild any object to any of its current or past state within XX days. Depending on the type of the file and nature of changes it can be much more efficient than keeping whole copies. All of this of course adds overhead but you can recover a lot of that overhead by relaxing requirements on duplication of your operational or other forms of backup storage with rationale that if you have a redo log where you can locate current or past version of the object independently of operational or backup storage you can count it as one more operational or backup copy.
- IshKebab 6y agoGood luck complying with the GDPA.
- klodolph 6y agoDo you mean GDPR? You don’t have to delete data immediately under the right to be forgotten.
- hxegon 6y agowow this is bad. I know you should keep local backups too but if you shoot RAW then that quickly becomes cumbersome, and only using cloud backup looks very convenient. If I were a professional photographer missing client material or someone who lost potentially irreplaceable memories I'd want a hell of a lot more then just "[Cannon's] deepest apologies".
- _Microft 6y agoThe software and service might have been ... provided "as is", without warranty of any kind, expressed or implied, including but not limited to...
- londons_explore 6y agoI'd personally prefer services to warrant a certain number of $$$'s in case of service outage, a certain number for data loss, and another number for data leaks. I'd then like then to have insurance to ensure they are able to pay out rather than go bankrupt. They can then proudly write those $$$ numbers on the feature list, and I'll use it to decide which service to buy.
- DaiPlusPlus 6y ago> provided "as is", without warranty of any kind, expressed or implied, including but not limited to... IANAL, but when a service is advertised as being ideal for storing your work or photos, isn't that directly implying that there is a warranty that their service is fit-for-purpose? You can't advertise something in big lettering and then countermand that in the small-print - so those magic words certainly don't shield the company from liability at all. I understand that the "without warranty of any kind, expressed or implied"-line we see in software-licenses and EULAs is when software is distributed without consideration (e.g. open-source software), but when there is consideration (i.e. people paying Canon to host their photos...) then there's a liability if Canon lost peoples' data - so I understand they will be sued for this if anyone lost anything of value. At the very least a 100% refund... They only way I can see Canon getting out of this is if they had prominent warnings displayed throughout their service's UX advising their users that their service was not suitable for long-term storage of valuable data. Again, IANAL - can anyone is is a lawyer chip-in?
- radicaldreamer 6y agoThis is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the service from a technical perspective to be resilient enough to not crash and burn.
- cbg0 6y agoMore likely than not some manager decided all those extra nines weren't really necessary.
- nikanj 6y agoThe manager probably put five 9s onto the spec sheet, and provided absolutely no resources. You’ll never guess who’s going to get the heat for this, the manager or the poor person who didn’t follow the spec.
- deleted 6y ago[deleted]
- jiofih 6y agoI don’t know why you’re being downvoted since you can clearly see echoes of this manager in the comments above...
- dheera 6y agoToo many managers, not enough engineers on HN. As an engineer I can definitely attest to managers saying "99.9% is enough"
- SturgeonsLaw 6y agoI take nightly images of my computer's primary drive and replicate them offsite, with alerting if it fails and automated scheduled backup restore testing. It blows my mind to think that some random schmuck like me has better backup procedures than these multibillion dollar global corporations. Companies that don't invest in their IT end up paying for it tenfold down the track.
- kristianov 6y agoWell who can blame them. The files are not theirs. /s
- Dylan16807 6y agoBut insufficient redundancy for that much data saves millions of dollars!
- newyorker2 6y ago...which can instead be pocketed by higher ups in the company food chain :)
- dannyw 6y agoOn topic topic, can I get HN’s advice on my backup strategy? All important files on Synology Ds218+ NAS, which has 2x12TB helium HDDs in mirror. Daily HyperBackup to Google Drive. I test restoring files on a monthly basis. Email alerts on failure. Anything I can improve?
- MrGilbert 6y agoLooks like a sweet-spot to me.
- ianlevesque 6y agoYou’re way ahead of the average. The only improvement I can think of would be to server side copy the backup files to another Google Drive account that the account linked to HyperBackup can not access. That way if the Synology gets hacked it cannot delete the offsite backup too.
- deleted 6y ago[deleted]
- ponker 6y agoThe camera companies are so bad at software that the software companies are building better cameras than them... in the few millimeters thickness of a phone! I salivate for the day that Google or Apple buys Nikon or Leica or whatever and shoves all of that computational photography goodness under a 24x36mm camera sensor.
- DaiPlusPlus 6y agoAs of 2019, Apple sources their iPhone camera sensor components from Sony and the camera lenses from Largan Precision (and other companies). The software that Apple contributes is important - yes (after-all, this is software-defined-photography now, with Portrait Mode and that fancy but fake depth-of-field stuff in the single-lens iPhones) - but let's not pretend Apple is "a camera company".
- richardwhiuk 6y agoI expect Apple sold more camera than Canon last year.
- ehnto 6y agoI'm not sure what the "software" companies would gain from buying Canon or Leica that they don't already have. That said, you're also dismissing one of the biggest value propositions camera companies have, which is decades of physical user interface experience and developed hardware modularity. Phones absolutely do not replace good cameras for usability. When travelling I take a mix of photos between my phone and camera, and when I need my camera it's irreplaceable, and when I don't need my camera, I use my phone. The phone is quick and easy, but the camera makes me a photo ninja and I get specifically configured shots quickly in a way I could never replicate with a phone interface.
- ponker 6y agoI think you're replying to the wrong comment. The software companies (Google and Apple) would buy Nikon and release a device that looked and operated like a Nikon FM3HP camera with a 24x36mm sensor, interchangeable lenses, and all of Apple's image processing tech onboard. You take the photo and in 2 seconds the image is in your iPhone camera roll. You get the picture-taking optics and ergonomics of the Nikon, the photo-processing AI magic from Apple/Google, and the picture-sharing immediacy of SMS/Instagram.
- saagarjha 6y agoHas GDPR compliance gone too far?
- danieka 6y agoNo. Losing data is just as bad from a GDPR perspective.
- saagarjha 6y agoOh, really? Can you actually be fined for destroying disks containing customer information?
- pilsetnieks 6y agoYes, deletion also counts as data processing. Unauthorized data processing in this case.
- mschuster91 6y agoYes! Accidental loss is a data breach, see page 7ff of http://ec.europa.eu/newsroom/document.cfm?doc_id=47741 http://ec.europa.eu/newsroom/document.cfm?doc_id=47741
- saagarjha 6y agoInteresting, they seem to call this an “availability breach”; of course, they require disclosure if it affects people. Do data storage providers need to essentially SLA, then?
- tonyedgecombe 6y agoThe article doesn't mention GDPR.
- saagarjha 6y agoI know it doesn’t.
- TheButlerian 6y agoCloud is a joke. Everyone with half a brain knows it...
- adaml_623 6y agoBrilliant username. But we're not reddit so content over style please :-)
- headmelted 6y agoOn-site and at least two off-site. It sucks for their users who maybe don’t know the golden rule above. I certainly don’t blame non-tech folk who paid a known photography company to handle a complex problem on their behalf - they did the right thing in handing this responsibility to people who they believed could be trusted. To the greater point - this is incredibly damning of Canon’s cloud storage going forward. As others have said, the amount of data lost overall is not much in the greater scheme of things, but that’s not the concern. What’s worrying is that they were able to lose any data at all. How much redundancy do they have? How are permissions managed? When I backup photos and videos at home I have a script to chattr -i all the files independently as they’re stored on top of the redundancy and backups. You need to protect your data from yourself, too.
- henvic 6y agoEveryone should be looking for a backup strategy that involves at least a hot backup and a cold backup, with some sort of offline validation. However this is easier said than done (because now you've to deal with multiple file systems, decisions about cryptography, lack of interest in creating something that you hope you'll never need, and so on). I wish there was a service (and don't tell me about Backblaze because it's incomplete and doesn't work that well - from my own experience) that would do all these things for you in a reliable and trustworthy way (including giving you snapshots in physical media from time to time - delivering to different addresses, possibly in different jurisdictions, and so on).
- ChrisMarshallNY 6y agoI have a multi-dimensional backup strategy, which includes “cloudy” backups of my most important assets, using services like GitHub, local hourly NAS (Synology double-redundant RAID-like) incremental backups of my whole system, and running “hot” backups of my system and development volumes; automatically updated every four hours (or on-demand. It's a bootable CCC clone). I seldom need to use anything more than my “hot” backups, but have occasionally needed to restore individual files from NAS. Whether or not I have a backup is never even something I think about at all, which is a big weight off my shoulders. But photos and videos are a different matter from code (my main assets). They require a lot of space. It’s easy for me to be smug about backups; but photographers have a much more intense set of assets.
- ghaff 6y agoA problem with digital media as well is that it encourages you to just save everything but it's easier that way. But as I'm discovering well into the digital era, I wish I had more curated photos and video--though that comes with its own problems.
- fokinsean 6y agoI dread the day I decide to go through all my photos and organize them. It's one reason why I'm so reluctant to move off of google photos, it's sooo easy to find stuff and it groups photos together nicely. But I really want to move my photos off of google.
- pabo 6y agoI can't access the article, but there was a recent ransomware attack on Canon [1]. I wonder if this could be related to it. [1] https://news.ycombinator.com/item?id=24185734 https://news.ycombinator.com/item?id=24185734
- waihtis 6y agoI think they claim that this was a coding error and not a ransomware case. The timing is convenient though.
- hellofunk 6y agoThere is one somewhat reassuring lesson here -- No matter the company, no matter the brand, no matter their worth: software development is wrought with challenges, and major mistakes for seemingly obvious things are made by everyone, all the time, at every level.
- hvidgaard 6y agoThis is not one of them. Ensuring proper data loss protection is something both Amazon and Google are very good at. A company of Canons size could easily get the proper guidance to avoid it. But it would not have been as cheap as storing the files on your own servers. So this happened because Canon did not want to spend the money on doing it right with offline backups or append only storage in a cloud with enough nines.
- qntmfred 6y agoCompanies of Canon's size can be constrained for resources just as much as any other company. Canon is largely a legacy company at this point, with significantly declining revenue the last several years. https://i.redd.it/lrlfen2x7sm31.jpg https://i.redd.it/lrlfen2x7sm31.jpg
- hvidgaard 6y agoEven more reason to spend a little bit more to make this right. It's not expensive to get a proper expert on the matter to figure things out. The last thing they want is bad publicity for things that could earn them recurring revenue.
- hellofunk 6y agoI don’t think they deliberately thought they were doing something inadequately, these problems are always obvious in hindsight. Issue is that eventually most companies will have this moment seeing through the lens of hindsight, the reasons are always different.
- hendry 6y agoI was wondering what those weird messages from https://image.canon/ https://image.canon/ were about. IIUC the idea of their new platform was to be ephemeral in any case. Just to give a pipeline to stream in to other accounts. I had no idea it was offering long term storage. Better off with S3 Glacier for that!
- Aeolun 6y agoNo backups? Wut?
- hadrien01 6y agoI'm redirected into a refresh loop with this url: https://www.digitalcameraworld.com/cc.html https://www.digitalcameraworld.com/cc.html
- obayesshelton 6y agoOuch, I am guessing they are using some on-premise or none major cloud provider? I feel for them though, without the right resource or budget they are doing their best. I doubt they intentionally decided to not make a good product.
- m0zg 6y agoAnyone who relies on any software camera companies produce deserves to lose their data IMO. Just look at Canon DPP. Almost 20 years old, and it's still a primitive, unusable, steaming pile of shit.
- deleted 6y ago[deleted]
- KingOfCoders 6y agoCanon is no software company and doesn't have the right mindset (also see most other camera manufacturers mobile app reviews). Probably outsourced with a focus on having low development and operating costs.
- ReptileMan 6y agoWith spinning rust so cheap nowadays - anything could be backed up easily both in the cloud and locally.
- dsabanin 6y agoFrom time to time, I have a perisistent feeling that iCloud might be misplacing my files. It's just a feeling, but I realized that it's quite challenging to be sure it's not happening.
- bitexploder 6y agoIt’s not so hard. A few command line commands on a Mac. Store the md5 of each file, throw it into a JSON file. Load it all up in a dict. I do something similar every month. Gotta do something with those cores overnight :) It’s very brute force and inelegant, but also simple and quick enough.
- dsabanin 6y agoOne day, I'll do it. Honestly, I'm just afraid to find out the answer… :)
- jarym 6y agoSo today I learnt: 1. Canon don’t have any backup procedures in place for their cloud platform. Any hacker now will be salivating at the idea of pulling a ransomware hack I imagine. 2. Canon developers follow the ‘test in production’ methodology of continuous integration.
- galoisgirl 6y agoHow do you know that? Gitlab had 5 backups and could restore none.
- strombofulous 6y agoIs it a backup if it can't be used like one?
- galoisgirl 6y agoIt is and is not at the same time until you try to restore it.
- jarym 6y agoQuantum mechanics goes over my head
- vsareto 6y ago>Canon developers follow the ‘test in production’ methodology of continuous integration. Assuming they are making those calls
- mad182 6y agoThere is no cloud, it's just someone else's computer.
- fredthomsen 6y agoThe cloud is just another backup option and you should have 3. That being said, you expect more from a paid service.
- Funes- 6y agoSaving your shit on "the cloud": not even once.
- canofbars 6y agoI store my data on the nextcloud server sitting in the same room. I guess for added safety I could also set up automated encrypted backups to backblaze or something.
- bluedino 6y agoI think people would be surprised at the amount of backup or file storage services that keep everything in one data center. Your files might be redundant across machines but it's not like they have a second data center somewhere with another copy of all your data, and they certainly don't backup everything up to tape or something.
- zxcvbn4038 6y agoThis sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually the one thing they should have backed up because all of the smaller files can be regenerated from the raw ones. I would not be surprised if all of this was running under someone’s desk.
- ibeckermayer 6y agoWhat’s the difference between replication and backups? Is the distinction that backups must be stored on separate infrastructure, whereas replication might still be 1 or 2 points of failure?
- lukevp 6y agoReplication is about having data level redundancy to protect from drive failure. Backups are about having point in time snapshots of the system state, and about having them tiered from a location perspective. The 3-2-1[1] principle says to have 3 total copies, 2 of which are local but on different devices, and 1 of which is offsite. This gives you tiers of recoverability. It’s important from a backup perspective that it’s point in time as well, otherwise as soon as you get ransomware that encrypts your file you now have replicated those changes everywhere. [1] https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
- mumblemumble 6y agoI would add to that, that the offsite one should also be offline.
- ashtonkem 6y agoOr at least in an append only data store.
- dharma1 6y agoJapanese camera companies are really bad at cloud software - Sony Playmemories probably the worst offender. Also the in-camera "mini-apps" tend to be terrible. They make good hardware but horrible software. They should really buy a computational photography company that also knows how to make good web and native apps, like https://skylum.com/ https://skylum.com/
- hughes 6y agoI remember paying $10 to buy a mini-app that added what should have been native functionality to my Sony camera. Then a few months later a firmware update made it incompatible.
- irrational 6y agoWe were actually on NTT (Nipon Telegraph and Telephone) cloud for many many years up until very recently. At the time we started using them AWS, Azure, Google Cloud, etc. did not exist. But over the past few years the quality of service has dropped considerably. My guess is that their best engineers were being poached by AWS, Azure, etc. and the leftovers couldn't keep it up.
- robotmay 6y agoI'd go further and say that a lot of software coming out of Japan is, if not of a poorer quality, then at least something to be assessed more carefully. This is particularly evident in videogames, where anything for PC is liable to be extremely weirdly built and feels like it was made for Windows XP. There's a strong tendency to use archaic cheat prevention tools like nProtect GameGuard, which is a rootkit, and a weird reluctance to run servers in sensible ways. Oddly this is all actually helpful when you want to run their games on Linux because even though they don't seem to know it exists, the games are so "old" that they're pretty easy to get running. Sony is actually probably one of the better companies for software. But they still do utterly weird nonsense that I'd never even think of. The PS4 throttles your download speed based on your ping to the download server. It's...totally wrong and confusing. But easily bypassed by running a proxy on your local network.
- jaylittle 6y agoI'm utterly shocked at how many people assume that "The Cloud" is both run by people infinitely more intelligent and more competent than they are and that everything there is backed up. What a crock of shit.
- irrational 6y ago"We will contact affected users shortly and offer..." I'm not sure how I expected this sentence to end, but it certainly wasn't anything as useless as "our deepest apologies".
- williesleg 6y agoFake news. Check out the correction. Needfuls doing the needful.
- moreorless 6y agoPeople call me crazy for having local backups, cloud backups, and offline backups for my critical assets.
- system2 6y ago3-2-1 backup is not crazy and also is an IT standard.
- rasmore 6y agoThey’re paying for the negligence against software developers. Those two incidents come down to the design flaws after all. The CEOs of Canon Inc and Canon USA are too old to admit the fact that software is more powerful than what it used to be when the hardware was selling well simply owing to its pure performance and features and the software was just a driver. You would be surprised if I tell you how many skilled software engineers have left the companies in the last three years. What makes things worse is that the old men aren’t aware of why they left.
- deleted 6y ago[deleted]
- inertially 6y agoThis is why we need basic certification and licensing for engineering software systems -- the same standards we have for building real-life buildings and structures. This was probably contracted out to the lowest bidder on some freelancing website.