10 ms·
Mail Data Loss in macOS 10.15
- saagarjha 7y agoIs there a way to detect if you have data loss? It sure looks like Mail has my emails, but I'm not sure…
- dv_dt 7y agoPerhaps if the mail database were blockchained or at least merkle tree linked or otherwise metadata secured.
- eli 7y agoOr just backed up.
- nitrogen 7y agoA hash tree for email actually seems like a semi-decent idea for verifying integrity of a synced mailbox.
- journalctl 7y agoSo use a hash function like in the olden days.
- nitrogen 7y agoThe benefit of either a tree or a sequence or hashes is that you could identify where and when the active mailbox differs from the backups. If each message is its own separate file then it's less interesting.
- plg 7y agogood LORD if this is true how are we supposed to trust apple with our information? BTW since updating my iPhone & iPad to iOS 13, and my desktop to MacOS 10.15, the Notes app does not propagate changes unless I force-quit the app and then restart it. Entering a new note, and quitting the app, the new note does not propagate to my other devices. If I re-open the Notes app then the update happens. WTF A walled garden ecosystem’s main raison d’etre is for everything to work together. If it doesn’t, it defeats the purpose. I’ve been an Apple advocate for a long time but things are getting dicey. Perhaps my macs will turn into defacto expensive chromebooks. In other words use them to run chrome and in turn gsuite everything, and nothing else. God awful
- brianprovost 7y agoI've also had issues with the iOS 13 on an iPhone X. Mail notifications appear but the messages don't download for 30+ seconds, and sometimes notifications don't disappear. Other people are also having issues with mail on iOS: https://twitter.com/CliffordAsness/status/1182407168286494726 https://twitter.com/CliffordAsness/status/118240716828649472...
- tyre 7y agoJust got an 11 Pro and some notifications don’t vibrate, other notifications no longer show up even though they have permissions to and are set to. Sometimes iMessages won’t vibrate either. It’s a mess and I can’t trust my phone to notify me. Now I have to constantly be checking it
- erichurkman 7y agoAdd in that alarms randomly don't fire, or fire at the wrong volume. I'm using my old phone as my clock and timer. I wonder if it's related to the bonkers notification settings. Either way, disaster of a phone, and disaster of an OS.
- journalctl 7y agoA walled garden’s raison d’être is to ensure that the garden’s owner makes as much money as possible.
- ggregoire 7y agoWhy do people use Mail? (genuine question)
- deleted 7y ago[deleted]
- Klonoar 7y agoLooks decent, doesn't hog obscene amounts of memory (for me - I know others have different experiences, and this can vary based on more factors than worth explaining). I also just use macOS/iOS Mail because FastMail can push notify to the clients, and FastMail's iOS app isn't ideal.
- geerlingguy 7y agoIt's very fast (native app), it supports practically every type of mail account ever, everything can be stored locally (great for offline batch processing of emails), and there are keyboard shortcuts for everything. It takes a bit of customization to get a layout that works nicely for my workflow (I use the classic layout). There are a bunch of other 'efficient' mail apps, but if you just need basic features and don't go for some exotic or trendy workflow, Mail works fine. I don't upgrade to the latest OS for at least a few weeks, though, because if something like this is systemic, it will be fixed in the .1 or .2 release. Also, always keep at least one local backup (I have Time Machine for that), and at least one remote backup.
- ganitarashid 7y agoI’ve never figured out how to force Apple Mail to download ALL attachments. How do I do this? In my experience it never downloads ALL of them and when you’re offline and click on a very old message it will not have the attachment unless you go online.
- njhaveri 7y agoYou can set this preference by going to the Mail menu > Preferences > Accounts. Select the account you want to adjust on the left, and in the first "Account Information" tab, there should be a pop-up for Download Attachments: [All, Recent, None]. The default for newly added accounts is "Recent".
- Alex3917 7y agoNot surprising. The new Podcasts app deleted years of downloaded podcasts when I updated, including a lot that are no longer hosted on the web. For some reason the "sensible default" in the new app is to delete anything beyond the most recent ten episodes. Now I need to remember to restore from backup before it gets overwritten, heh.
- derefr 7y ago> For some reason Giving you the other side of this: Podcasts was essentially a disk-space-leak made app, if you subscribed to any podcasts that you never actually listened to (but still listened to others, and therefore regularly opened the app.) It would just pile up new unplayed episodes forever, eventually taking up tens of GBs of space, on a device that doesn't have all that much space to spare.
- userbinator 7y agoI don't know if Apple has changed the format again, but in the past at least it seemed reasonably sane and filesystem-based: http://mike.laiosa.org/2009/03/01/emlx.html http://mike.laiosa.org/2009/03/01/emlx.html https://wiki.fileformat.com/email/emlx/ https://wiki.fileformat.com/email/emlx/ It doesn't seem that hard to write some more tools to manipulate them.
- rcarmo 7y agoI have had code to handle emlx for over 10 years: https://taoofmac.com/space/blog/2008/03/03/2211 https://taoofmac.com/space/blog/2008/03/03/2211 The thing here is that a) there are also a number of SQLite databases that are used by Mail to store metadata and sundry b) Mail's IMAP handling has been notoriously buggy at times My guess is that this particular data loss bug in Catalina stems from a combination of the IMAP implementation and the metadata storage, but since they keep playing whack-a-mole inside ~/Library/Mail, I can't offer more details.
- thelittleone 7y agoThe position of Apple today reminds me a lot of Microsoft after they peaked (~2000?). Cocky to the point of arrogance because of market position, past innovations and brand loyalty (e.g., it is cool to own Apple). But loyalty and market sentiment can only endure so much amidst plummeting product quality, nonsensical pricing (e.g. dongles) and lack of any meaningful innovation (thinner doesn't count). Apple doesn't listen because they think they know better (and that used to be true). - Peaked in terms of brand perception rather than market cap or market share.
- xattt 7y agoThe Mail app in Leopard had a bug that would spam recipients with copies of drafts as they were typed and saved automatically by the app. Not an apologist, but showstopper bugs were a thing even under Jobs.
- sroussey 7y agoYeah, I hit a bug where it deleted my contacts and then synced the deletions across devices. :(
- hesk 7y agoI ran into the same problem but was able to fix it with Time Machine. The contacts app has special support for Time Machine (you can slide through time and don't have to go to the Finder) and I could restore all my contacts from a few weeks ago.
- fastball 7y agoIt wasn't obvious at the time, but I think in retrospect the case can be made that Apple's ability to innovate effectively (both in terms of time and quality) died with Jobs. Jony Ive made a good effort to keep pushing hardware design forward, but without Jobs to curb his worst impulses (in my opinion), the end was result was "thinner above all else" (as you mention). If someone can come up with a corporate structure that allows for the dynamism provided by having a visionary dictator at the helm, without actually needing a visionary dictator, that would be a corporate structure that would take the world by storm.
- herpderperator 7y agoI always find it strange when OS updates affect apps in this way. It just shouldn't happen. It reminds me of when Linus preaches that "the kernel should never break userspace". Why would a OS update suddenly cause the Mail app -- developed by the same company as the OS, for what it's worth -- to have these strange (and catastrophic) data syncing issues? And, just to add, I am tired of OS updates updating and/or releasing userspace things like "Music" and saying: > iTunes forever changed the way people experienced music, movies, TV shows, and podcasts. It all changes again with three all-new, dedicated apps — Apple Music, Apple TV, and Apple Podcasts — each designed from the ground up to be the best way to enjoy entertainment on your Mac. [0] It is literally the FIRST thing they advertise. What does that have to do with the OS? Nothing! They should release apps separately from major OS versions. [0] https://www.apple.com/macos/catalina/ https://www.apple.com/macos/catalina/
- userbinator 7y agoThe new OS comes with new versions of apps, and that's usually fine because they're either fixed bugs or added features; the main problem is Apple's internal culture that seems to be preaching "change is good, constant change is always good" --- as I explained in this comment I made a few days ago, the love of constant massive internal changes to their code always tends to cause such bugs: https://news.ycombinator.com/item?id=21174947 https://news.ycombinator.com/item?id=21174947
- a012 7y agoIt makes sense now because they said bug is a feature. No surprise.
- yjftsjthsd-h 7y ago> I always find it strange when OS updates affect apps in this way. It just shouldn't happen. Right? Are there not automated tests for this stuff? I would expect that if you commit something that causes this, the next day you get an email that says, "your name is on commit $FOO, which caused regression tests to fail in tests/firstparty/apps/mail; please fix it (sincerely, Jenkins)" or so. Heck, it's more egregious here because it's a first party application, but it seems like it would be in their interest to keep copies of the most popular third-party applications and include those in their test suites as well. ("Hi, Adobe? Hey, it's Apple. Hey, so this is a heads up that Photoshop doesn't work with the new OS version; we're sending you a RCA so you can fix it before it hits end users.")
- reaperducer 7y agoI wish I only lost mail data in the upgrade to Catalina. I lost 400GB of data when the installer repartitioned the hard drive and the files that were in folders in root went bye bye. I know I'm not supposed to put stuff there, but it was a family video project I was putting together for my wife for Christmas, and she would have seen it anywhere else. There should have at least been a notice or a warning or some kind of confirmation. But there was nothing. Nothing in the "Relocated" folder, either.
- userbinator 7y agoIf you can reproduce this (I suggest using a VM, they're perfect for this sort of thing), it would be great to write it up as an article and get more visibility on it, because removing files that it did not put there is a really really serious bug. Microsoft suspended an update after reports of data loss, and was still severely chatised for doing it (explicitly deleting files it did not create without asking permission first): https://news.ycombinator.com/item?id=18189139 https://news.ycombinator.com/item?id=18189139 I'm a long-time Windows user, have put files in the roots of drives (maybe this is not such a common thing for Mac users to do, but nonetheless the OS should not assume...), and have never had any OS upgrade remove them. It would be interesting to see what, if anything, Apple will do in response.
- sys_64738 7y agoDid you backup before you upgraded?
- jbverschoor 7y agoDamn. Thanks for sharing. I hope you had something on a backup somewhere
- jumpingmice 7y agoThe Mojave installer just up and trashed my whole file system. No reason to believe it can’t happen again. Always backup.
- ghuntley 7y agoCheck /Users/Shared/Relocated\ Items/Security/
- segmondy 7y agoI have earned many thumbs down on the net for saying that Apple can't write software for shit. Apple is a design company, they make nice hardware, nice really looking hardware and sometimes performing hardware. That's it, a nice hardware design company. There software is straight up garbage and has been for a long time. Just as I have always been suspicious of those whose main OS is windows and they are not writing native windows app, I'm suspicious of those who use Apple if there full time job is not developing iOS apps. I do own a used Mac mini I bought a few years ago for the solo purpose of developing iOS apps. First time I ever upgraded it after being annoyed to death, it trashed the system and I had to do a fresh reinstall.
- blub 7y agoBlaming "the company" for bugs in complex human-machine systems doesn't really say anything insightful, actionable or useful. Android, Linux, Windows, macOS and iOS are all broken, just in different ways. This is the quality that can be reached for mass-market consumer products with the standard development processes.
- auggierose 7y agoYeah, good point. I am quite happy with Catalina so far so I have battled through a few issues. I think people basically have no idea what it takes to deliver perfectly functioning software. Basically, you need to combine user testing, design and theorem proving. It would be just too costly even for Apple.
- zaphirplane 7y agoYou know I thought apple’s innovation is a hype part of jobs reality distortion field. Then I compared iPhone and android the ux and form innovations start with Apple and are borrowed by android. Same for OS X and windows, specially the gestures and form it’s flowing from Apple.
- xvector 7y agoIMO Apple usually makes good software. That’s why the Catalina release is a big deal. If Apple made shit software this story would be background noise, but it is surprising to see a company that usually delivers relatively polished experiences do this.
- atmosx 7y agoIt’s a good idea to create a bootable backup and come up with a clean install of macOS with each major release. Every time I updated the existing system straight up, I had minor or major issues solved by a clean install.
- pcr910303 7y agoAs a daily Mac user who started to use macOS (OS X) about 7-8 years ago, I find the insurmountable number of bugs of macOS Catalina alarming. Don’t get me wrong, the situation is still much, much compared to Windows or Linux[0], buy it is alarming to see the general quality of macOS software to decrease. I personally think it’s due to Apple engineers trying to do so much; I remember being impressed by the number of new features and changes introduced this year while watching the WWDC, but it looks like Apple couldn’t keep up its own enthusiasm. Project Catalyst was poorly executed; I was disappointed that Catalyst doesn’t default to ‘HIG-respecting’ AppKit controls that correspond well with the UIKit equivalents (so that modal form sheets in UIKit are translated into floating windows, e.g.), and the inability to mix in AppKit controls and UIKit ones into the same view hierarchy (NSColorWell or NSPopUpButton doesn’t have UIKit equivalents), e.g. SwiftUI is great in theory, but the tooling is immature, it’s not available to write any decently complex apps with them. The iCloud changes were reverted right before Catalina’s release; looks like the iCloud team didn’t have enough manpower to fix all the bugs until the release. It’s concerning, as one of the premise of macOS was that it works as expected, though that now the newly introduced parts doesn’t. I’m expecting a stabilizing release for macOS 10.16 akin to the stone-stable 10.14 Mojave, but I don’t remember 10.13 High Sierra to be unstable like this one. [0]: For Linux users who don’t agree with this: I’ve never set up a Linux desktop system with all the hardware working without doing any ‘geek’ stuff like running some shell commands. And no, the fact that Apple only allows to use macOS only on its own computers isn’t an excuse. And no, even on well supported laptops like the ThinkPad, there are a furious amount of bugs that need ‘configuring’
- akkawwakka 7y agoThe company and its engineering teams are absolutely subjected to delivering too much. I think the prior perceived quality of Apple software was propped up by a lot of people internally grinding very hard on a far smaller set of products than the company ships today. That might work when you are at 10k people, not 120k. It’s really time for Apple to make a good faith effort to improve its software development process. Honestly it seems more like an executive leadership issue, incessant and hasty product changes come at the cost of stability.
- deergomoo 7y ago
- jquery 7y agoMy iCloud backup, with years of data, including apps which only had local storage, was completely borked when I upgraded to iOS 13 (on my iPhone XS). My phone would enter a boot loop during the iCloud restore process, and I tried 15 different restores (different iOS 13 versions including beta versions, iTunes backups, a couple different types of restores they have at the Genius Bar, a direct phone-to-phone restore, curating my iCloud backup with only the most important apps, etc). None of them worked, all ended in a boot loop. Apple Support basically gave up on me, telling me my "backup was corrupted". In the end I had to start with a fresh iOS install, which was very disappointing, as my years of accumulated app data were lost. I manually backed up what I could, but some apps had their data lost forever, and I wasted many hours of time. Very disappointing.
- forgotmypw3 7y agoI feel your pain. I hope you remember in the future: Unless a backup can be verified, it is not a backup. Backing up to a complex or binary format is trouble.
- jquery 7y agoSo true, but I’m not sure what I could have done here, except stay on iOS 12* indefinitely. This wasn’t a case of having an invalid backup per se (I was able to restore my phone back to iOS 12), this was a case of being unable to port forward data from iOS 12 to iOS 13. In other words, a perfectly working system could not be upgraded via Apple’s upgrade process, nor did Apple provide good workarounds for this scenario, such as allowing the restoration of individual apps. That’s why I’m extra disappointed in Apple. I did everything by the book, including testing my backups, and my data was still lost.
- forgotmypw3 7y agoI think what you can do now is avoid complex, binary, and/or proprietary formats for your data, because they are more difficult to back up and restore. You can also learn to not trust software which promiess you the world, but doesn't let you see its internals.
- quadrifoliate 7y agoA lot of this is a repeat of weird issues that we saw 8(!) years ago with OSX Lion. Perhaps I'm off the mark here, but I strongly suspect there's lack of a proper manual QA team at Apple. QA at most so-called "modern" companies these days has turned into overly fetishising automated testing. Automation is valuable, but crucially also leads to situations where the least-common-denominator situations get tested since it's easier to automate the conditions for testing them. So for example you might write an automated test that verifies that the mail index is correctly upgraded from (say) El Capitan to Mojave to Catalina; but that doesn't show the whole picture. There could be 20 other complex factors that affect it when it's used as part of an OS upgrade, and none of those get tested. Project managers, line managers, and other suits also seem to love automated testing, because you can show off some kind of real number to someone in an "Oh we wrote 200 tests this cycle, isn't it great that we can run them everyday" way, even if there are 0 issues found. Manual testing in the same case leaves you with no output that can be run in an automated fashion. Also, no one wants to argue against automation (even if it's taking away manual testing resources) because that can make you seem old-school and out of touch. And once this attitude gains steam, no one on the QA team will want to go against the grain and say "You know what, we should spend more time not writing automated tests". All of this is to say -- sometimes during testing, you simply have to put yourself in a user's place (for as many different kinds of user you can imagine) and actually use the thing you're supposed to be testing. I strongly suspect that Apple is not investing enough, or at all in this kind of testing. Instead, they are pushing it off to those enthusiasts who install and use beta or newly released versions of their software. So far, this strategy seems to have paid off for them, but perhaps people will wise up and stop testing their stuff for free. I am certainly not upgrading my machines to Catalina any time soon.
- emmelaich 7y agoNot so much automated testing but too much focus on unit and shallow integrations and not enough the big picture testing. Such as load testing, end-to-end testing, fuzzing. All of which can be automated to a large degree.
- quadrifoliate 7y ago
- newscracker 7y agoI’ve never used Mail because I’ve never felt comfortable with Apple’s apps when it comes to long term data storage. Here Mail is said to delete data. This has happened before, IIRC. Someone in this discussion has written that their old downloaded podcast episodes were deleted on upgrade. In the distant past, Apple has also broken compatibility in data formats (for its iWork apps) and left those who upgraded with almost nothing. Apple seems to focus too much on reinventing things, as if every app team gets a new (and inexperienced) set of developers each year and they see whatever is there as bad, incomprehensible code...so they write things from scratch, introducing tons of bugs and removing backward compatibility. Look at its app history and you’ll see a lot of work just thrown away every few years, only to be replaced with crippled applications that would take long to mature but aren’t allowed to. Mozilla Thunderbird is “boring software”, and hasn’t made huge strides in UX in the last several years, but I trust it more than I would ever trust Apple Mail. If I want “office software”, I have LibreOffice, which has its own issues in UX, but I can trust that it won’t drop backward compatibility at the drop of a hat. If I want a browser, there’s Firefox. Safari 13 has become just ok now (from decent) because of the new extensions system that has junked all old extensions. It’s fine for firing up an ephemeral window, and it still provides the best battery life. I feel the days of power users on the platform are numbered. I doubt if AppleScript and Automator will survive and be around in the next two years. Is there any Apple app on Mac that’s been great and really improved in a stable manner?
- Santosh83 7y ago> If I want a browser, there’s Firefox. Safari 13 has become just ok now (from decent) because of the new extensions system that has junked all old extensions. To be fair Firefox did that recently with their transition to WebExtensions too, and Chrome will soon do something similar when they move to Manifest v3. Sadly all mainstream browsers have become "move fast and break stuff", mainly in their desperate bid to keep pace with the de facto standard that is Chrome. But most of them don't have browser dev teams the size of Chrome's...
- newscracker 7y agoFirefox’s transition to WebExtensions has caused pain to developers and users, but there was a warning for some years about the transition. I still don’t like where the WebExtensions API support is, and Mozilla hasn’t done much for erstwhile popular extensions like Tab Mix Plus and many others. The case with Safari 13, IIRC, is that extensions have to be developed in Swift and have lesser access than before. While I get the privacy benefits with block lists that have been/are being/will be implemented in these browsers, the overall extensions scene doesn’t look very good. Hopefully Firefox will delay or not go with everything that Manifest V3 proposes, whenever that becomes standard in Chrome.
- elevenbits 7y agoIt's temping to treat this as more of the same "Catalina is buggy, Apple is doomed" news, but note that Mail.app has ALWAYS been less than rock solid. For some reason, it seems to be a prime target for new features and reconfiguration every release, resulting in a lot of churn which unsurprisingly brings about bugs like this. A cursory search indicates a bug about some messages not showing up in 10.8, a Gmail issue in 10.9, VIP issues in 10.11, etc. But hey, we have gratuitous, cannot-be-disabled window animations when replying to a message, so what's a little data loss compared to that?
- jbverschoor 7y ago> Moving messages between mailboxes, both via drag-and-drop and AppleScript, can result in a blank message (only headers) on the Mac. If the message was moved to a server mailbox, other devices see the message as deleted. And eventually this syncs back to the first Mac, where the message disappears as well. Actually this was already a problem on Mohave. I lost all my mail from one mailbox when moving messages from one folder to another (gmail)
- deleted 7y ago[deleted]
- techslave 7y agoyou people are going batshit crazy over completely unsubstantiated report. It even says, > I don’t know whether these are due to Mail bugs or to other factors such as problems on the Mac or with the mail server.