13 ms·

IOS 5's "Cleaning" Behavior

568 points by JeffDClark 15y ago

hernan7 15y ago
I wouldn't say that the articles' metadata (URL, title, date of download, maybe thumbnail of the 1st page) is downloaded content. It's clearly user-generated, and should be in the "home" of the app IMHO. The articles themselves, yes, send them to the cache. If the user needs to reclaim the storage used up by the articles, let the OS delete them. Then, when the user needs to read the article again, it will take some time to download. But don't get into an "all articles gone" situation. Just my 2 cents.
- masklinn 15y ago
  I'd submit that this is, in some ways, worse: instead of all articles being gone they're now all grayed out, so the user knows they were here, can still see there, but if he's offline he can not access them because only the metadata is left, the data itself is gone.
- scdc 15y ago
  I thought Instapaper scraped the page and stored that, to accommodate sites that require login for access. So not trivial to just re-download the articles. Plus the whole point is to have them available when you're offline. I'd say in this case, the content is user-generated and should be backed up.
  - masklinn 15y ago
    That's a tough line to argue: the scraped page is actually stored by Instapaper (otherwise you could not access it from the website), so you re-download it from there, and technically it's not user-generated it's at best user-curated. But this does indeed defeat the point of having it offline, which is a significant part of IP's purpose.
- deleted 15y ago
  [deleted]
- luchak 15y ago
  What problem are you trying to fix? If I'm offline, what good is being able to see my article list if I can't read the articles? If I'm online, why do I feel any less irritated when I try to go back to an article I was just reading only to find that I need to wait for it to download again?
  - adgar 15y ago
    The problem fixed is that the user has an indication as to what happened to all of their saved articles. That's a very big difference. It's not a full solution, as the parent noted, but the UX is far superior.
theatrus2 15y ago
Has anyone figured out where the high water mark is? When will the cleaning behavior kick in? This is an interesting twist especially with the 16GB devices which tend to actually be above 50%.
deleted 15y ago
[deleted]
oscardelben 15y ago
If the device is low on space I think it makes sense for Apple to purge cached data and tmp directories. For an app like Instapaper which relies on local documents, some of the documents should (imho) be saved to the home directory instead. And yes, I know this is hard but I'm just thinking aloud.
wmf 15y ago
Clearly, apps that cache data need to be modified to show the difference between having no data and having no data cached. IMAP clients have dealt with this, for example; they show a message like "the contents of this folder are not available offline". Perhaps Apple could have made cache cleaning opt-in on a per-app basis until iOS 6, though.
- smackfu 15y ago
  The main issue is that iOS is deciding that you don't need something offline without asking you. I don't want my cached email to turn out to have been deleted because I downloaded something else entirely. Many people would prefer the old method, where it would say you were out of space, and you would have to go free up some space.
  - politician 15y ago
    That's right; the solution is to make it easier to see which apps are using how much space and then provide a "clear" button. My Nintendo Wii uses a "blocks" concept for this. It works fine, Apple should just copy it.
jmcnevin 15y ago
Is it possible that a user could use Instapaper to save a document that they wouldn't be able to download later, even if they had access to an internet connection? If that's possible, I think Instapaper would have every right to store things in the Documents folder, since you're asking it to create something more akin to an archive than a temporary cache of data.
- sp332 15y ago
  I could be wrong, but I think all the articles are on Instapaper's servers, so you can re-download them.
  - lukeredpath 15y ago
    But only if you are online, which defeats the object of "offline access".
    - sp332 15y ago
      That's not what jmcnevin asked.
    - sp332 15y ago
      Oh, I see what you mean now. I feel dumb :)
- baddox 15y ago
  That seems to be the only way for Instapaper to give their users a reasonable experience, but it's still a bad solution. It doesn't make much sense for the Instapaper app to backup its users' cached articles to iCloud, when the whole point of the app is to backup the articles on the iPhone. If you have access to iCloud, you have access to the original article, so it's a waste of bandwidth. Not to mention that Instapaper risks getting hounded by Apple for the iCloud usage.
  - lukeredpath 15y ago
    I honestly don't think there is an issue with allowing Instapaper content to get backed up to iCloud here. The analogy of a user created archive is a good one. If people don't want the benefit of having their Instapaper content transfers across their devices automatically (and I do think this is a good feature) then they are free to disable iCloud backup for Instapaper in Settings.
sadfasdfads 15y ago
This highlights the fact that Apple doesn't have the resources to be able to work through issues like this with developers of well-used apps before changes are made. Maybe they should open more positions like these: http://apple-careers.com/job/Austin-Mac-2FiOS-Developer-Support-Representative-Job-TX-73301/1477606/ http://apple-careers.com/job/Austin-Mac-2FiOS-Developer-Supp... Sending an email out is insufficient. I've had similar experiences with a company that develops a product that I've written extensions for. If your product relies on an ecosystem, and you do something that is going to disturb that ecosystem, you'd better be prepared for the backlash.
JeffDClark 15y ago
The whole idea of an app like Instapaper (or any of the other examples presented) is that the "stuff" that is saved for later is all user-generated content. Some articles may even vanish (different location, move behind a paywall, deleted, etc...). In this case those articles would become inaccessible when the OS deletes the cache. It seems that the argument that only the list of metadata is user-generated can apply to any type of media (music, movies, etc...). Technically speaking all of the music on my phone could be downloaded on demand. Of course this requires an always connected, fat, and cheap network connection. Which is pretty much the opposite of what most folks have. This also breaks the user's expectations. I was annoyed/surprised when I upgraded to iOS 5 and Instapaper had to re-download everything.
- masklinn 15y ago
  > Some articles may even vanish (different location, move behind a paywall, deleted, etc...). In this case those articles would become inaccessible when the OS deletes the cache. As far as I know, all scraped articles are stored in Instapaper's server and downloaded from there, you don't get them directly from the source page.
  - JeffDClark 15y ago
    I have had articles in my list that have become unavailable. No amount of trying to re-download them works. What you say makes sense but I am unfamiliar with the exact way that it works.
ender7 15y ago
Apple's in a tight spot here - either they leave things as they are and piss off developers (and, by fiat, piss off users), or they have to start forcing users to more proactively manage their remaining space. One can probably easily imagine an interface for showing the user how much data a particular app is using and allow them to nuke the temporary stuff. It might even look beautiful. It might even be fun to use. But it's going to introduce a lot of hand-wringing and micro-managing and lots of mental overhead that Apple really, really wants to avoid.
- thehigherlife 15y ago
  iOS 5 now lets you see per app space usage. (it's under Settings > General > Usage) I don't see anyway to delete that temp data, however, only the app itself (for non-apple apps).
  - mrcharles 15y ago
    That's only the data that will be backed up to iCloud though, so it probably doesn't include cache/tmp.
    - 77ko 15y ago
      It does
    - freedompeace 15y ago
      It does - my Downloads app is reported as using 10 GB.
- deleted 15y ago
  [deleted]
- sudoman 15y ago
  This general problem is a symptom of a major issue regarding the use of software that does not respect the freedom of its users. Every non-apple developer and user is required to bend to the will of the programmers who develop the proprietary iOS. Any negative choices or limitations imposed by Apple Inc. are virtually uncircumventable. On the other hand, if users and developers in a community are free to view, modify and share the source code of the operating system and programs they are using, then that creates a dynamic where software cannot remain defective by design, something which cannot be said about Apple's hardware or software. If we were talking about free (as in speech) software, users would have control of their data, and their choice of how to manage it; not some distant programmers working for a corporation's bottom line.
pohl 15y ago
I'm sympathetic to Marco's blog post here, but I wonder if he's failing to interpret the two paragraphs from the documentation within the context of Instapaper's purpose. With respect to point #1, and in the context of what Instapaper is for, the user is intending to read an article offline, and the local copy of the article was generated by the user's intent (and can, therefore, be considered user-generated even though the user is not the author of the article.) With respect to point #2, the articles fail the "can be downloaded again" test in light of the app's purpose of making the articles available to the user when the network is not available. When the network is unavailable, the articles cannot be downloaded again. Edit: JeffDClark makes another excellent point here that the article may have originally been behind a paywall and therefore cannot be guaranteed to be re-downloadable. The same is true if the author removed the original article from their webserver. Ergo, put them in the Documents folder. Whether this would satisfy the app reviewer is another question, but it's worth a shot to carefully explain to them how your app is not violating the letter of the law.
- rcthompson 15y ago
  But it doesn't make sense to back this data up to iCloud, which is what happens to the Documents folder.
  - pohl 15y ago
    Really? If I generate an offline copy of an article on my phone, wouldn't I want to have access to it on my iPad? Why wouldn't I want the truth to be in the cloud?
    - Mavrik 15y ago
      Because there's no point in burning your cloud storage, Apple bandwidth and prolong backup times when those articles can be easly re-downloaded from Instapapers servers on iPad. Same result, without long backup times, Apple server bandwidth usage and your iCloud storage usage.
      - 15y ago
j_baker 15y ago
What about apps like spotify and rdio? Where do they store music if those two directories are constantly cleaned?
- jmcnevin 15y ago
  For the record, I almost immediately filled up my allotted iCloud backup space because of rdio, because I have 4.6 gigs of music synced for offline use, and it wanted to back that up. Solution: I disabled backups of rdio data.
smackfu 15y ago
I don't understand Apple's logic here. You can't reconcile the idea of "cleaning because you can redownload" with "available offline". As soon as you clean up, you are going to break offline uses. I would guess the idea is to help enable these 500 MB per issue magazine downloads. You download a new issue, you nuke some old issue, no one cares. As long as that wasn't an issue you cared about.
- voxmatt 15y ago
  I had the exact same question and your idea as to the answer is interesting. Perhaps this new behavior is in response to Newsstand's ability to automatically download new issues--users may allow those to stack up without any thought and run out of space. The question then becomes: why not just enforce this behavior for Newsstand?
  - morrow 15y ago
    Somewhat of a conspiracy theory - but if the choice for developers is risk losing losing local data or back data up to the cloud unnecessarily, I think they'll choose to store it to the cloud, as they will be the ones blamed when their app "deletes data randomly". The more storage space taken up by local files being backed up to iCloud, the more people will end up going over their 5GB capacity, and will have to upgrade to the higher capacity plans to cover it. I'm not sure I actually believe this right now, but if they refuse to fix the problem, then I might...
euroclydon 15y ago
Seem like the Dropbox app will have this quandary but on an even larger scale.
psychotik 15y ago
Isn't NSApplicationSupportDirectory for stuff that isn't "Documents" and yet needs to be managed as app state (not in 'Caches')? Why not just use that instead? That's what my app does and it seems to be OK with iOS 5.
- ufuk 15y ago
  Well, according to the documentation that folder is backed up to iTunes as well. Actually, it seems, the only folder that skips backups to iTunes in the Library folder is the Caches folder. The relevant part of the documentation is here: http://developer.apple.com/library/mac/#documentation/FileManagement/Conceptual/FileSystemProgrammingGUide/FileSystemOverview/FileSystemOverview.html http://developer.apple.com/library/mac/#documentation/FileMa...
- mrgrieves 15y ago
  I do the same thing. Apple is unclear about this, however. ApplicationSupport is at /Library/Application Support, and /Library definitely does get backed up to iTunes. http://developer.apple.com/library/ios/#DOCUMENTATION/FileManagement/Conceptual/FileSystemProgrammingGUide/FileSystemOverview/FileSystemOverview.html http://developer.apple.com/library/ios/#DOCUMENTATION/FileMa... That said, I think we're in the clear with respect to iCloud backups: "Devices with an active iCloud account have their app data backed up to iCloud at appropriate times. And for devices that are plugged into a computer, iTunes performs an incremental backup of the app’s data files." It's ambiguous. But "app data" sounds like documents, whereas "app's data files" sounds like big data files in /Library to me.
droithomme 15y ago
That's pretty interesting. So, the three non-backed locations are tmp, Caches and the Application Bundle. tmp and Caches are now swept. So, if you store your real cache stuff in the Application Bundle, it won't be backed up or auto-deleted. But maybe it gets trashed when you update the app. I wonder if OS X, in line with its trend to be more like iOS, is going to start automatically clearing the ~/Library/Caches directory as well.
- hamrickdavid 15y ago
  Unfortunately you can't write data to your application bundle. That folder is read-only.
- ryanpetrich 15y ago
  The application bundle is signed. Writing to that location should be blocked by the sandbox.
zamfi 15y ago
There needs to be a file storage location that behaves the way Caches did before iOS 5 I'm confused. If the goal is for documents to be available in the near future in offline form, why not keep documents in /Documents until the user has read them (or some sane amount of time has passed), and then move them to /Caches or some temporary storage? I've never used Instapaper, so perhaps documents are only stored until read anyway?
wrs 15y ago
It seems fair to say that Instapaper's version of an article can't be "redownloaded" for various reasons (offline, paywall, article removed, etc.) so it would be OK to put it in the Documents folder. The argument against that is that you're now syncing that article with iCloud in addition to Instapaper. But I wonder whether the correct answer is instead to eliminate Instapaper's sync feature, and just let iCloud do it. Once you have system-level cloud sync, don't you want to let Apple do the work? Sync is hard, and it isn't really the core value of Instapaper. Edit: I was wondering about iCloud only for iOS 5/MacOS 10.7.2 devices with iCloud accounts. But that story does fall apart for people with mixed devices. So never mind.
- akmiller 15y ago
  Instapaper is more than just an iOS application, so no, iCloud is not the answer for syncing.
- biturd 15y ago
  I see it being a long while before everyone has iCloud up and running. We installed it last night on a relatively new MBP, to learn it doesn't work without Lion, so now we are upgrading to Lion. It will only work on certain phones, I believe 3GS and above. Maybe Apple will release iCloud for Snow Leopard, but as it is now, instapaper works all the way down to the terrible version 2 iPhone I have had to resort to using in order to not sign into a new contract waiting on the release of the 4s. Instapaper would hurt too many current users who are just fine with how things work without the cloud.
Scorponok 15y ago
Maybe the solution is to make it a choice for the user? An option to "clean up documents when space is low on the device" in the instapaper options. If checked, stuff gets stored in cache. If not, in documents. That way, the default behavior is that "download something = want to keep it on device", but users can do the other one too if they want. I don't think the option is particularly useful, but it might make the app reviewer happy?
- WiseWeasel 15y ago
  This is what needs to happen. Either there needs to be a user confirmation before information can be "cleaned" from an app, or there needs to be a setting in the Settings/Appname for disallowing cleaning of that app's data. [Edit] Upon further examination of the new Settings/iCloud in iOS 5, what will need to happen is that the user should be able to disable iCloud syncing for individual 3rd party apps. That's the most consistent solution for the current interface. [/Edit]
- high5ths 15y ago
  I would bet Apple's never going to make this sort of a choice obvious to the user -- it goes against the "just working" (whether or not it works the way you want) they're currently pushing. It sure does worry me to read all these comments though; it seems like you'll never be able to count on an app having the data it needs to run, without re-downloading. As somebody who spends half his day in an airplane or a subway, plus plenty of time in expensive places (like overseas), that's terrifying.
  - lreeves 15y ago
    Actually a couple video players right now do make this choice available to the user - OPlayer and AVPlayer. They both say something to the effect of "Backup media" in their settings. This simply toggles which location the media is stored in (AVPlayer even seems to move the media on the fly). This is literally exactly how I would implement the choice if I were Marco.
    - WiseWeasel 15y ago
lukeredpath 15y ago
Lots of people are focussing on Marco's particular use case in the comments, and I think it's a valid one, but this extends beyond simple documents. There is a category of data that is aimed at offline use. Streaming apps like Spotify, that let you download playlists for offline use. GPS apps that download hundreds of MB of map data. You get the idea. On one hand, this data is a form of cache. The data is always available elsewhere (on the content provider servers) and it can be restored if necessary in a worst case scenario. But the key word here is "offline". This is the kind of data that, by definition depends on being around if the user is offline and therefore cannot be easily restored on demand, when the user needs it. Obviously, having all of this stuff backed up to iCloud and using up GBs of people's capacity is not feasible or even logical. So this kind of data does not belong anywhere that iCloud will back up. But it must be stored somewhere that is safe from being purged. Yes, a users GPS maps can be restored eventually but that doesn't help them when they are stranded in the middle of nowhere with nothing but a weak GPRS signal and all of their maps gone. Apple have made an almighty cockup in overlooking the "offline data" use case. In Marco's case, I'd agree that the articles represent user data that should be stored somewhere like Application Support, which will be backed by iCloud but I think that's probably fine in this case.
- jeromeparadis 15y ago
  I entirely agree. Apple forgot an important use case? I hate it when software is built to think in place of the user. There are two culprits here: * the lack of persistent, non backable offline storage * the addition of a auto-cleaning procedure The auto-cleanup without a storage alternative is a blunder. iOS 5 decides for the user what should be deleted to free up space. An alternative would be to prompt what application data you would like to remove instead of a blind decision. Otherwise, that's the kind of programming that will one day lead robots to cleanup the human race! ;) Of course, having permanent local storage that's not backed up would be a more user-friendly solution. However, it would probably lead people to run out of space when developers begin abusing it like some did with temp or cache storage. It seems like a trend at Apple these days. With Lion, the OS now decides when an application should quit when unused.
smokey_the_bear 15y ago
I write several offline mapping apps, and this is totally throwing us for a loop. We're recommending our power users not upgrade to iOS 5. Users download gigabytes of maps to their cache directory, they don't want to eat their iCloud allotment with that, or their slow their iTunes sync. But they also don't want to have to download those maps again, or find themselves in the middle of the woods without the maps they downloaded.
- andrewljohnson 15y ago
  I guess I will have to write a blog post for us too. I wonder if there is any chance that Apple will change how this works - it seems like this should be doomsday for a large swath of apps. (married to Smokey and partner on all these apps!)
  - moonlighter 15y ago
    The bigger question is why wasn't this considered DURING the iOS5 beta? If it got brought to Apple's attention during the beta cycles and they decided to not fix it then they'll most likely won't fix it now either (although it seems like they really don't have much choice here but to provide a solution one way or another)
    - smackfu 15y ago
      All the developers in the beta are under NDA, so they can't write attention-getting blog posts that might actually influence Apple to do something.
      - moonlighter 15y ago
        Fair point, but attention-grabbing blog posts are not the only way to get your voice heard as a developer. There's a difference between 'feedback' and 'creating public pressure'. But I hear ya...
- chacha102 15y ago
  Isn't this what innovation is about? Finding ways of solving problems? Is forcing apps to make more efficient of their use of space such a bad thing? From what I see, your users have 3 choices: They can use your app and use up their iCloud allotment, they can use your app and not upgrade, or they can upgrade and find a mapping app that is more efficient in its use of space.
xpaulbettsx 15y ago
It sounds like what Marco is looking for is something equivalent to Windows's AppData\Local, machine local, doesn't get nuked, but doesn't sync over roaming profiles either
cschep 15y ago
So, people (developers) are going to have to put their stuff in Documents, and instruct users to disable the iCloud sync for their particular app, unless the user really wants to have it eat into their iCloud storage. Would that work?
- marchdown 15y ago
  The whole quagmire is bigger than just flushing Cache. Apple prohibits developers to store stuff that can be regenerated or re-downloaded. So, arguably, you can't just switch to using Documents for downloadable data.
  - deleted 15y ago
    [deleted]
  - high5ths 15y ago
    Is there a source for this? By definition, isn't nearly everything we have on our devices capable of being regenerated or re-downloaded? This is clearly the crux of the problem -- Apple should allow developers (or, god forbid, users!) the choice to install things in a secure location regardless of "ease of re-download." How do they know when I'll have the bandwidth to re-download?
jackvalentine 15y ago
The first time that one of my several hundred megabyte foreign language dictionary files isn't available and needs to be re-downloaded when I need it in say, a meeting will trigger a severe re-evaluation of my use of the phone.
- jackvalentine 15y ago
  I just emailed the company that makes the dictionary app in question (pleco chinese dictionary, if anyone cares) and they confirmed this issue will affect them. The ball is in Apple's court.
staufman 15y ago
Could you use NSUserDefaults? I don't think there is a limit on the data stored there.
- apike 15y ago
  NSUserDefaults gets very slow when you put megabytes of data in it.
- phil 15y ago
  NSUserDefaults is backed up.
mw1234 15y ago
This is also particularly problematic for my own apps, which are offline photo browsers that sync your collection of photos. Keeping GBs of data in the Caches folder was the only way to have iPhone backups occur reasonably fast. Sure, they can be re-synced, but that will be a very time-consuming process for thousands of photos.
falling 15y ago
He says he knew about this behavior, and he also explicitly said (on Twitter, can't be bothered looking for it) that he was not going to report bugs to Apple during the beta. I guess Marco just prefers venting after the fact.
- russellallen 15y ago
  Maybe if you are going to slag someone off you should be bothered looking for the tweet you base your slander on...
  - falling 15y ago
    Fair enough. I tried now but that was months ago and Twitter search is unhelpful. However, he's being asked by multiple people right now for the radar# and he's ignoring the questions. He just said he's known it for two weeks, which might have been too late for Apple to fix it anyway.
    - dextorious 15y ago
      Yeah, one thing I don't understand is Marco had access to dev builds of iOS for months and iCloud dev access I presume. Why he only found about it now instead of doing something about it all this time?
      - allwein 15y ago
        Probably because, like most developers, he has separate development devices that he only uses for development. So he probably never ran into storage constraint limits while just testing Instapaper. As an example, I have a 32GB iPod Touch that I've been using for all of the iOS5 testing for my applications and I'm only using around 600MB total. I'd never run into this issue either unless I loaded this up with tons of other apps and movies and music.
        dolbz 15y ago
        That's incorrect, he stated on his podcast that he's used iOS5 on his main device since WWDC. Whether he hits the limit regularly or not is another question...
sidwyn 15y ago
I develop Definition (http://definitionapp.com http://definitionapp.com) and I store the database in the Caches directory as well, after receiving the email from Apple to move. This is bad news for me, the entire offline dictionary could be wiped out.
sjwright 15y ago
There are a number of possible scenarios for file storage, the problem is a lack of clarity or documentation about the properties of the various locations as they stand now. As a developer, I could imagine desiring the following choices: 1. Temp: No backup, cleared regularly 2. Cache: No backup, cleared when space is tight 3. Local: Local backup only, never cleared 4. Documents: Local/cloud backup, never cleared 5. Cloud: Cloud backup, cleared when space is tight The problem seems to be that #3 doesn't exist. Yet you'd think it would be a common requirement for stuff like in-app purchases of large and essential content packs, for example, turn-by-turn navigation maps. I'd hate to be on holiday and have a 10 megabyte podcast download automatically trigger the erasure of 1000 megabytes of navigation data.
- mw1234 15y ago
  You are missing "No backup, never cleared" which is what the old "2. Cache" scenario used to be.
  - sjwright 15y ago
    I disagree. If it's not worth backing up (locally), it doesn't deserve protected status on the phone. Or to put it another way, if it's important enough to retain when storage space is low, it's important enough to retain for when you have to recover from a backup.
    - hboon 15y ago
      You left out the case of storing data for offline use, which is one of the key use of Instapaper, offline maps, cached notebooks, etc. If I stored something specifically for offline use, I want it to be available as long as my device is with me, even when I'm offline. (1) Some of it is important as you say, and they should be backed-up, but (2) some are important, but is capable of being downloaded again and shouldn't be backed up and shouldn't'. The latter is the one that is now automatically cleaned up.
    - mw1234 15y ago
      I think that is a decision best left to the developer or user. Take an app that syncs your photos from a site you may already use, like Picasa, Flickr or SmugMug. You want to have these available on your iPad or iPhone instantly, even when you are without internet service. But you don't need them backed up to your computer every time you sync, causing the backup times to skyrocket, and if you lose them on a restore, it's a minor inconvenience, but not a huge deal. Losing them unexpectedly when you don't have the time or ability to re-downlaod them is a much more major problem for some people.
tjmc 15y ago
DHH declared the solution in 2007 [1] - apparently nobody needs offline applications! [1] http://37signals.com/svn/posts/347-youre-not-on-a-fucking-plane-and-if-you-are-it-doesnt-matter http://37signals.com/svn/posts/347-youre-not-on-a-fucking-pl...
- shinratdr 15y ago
  I'm very rarely on a plane. However I take the subway every day, and I'm extremely thankful that Tweetbot, Reeder and Newsstand cache everything so that I have something to read. I also don't consider the subway a niche or isolated pocket. I just don't buy the lame justification about not always being connected. If you don't want to be connected, turn off your damn phone. Don't insist that everyone else shouldn't be able to enjoy their time the way they want to because you associate your mobile device with stress.
charlieok 15y ago
I must be missing something. What is the problem with backing up pages stored in InstaPaper to iCloud? That's exactly the behavior I would want if I were using InstaPaper.
- hboon 15y ago
  1. The user can re-download from Instapaper's servers. 2. It goes against the spirit of Apple's policy here, which is not to backup to iCloud what ca be re-downloaded (see 1) 3. The user might have lots of content stored and again due to 1, those don't need to be backed up to iCloud. And really Instapaper actually has a small version of the problem. What is critical is applications that has huge amounts of data like offline maps and wikipedia. Those can be re-downloaded from their servers yet most definitely should not be backed up to iCloud due to time and cost.
nrser 15y ago
manage it yourself: put things that need to be persistent in Documents. put the rest in Cache. move 'em as needed. do it automatically by download and access dates and/or provide an interface for people to manage it. your app absolutely needs tons and tons of data to function? doesn't seem like your day. it's their device, their cloud, their decision. Apple doesn't give a shit about your day; they're going cloud. they may be wrong, but i'd guess they're going to have to find that out for themselves. i'd assume they acknowledge this may kill some apps. i don't think they ever promised anyone a business; on the contrary, they seem to remind developers that they are there at their good grace all the time. as someone that built Facebook apps since '07, trust me, i know what this is like. start coding and start calling. best of luck.
rubergly 15y ago
I'm definitely at risk of running into this predicament as a user. I'm more worried about the hundreds of megabytes of podcasts I download when WiFi is available for use throughout the day when WiFi isn't available. To avoid this issue and enjoy the benefits of iOS 5, I'm going to have to clear out a bunch of apps, music, and photos and ensure that I always have a sufficient amount of "buffer" space so the cleaning is never triggered. I cannot be alone in this, and the fact that Apple is making this kind of thinking necessary for end users is kind of ridiculous. I really can't see this behavior lasting for very long, and I'm sure Apple will address it soon; this is the antithesis to the traditional iPhone experience. The only scenario where I could see this being purposeful is if Apple is really trying to hurt offline apps to increase data usage and appease carriers (maybe for pissing them off with iMessage?).
- Terry_B 15y ago
  Yeah, glad I read this before upgrading. My large collection of podcasts stored in Instacast will probably be lost straight away.
- phillmv 15y ago
  I would wait until more reports come in. Alternatively, just turn off iCloud back ups.
  - rubergly 15y ago
    Will turning off iCloud back ups turn off the auto-cleaning behavior?
    - cschep 15y ago
      It won't matter if the files are saved in the "Documents" folder as that isn't cleaned.
      - rubergly 15y ago
        Well yeah, but the point is developers will start moving it out of the Documents folder.
minga 15y ago
"Prior to iOS 5, the system never deleted the contents of Caches and tmp, so they were safe places for apps to put data that should always be available" Not exactly.
peterclary 15y ago
What about offline web apps which are saved onto the home screen? Are they also "cleaned"? If we can't depend upon an offline web app to be there when offline then that would fundamentally defeat the purpose. Of course, one could argue that the cleaning behaviour fundamentally defeats the purpose of a lot of apps, as already covered above (Instapaper, Offline Maps, etc.). My iMac is packed up for building work, so I can't upgrade my iPad and check this out for myself. Sorry.
qjz 15y ago
Apple - iCloud - Your content. On all your devices. That is the title of Apple's main iCloud page at http://www.apple.com/icloud/ http://www.apple.com/icloud/. iCloud is seamlessly integrated into your apps, so you can access your content on all your devices. This is Apple's definition for the iCloud service. It doesn't matter what the data is, it's your data and Apple is promising to sync it between your devices, to preserve your experience. In the case of Instapaper, the solution is obvious: Put the files in Documents. That is Instapaper's content and part of the experience that users want synced between devices. If Apple penalizes developers and undermines the promise it is making to users because it decides to be miserly about bandwidth, then it has to admit it launched iCloud before it was ready.
- deleted 15y ago
  [deleted]
- ChrisLTD 15y ago
  "If Apple penalizes developers and undermines the promise it is making to users because it decides to be miserly about bandwidth, then it has to admit it launched iCloud before it was ready." That's the key right there. The size of the Instapaper database is determined by the user. It will only fill up a user's iCloud space if they decide to save a lot of documents. Plus, they always have the option to buy additional space! Aside from making sure the documents are reasonably compressed, the size of the Instapaper database should not be Marco's concern.
- mbateman 15y ago
  Their marketing is definitely hyperbolic. Even in terms of Apple's own services. The supposed data syncing in iWork actually doesn't work on the Mac versions. It only works between iOS devices. So if I edit a paper on my Mac, and I want to use it on my iPad, I still have to go through an annoying uploading/downloading phase. There's nothing seamless about it. If you go to the bottom of http://www.apple.com/icloud/features/documents.html http://www.apple.com/icloud/features/documents.html you can see the web interface you have to use. I know this is a tangent to the OP but I was very annoyed to discover this last night, I feel like Apple was pretty misleading here. There are clearly technical difficulties that still need to be surmounted.
zbowling 15y ago
Add .nosync to the file/folder name path and keep it in the home or documents directory. Problem solved. edit: I'm shocked this thread is so long and no body mentioned this. It's been on the apple developer forums for months as a solution.
- ColinDabritz 15y ago
  Well that could use better publicity. Do you have a citation? Has anyone used this successfully?
- mpakes 15y ago
  I haven't had a chance to verify this myself, but according to Marco, this does not actually avoid the iCloud sync: https://twitter.com/#!/marcoarment/status/124587437799374848 https://twitter.com/#!/marcoarment/status/124587437799374848
  - zbowling 15y ago
    It actually does. I know because I've been testing with our app. It works perfectly. We build up a fairly large 25MB sqlite db and it doesn't show in iCloud anywhere. In fact, if you use UIManagedDocument, it creates a .nosync directory in the document bundle to prevent it from syncing in the framework itself.
  - kcbanner 15y ago
    The docs specifically say directory, not file. Maybe that is the problem?
- smokey_the_bear 15y ago
  Do you have any documentation about this? I see a few scant references in the developer forum and nothing in the docs.
  - smokey_the_bear 15y ago
    I moved our large data files from the library cache directory to a data.nosync directory in the documents directory. The data in the nosync folder is being backed up to iCloud.
martinbech 15y ago
I dont get it.. he get furios because he saves files he needs in a folder called /cache , and /cache gets purged when the system runs low on space.... What did you think cache meant? Why dont you just update your app?
MatthewPhillips 15y ago
I think the problem is that Marco sees Instapaper users as his customers. I sympathize with this, but I think iOS is an assertion by Apple that App Store customers are Apple customers. App Store Developers are providing a service to Apple's customers. But everything that happens must point back to Apple's servers, not their own. In the same way Apple doesn't want every developer operating their own independent payment system, they also don't want developers operating their own cloud storage services. If Apple holds the data they can guarantee its security. They likely see these problems as temporary, until developers and customers learn to adjust to the fact that iOS data is controlled by the iCloud service.
tvaughan 15y ago
Or worse: http://blog.foreflight.com/2011/10/14/flying-with-foreflight-and-ios-5/ http://blog.foreflight.com/2011/10/14/flying-with-foreflight...
faizanaziz 15y ago
Apple wants Cache (stuff that can be re downloaded and not core to the experience, just to improve performance) to actually be deletable when space is low. A valid point. Developers want a place to store app information that is critical and sometimes a very important part of the app(example main feature of instapaper is to view offline content). This should not be backed up since its redundant and when the phone needs to be restored(rarely) this data can be re-downloaded. Also backing up will be slow and destroy user experience. Solution make another location which is used for storing the data(not backed up) so then all are happy. Maybe you can have data from hidden directories not backed up inside the Documents directory. Then Apple can delete caches in apps like browsers. User critical data like settings can be backed up and app critical data that need not be backed up can also exist

568 points by JeffDClark 15y ago

I wouldn't say that the articles' metadata (URL, title, date of download, maybe thumbnail of the 1st page) is downloaded content. It's clearly user-generated, and should be in the "home" of the app IMHO. The articles themselves, yes, send them to the cache. If the user needs to reclaim the storage used up by the articles, let the OS delete them. Then, when the user needs to read the article again, it will take some time to download. But don't get into an "all articles gone" situation. Just my 2 cents.

masklinn 15y ago
I'd submit that this is, in some ways, worse: instead of all articles being gone they're now all grayed out, so the user knows they were here, can still see there, but if he's offline he can not access them because only the metadata is left, the data itself is gone.
scdc 15y ago
I thought Instapaper scraped the page and stored that, to accommodate sites that require login for access. So not trivial to just re-download the articles. Plus the whole point is to have them available when you're offline. I'd say in this case, the content is user-generated and should be backed up.
- masklinn 15y ago
  That's a tough line to argue: the scraped page is actually stored by Instapaper (otherwise you could not access it from the website), so you re-download it from there, and technically it's not user-generated it's at best user-curated. But this does indeed defeat the point of having it offline, which is a significant part of IP's purpose.
deleted 15y ago
[deleted]
luchak 15y ago
What problem are you trying to fix? If I'm offline, what good is being able to see my article list if I can't read the articles? If I'm online, why do I feel any less irritated when I try to go back to an article I was just reading only to find that I need to wait for it to download again?
- adgar 15y ago
  The problem fixed is that the user has an indication as to what happened to all of their saved articles. That's a very big difference. It's not a full solution, as the parent noted, but the UX is far superior.

theatrus2 15y ago

Has anyone figured out where the high water mark is? When will the cleaning behavior kick in? This is an interesting twist especially with the 16GB devices which tend to actually be above 50%.

deleted 15y ago

[deleted]

oscardelben 15y ago

If the device is low on space I think it makes sense for Apple to purge cached data and tmp directories. For an app like Instapaper which relies on local documents, some of the documents should (imho) be saved to the home directory instead. And yes, I know this is hard but I'm just thinking aloud.

wmf 15y ago

Clearly, apps that cache data need to be modified to show the difference between having no data and having no data cached. IMAP clients have dealt with this, for example; they show a message like "the contents of this folder are not available offline". Perhaps Apple could have made cache cleaning opt-in on a per-app basis until iOS 6, though.

smackfu 15y ago
The main issue is that iOS is deciding that you don't need something offline without asking you. I don't want my cached email to turn out to have been deleted because I downloaded something else entirely. Many people would prefer the old method, where it would say you were out of space, and you would have to go free up some space.
- politician 15y ago
  That's right; the solution is to make it easier to see which apps are using how much space and then provide a "clear" button. My Nintendo Wii uses a "blocks" concept for this. It works fine, Apple should just copy it.

jmcnevin 15y ago

Is it possible that a user could use Instapaper to save a document that they wouldn't be able to download later, even if they had access to an internet connection? If that's possible, I think Instapaper would have every right to store things in the Documents folder, since you're asking it to create something more akin to an archive than a temporary cache of data.

sp332 15y ago
I could be wrong, but I think all the articles are on Instapaper's servers, so you can re-download them.
- lukeredpath 15y ago
  But only if you are online, which defeats the object of "offline access".
  - sp332 15y ago
    That's not what jmcnevin asked.
  - sp332 15y ago
    Oh, I see what you mean now. I feel dumb :)
baddox 15y ago
That seems to be the only way for Instapaper to give their users a reasonable experience, but it's still a bad solution. It doesn't make much sense for the Instapaper app to backup its users' cached articles to iCloud, when the whole point of the app is to backup the articles on the iPhone. If you have access to iCloud, you have access to the original article, so it's a waste of bandwidth. Not to mention that Instapaper risks getting hounded by Apple for the iCloud usage.
- lukeredpath 15y ago
  I honestly don't think there is an issue with allowing Instapaper content to get backed up to iCloud here. The analogy of a user created archive is a good one. If people don't want the benefit of having their Instapaper content transfers across their devices automatically (and I do think this is a good feature) then they are free to disable iCloud backup for Instapaper in Settings.

sadfasdfads 15y ago

This highlights the fact that Apple doesn't have the resources to be able to work through issues like this with developers of well-used apps before changes are made. Maybe they should open more positions like these: http://apple-careers.com/job/Austin-Mac-2FiOS-Developer-Support-Representative-Job-TX-73301/1477606/ http://apple-careers.com/job/Austin-Mac-2FiOS-Developer-Supp... Sending an email out is insufficient. I've had similar experiences with a company that develops a product that I've written extensions for. If your product relies on an ecosystem, and you do something that is going to disturb that ecosystem, you'd better be prepared for the backlash.

JeffDClark 15y ago

The whole idea of an app like Instapaper (or any of the other examples presented) is that the "stuff" that is saved for later is all user-generated content. Some articles may even vanish (different location, move behind a paywall, deleted, etc...). In this case those articles would become inaccessible when the OS deletes the cache. It seems that the argument that only the list of metadata is user-generated can apply to any type of media (music, movies, etc...). Technically speaking all of the music on my phone could be downloaded on demand. Of course this requires an always connected, fat, and cheap network connection. Which is pretty much the opposite of what most folks have. This also breaks the user's expectations. I was annoyed/surprised when I upgraded to iOS 5 and Instapaper had to re-download everything.

masklinn 15y ago
> Some articles may even vanish (different location, move behind a paywall, deleted, etc...). In this case those articles would become inaccessible when the OS deletes the cache. As far as I know, all scraped articles are stored in Instapaper's server and downloaded from there, you don't get them directly from the source page.
- JeffDClark 15y ago
  I have had articles in my list that have become unavailable. No amount of trying to re-download them works. What you say makes sense but I am unfamiliar with the exact way that it works.

ender7 15y ago

Apple's in a tight spot here - either they leave things as they are and piss off developers (and, by fiat, piss off users), or they have to start forcing users to more proactively manage their remaining space. One can probably easily imagine an interface for showing the user how much data a particular app is using and allow them to nuke the temporary stuff. It might even look beautiful. It might even be fun to use. But it's going to introduce a lot of hand-wringing and micro-managing and lots of mental overhead that Apple really, really wants to avoid.

thehigherlife 15y ago
iOS 5 now lets you see per app space usage. (it's under Settings > General > Usage) I don't see anyway to delete that temp data, however, only the app itself (for non-apple apps).
- mrcharles 15y ago
  That's only the data that will be backed up to iCloud though, so it probably doesn't include cache/tmp.
  - 77ko 15y ago
    It does
  - freedompeace 15y ago
    It does - my Downloads app is reported as using 10 GB.
deleted 15y ago
[deleted]
sudoman 15y ago
This general problem is a symptom of a major issue regarding the use of software that does not respect the freedom of its users. Every non-apple developer and user is required to bend to the will of the programmers who develop the proprietary iOS. Any negative choices or limitations imposed by Apple Inc. are virtually uncircumventable. On the other hand, if users and developers in a community are free to view, modify and share the source code of the operating system and programs they are using, then that creates a dynamic where software cannot remain defective by design, something which cannot be said about Apple's hardware or software. If we were talking about free (as in speech) software, users would have control of their data, and their choice of how to manage it; not some distant programmers working for a corporation's bottom line.

pohl 15y ago

I'm sympathetic to Marco's blog post here, but I wonder if he's failing to interpret the two paragraphs from the documentation within the context of Instapaper's purpose. With respect to point #1, and in the context of what Instapaper is for, the user is intending to read an article offline, and the local copy of the article was generated by the user's intent (and can, therefore, be considered user-generated even though the user is not the author of the article.) With respect to point #2, the articles fail the "can be downloaded again" test in light of the app's purpose of making the articles available to the user when the network is not available. When the network is unavailable, the articles cannot be downloaded again. Edit: JeffDClark makes another excellent point here that the article may have originally been behind a paywall and therefore cannot be guaranteed to be re-downloadable. The same is true if the author removed the original article from their webserver. Ergo, put them in the Documents folder. Whether this would satisfy the app reviewer is another question, but it's worth a shot to carefully explain to them how your app is not violating the letter of the law.

rcthompson 15y ago
But it doesn't make sense to back this data up to iCloud, which is what happens to the Documents folder.
- pohl 15y ago
  Really? If I generate an offline copy of an article on my phone, wouldn't I want to have access to it on my iPad? Why wouldn't I want the truth to be in the cloud?
  - Mavrik 15y ago
    Because there's no point in burning your cloud storage, Apple bandwidth and prolong backup times when those articles can be easly re-downloaded from Instapapers servers on iPad. Same result, without long backup times, Apple server bandwidth usage and your iCloud storage usage.
    - 15y ago

j_baker 15y ago

What about apps like spotify and rdio? Where do they store music if those two directories are constantly cleaned?

jmcnevin 15y ago
For the record, I almost immediately filled up my allotted iCloud backup space because of rdio, because I have 4.6 gigs of music synced for offline use, and it wanted to back that up. Solution: I disabled backups of rdio data.

smackfu 15y ago

I don't understand Apple's logic here. You can't reconcile the idea of "cleaning because you can redownload" with "available offline". As soon as you clean up, you are going to break offline uses. I would guess the idea is to help enable these 500 MB per issue magazine downloads. You download a new issue, you nuke some old issue, no one cares. As long as that wasn't an issue you cared about.

voxmatt 15y ago
I had the exact same question and your idea as to the answer is interesting. Perhaps this new behavior is in response to Newsstand's ability to automatically download new issues--users may allow those to stack up without any thought and run out of space. The question then becomes: why not just enforce this behavior for Newsstand?
- morrow 15y ago
  Somewhat of a conspiracy theory - but if the choice for developers is risk losing losing local data or back data up to the cloud unnecessarily, I think they'll choose to store it to the cloud, as they will be the ones blamed when their app "deletes data randomly". The more storage space taken up by local files being backed up to iCloud, the more people will end up going over their 5GB capacity, and will have to upgrade to the higher capacity plans to cover it. I'm not sure I actually believe this right now, but if they refuse to fix the problem, then I might...

euroclydon 15y ago

Seem like the Dropbox app will have this quandary but on an even larger scale.

psychotik 15y ago

Isn't NSApplicationSupportDirectory for stuff that isn't "Documents" and yet needs to be managed as app state (not in 'Caches')? Why not just use that instead? That's what my app does and it seems to be OK with iOS 5.

ufuk 15y ago
Well, according to the documentation that folder is backed up to iTunes as well. Actually, it seems, the only folder that skips backups to iTunes in the Library folder is the Caches folder. The relevant part of the documentation is here: http://developer.apple.com/library/mac/#documentation/FileManagement/Conceptual/FileSystemProgrammingGUide/FileSystemOverview/FileSystemOverview.html http://developer.apple.com/library/mac/#documentation/FileMa...
mrgrieves 15y ago
I do the same thing. Apple is unclear about this, however. ApplicationSupport is at /Library/Application Support, and /Library definitely does get backed up to iTunes. http://developer.apple.com/library/ios/#DOCUMENTATION/FileManagement/Conceptual/FileSystemProgrammingGUide/FileSystemOverview/FileSystemOverview.html http://developer.apple.com/library/ios/#DOCUMENTATION/FileMa... That said, I think we're in the clear with respect to iCloud backups: "Devices with an active iCloud account have their app data backed up to iCloud at appropriate times. And for devices that are plugged into a computer, iTunes performs an incremental backup of the app’s data files." It's ambiguous. But "app data" sounds like documents, whereas "app's data files" sounds like big data files in /Library to me.

droithomme 15y ago

That's pretty interesting. So, the three non-backed locations are tmp, Caches and the Application Bundle. tmp and Caches are now swept. So, if you store your real cache stuff in the Application Bundle, it won't be backed up or auto-deleted. But maybe it gets trashed when you update the app. I wonder if OS X, in line with its trend to be more like iOS, is going to start automatically clearing the ~/Library/Caches directory as well.

hamrickdavid 15y ago
Unfortunately you can't write data to your application bundle. That folder is read-only.
ryanpetrich 15y ago
The application bundle is signed. Writing to that location should be blocked by the sandbox.

zamfi 15y ago

There needs to be a file storage location that behaves the way Caches did before iOS 5 I'm confused. If the goal is for documents to be available in the near future in offline form, why not keep documents in /Documents until the user has read them (or some sane amount of time has passed), and then move them to /Caches or some temporary storage? I've never used Instapaper, so perhaps documents are only stored until read anyway?

wrs 15y ago

It seems fair to say that Instapaper's version of an article can't be "redownloaded" for various reasons (offline, paywall, article removed, etc.) so it would be OK to put it in the Documents folder. The argument against that is that you're now syncing that article with iCloud in addition to Instapaper. But I wonder whether the correct answer is instead to eliminate Instapaper's sync feature, and just let iCloud do it. Once you have system-level cloud sync, don't you want to let Apple do the work? Sync is hard, and it isn't really the core value of Instapaper. Edit: I was wondering about iCloud only for iOS 5/MacOS 10.7.2 devices with iCloud accounts. But that story does fall apart for people with mixed devices. So never mind.

akmiller 15y ago
Instapaper is more than just an iOS application, so no, iCloud is not the answer for syncing.
biturd 15y ago
I see it being a long while before everyone has iCloud up and running. We installed it last night on a relatively new MBP, to learn it doesn't work without Lion, so now we are upgrading to Lion. It will only work on certain phones, I believe 3GS and above. Maybe Apple will release iCloud for Snow Leopard, but as it is now, instapaper works all the way down to the terrible version 2 iPhone I have had to resort to using in order to not sign into a new contract waiting on the release of the 4s. Instapaper would hurt too many current users who are just fine with how things work without the cloud.

Scorponok 15y ago

Maybe the solution is to make it a choice for the user? An option to "clean up documents when space is low on the device" in the instapaper options. If checked, stuff gets stored in cache. If not, in documents. That way, the default behavior is that "download something = want to keep it on device", but users can do the other one too if they want. I don't think the option is particularly useful, but it might make the app reviewer happy?

WiseWeasel 15y ago
This is what needs to happen. Either there needs to be a user confirmation before information can be "cleaned" from an app, or there needs to be a setting in the Settings/Appname for disallowing cleaning of that app's data. [Edit] Upon further examination of the new Settings/iCloud in iOS 5, what will need to happen is that the user should be able to disable iCloud syncing for individual 3rd party apps. That's the most consistent solution for the current interface. [/Edit]
high5ths 15y ago
I would bet Apple's never going to make this sort of a choice obvious to the user -- it goes against the "just working" (whether or not it works the way you want) they're currently pushing. It sure does worry me to read all these comments though; it seems like you'll never be able to count on an app having the data it needs to run, without re-downloading. As somebody who spends half his day in an airplane or a subway, plus plenty of time in expensive places (like overseas), that's terrifying.
- lreeves 15y ago
  Actually a couple video players right now do make this choice available to the user - OPlayer and AVPlayer. They both say something to the effect of "Backup media" in their settings. This simply toggles which location the media is stored in (AVPlayer even seems to move the media on the fly). This is literally exactly how I would implement the choice if I were Marco.
  - WiseWeasel 15y ago

lukeredpath 15y ago

Lots of people are focussing on Marco's particular use case in the comments, and I think it's a valid one, but this extends beyond simple documents. There is a category of data that is aimed at offline use. Streaming apps like Spotify, that let you download playlists for offline use. GPS apps that download hundreds of MB of map data. You get the idea. On one hand, this data is a form of cache. The data is always available elsewhere (on the content provider servers) and it can be restored if necessary in a worst case scenario. But the key word here is "offline". This is the kind of data that, by definition depends on being around if the user is offline and therefore cannot be easily restored on demand, when the user needs it. Obviously, having all of this stuff backed up to iCloud and using up GBs of people's capacity is not feasible or even logical. So this kind of data does not belong anywhere that iCloud will back up. But it must be stored somewhere that is safe from being purged. Yes, a users GPS maps can be restored eventually but that doesn't help them when they are stranded in the middle of nowhere with nothing but a weak GPRS signal and all of their maps gone. Apple have made an almighty cockup in overlooking the "offline data" use case. In Marco's case, I'd agree that the articles represent user data that should be stored somewhere like Application Support, which will be backed by iCloud but I think that's probably fine in this case.

jeromeparadis 15y ago
I entirely agree. Apple forgot an important use case? I hate it when software is built to think in place of the user. There are two culprits here: * the lack of persistent, non backable offline storage * the addition of a auto-cleaning procedure The auto-cleanup without a storage alternative is a blunder. iOS 5 decides for the user what should be deleted to free up space. An alternative would be to prompt what application data you would like to remove instead of a blind decision. Otherwise, that's the kind of programming that will one day lead robots to cleanup the human race! ;) Of course, having permanent local storage that's not backed up would be a more user-friendly solution. However, it would probably lead people to run out of space when developers begin abusing it like some did with temp or cache storage. It seems like a trend at Apple these days. With Lion, the OS now decides when an application should quit when unused.

smokey_the_bear 15y ago

I write several offline mapping apps, and this is totally throwing us for a loop. We're recommending our power users not upgrade to iOS 5. Users download gigabytes of maps to their cache directory, they don't want to eat their iCloud allotment with that, or their slow their iTunes sync. But they also don't want to have to download those maps again, or find themselves in the middle of the woods without the maps they downloaded.

andrewljohnson 15y ago
I guess I will have to write a blog post for us too. I wonder if there is any chance that Apple will change how this works - it seems like this should be doomsday for a large swath of apps. (married to Smokey and partner on all these apps!)
- moonlighter 15y ago
  The bigger question is why wasn't this considered DURING the iOS5 beta? If it got brought to Apple's attention during the beta cycles and they decided to not fix it then they'll most likely won't fix it now either (although it seems like they really don't have much choice here but to provide a solution one way or another)
  - smackfu 15y ago
    All the developers in the beta are under NDA, so they can't write attention-getting blog posts that might actually influence Apple to do something.
    - moonlighter 15y ago
      Fair point, but attention-grabbing blog posts are not the only way to get your voice heard as a developer. There's a difference between 'feedback' and 'creating public pressure'. But I hear ya...
chacha102 15y ago
Isn't this what innovation is about? Finding ways of solving problems? Is forcing apps to make more efficient of their use of space such a bad thing? From what I see, your users have 3 choices: They can use your app and use up their iCloud allotment, they can use your app and not upgrade, or they can upgrade and find a mapping app that is more efficient in its use of space.

xpaulbettsx 15y ago

It sounds like what Marco is looking for is something equivalent to Windows's AppData\Local, machine local, doesn't get nuked, but doesn't sync over roaming profiles either

cschep 15y ago

So, people (developers) are going to have to put their stuff in Documents, and instruct users to disable the iCloud sync for their particular app, unless the user really wants to have it eat into their iCloud storage. Would that work?

marchdown 15y ago
The whole quagmire is bigger than just flushing Cache. Apple prohibits developers to store stuff that can be regenerated or re-downloaded. So, arguably, you can't just switch to using Documents for downloadable data.
- deleted 15y ago
  [deleted]
- high5ths 15y ago
  Is there a source for this? By definition, isn't nearly everything we have on our devices capable of being regenerated or re-downloaded? This is clearly the crux of the problem -- Apple should allow developers (or, god forbid, users!) the choice to install things in a secure location regardless of "ease of re-download." How do they know when I'll have the bandwidth to re-download?

jackvalentine 15y ago

The first time that one of my several hundred megabyte foreign language dictionary files isn't available and needs to be re-downloaded when I need it in say, a meeting will trigger a severe re-evaluation of my use of the phone.

jackvalentine 15y ago
I just emailed the company that makes the dictionary app in question (pleco chinese dictionary, if anyone cares) and they confirmed this issue will affect them. The ball is in Apple's court.

staufman 15y ago

Could you use NSUserDefaults? I don't think there is a limit on the data stored there.

apike 15y ago
NSUserDefaults gets very slow when you put megabytes of data in it.
phil 15y ago
NSUserDefaults is backed up.

mw1234 15y ago

This is also particularly problematic for my own apps, which are offline photo browsers that sync your collection of photos. Keeping GBs of data in the Caches folder was the only way to have iPhone backups occur reasonably fast. Sure, they can be re-synced, but that will be a very time-consuming process for thousands of photos.

falling 15y ago

He says he knew about this behavior, and he also explicitly said (on Twitter, can't be bothered looking for it) that he was not going to report bugs to Apple during the beta. I guess Marco just prefers venting after the fact.

russellallen 15y ago
Maybe if you are going to slag someone off you should be bothered looking for the tweet you base your slander on...
- falling 15y ago
  Fair enough. I tried now but that was months ago and Twitter search is unhelpful. However, he's being asked by multiple people right now for the radar# and he's ignoring the questions. He just said he's known it for two weeks, which might have been too late for Apple to fix it anyway.
  - dextorious 15y ago
    Yeah, one thing I don't understand is Marco had access to dev builds of iOS for months and iCloud dev access I presume. Why he only found about it now instead of doing something about it all this time?
    - allwein 15y ago
      Probably because, like most developers, he has separate development devices that he only uses for development. So he probably never ran into storage constraint limits while just testing Instapaper. As an example, I have a 32GB iPod Touch that I've been using for all of the iOS5 testing for my applications and I'm only using around 600MB total. I'd never run into this issue either unless I loaded this up with tons of other apps and movies and music.
      - dolbz 15y ago
        That's incorrect, he stated on his podcast that he's used iOS5 on his main device since WWDC. Whether he hits the limit regularly or not is another question...

sidwyn 15y ago

I develop Definition (http://definitionapp.com http://definitionapp.com) and I store the database in the Caches directory as well, after receiving the email from Apple to move. This is bad news for me, the entire offline dictionary could be wiped out.

sjwright 15y ago

There are a number of possible scenarios for file storage, the problem is a lack of clarity or documentation about the properties of the various locations as they stand now. As a developer, I could imagine desiring the following choices: 1. Temp: No backup, cleared regularly 2. Cache: No backup, cleared when space is tight 3. Local: Local backup only, never cleared 4. Documents: Local/cloud backup, never cleared 5. Cloud: Cloud backup, cleared when space is tight The problem seems to be that #3 doesn't exist. Yet you'd think it would be a common requirement for stuff like in-app purchases of large and essential content packs, for example, turn-by-turn navigation maps. I'd hate to be on holiday and have a 10 megabyte podcast download automatically trigger the erasure of 1000 megabytes of navigation data.

mw1234 15y ago
You are missing "No backup, never cleared" which is what the old "2. Cache" scenario used to be.
- sjwright 15y ago
  I disagree. If it's not worth backing up (locally), it doesn't deserve protected status on the phone. Or to put it another way, if it's important enough to retain when storage space is low, it's important enough to retain for when you have to recover from a backup.
  - hboon 15y ago
    You left out the case of storing data for offline use, which is one of the key use of Instapaper, offline maps, cached notebooks, etc. If I stored something specifically for offline use, I want it to be available as long as my device is with me, even when I'm offline. (1) Some of it is important as you say, and they should be backed-up, but (2) some are important, but is capable of being downloaded again and shouldn't be backed up and shouldn't'. The latter is the one that is now automatically cleaned up.
  - mw1234 15y ago
    I think that is a decision best left to the developer or user. Take an app that syncs your photos from a site you may already use, like Picasa, Flickr or SmugMug. You want to have these available on your iPad or iPhone instantly, even when you are without internet service. But you don't need them backed up to your computer every time you sync, causing the backup times to skyrocket, and if you lose them on a restore, it's a minor inconvenience, but not a huge deal. Losing them unexpectedly when you don't have the time or ability to re-downlaod them is a much more major problem for some people.

tjmc 15y ago

DHH declared the solution in 2007 [1] - apparently nobody needs offline applications! [1] http://37signals.com/svn/posts/347-youre-not-on-a-fucking-plane-and-if-you-are-it-doesnt-matter http://37signals.com/svn/posts/347-youre-not-on-a-fucking-pl...

shinratdr 15y ago
I'm very rarely on a plane. However I take the subway every day, and I'm extremely thankful that Tweetbot, Reeder and Newsstand cache everything so that I have something to read. I also don't consider the subway a niche or isolated pocket. I just don't buy the lame justification about not always being connected. If you don't want to be connected, turn off your damn phone. Don't insist that everyone else shouldn't be able to enjoy their time the way they want to because you associate your mobile device with stress.

charlieok 15y ago

I must be missing something. What is the problem with backing up pages stored in InstaPaper to iCloud? That's exactly the behavior I would want if I were using InstaPaper.

hboon 15y ago
1. The user can re-download from Instapaper's servers. 2. It goes against the spirit of Apple's policy here, which is not to backup to iCloud what ca be re-downloaded (see 1) 3. The user might have lots of content stored and again due to 1, those don't need to be backed up to iCloud. And really Instapaper actually has a small version of the problem. What is critical is applications that has huge amounts of data like offline maps and wikipedia. Those can be re-downloaded from their servers yet most definitely should not be backed up to iCloud due to time and cost.

nrser 15y ago

manage it yourself: put things that need to be persistent in Documents. put the rest in Cache. move 'em as needed. do it automatically by download and access dates and/or provide an interface for people to manage it. your app absolutely needs tons and tons of data to function? doesn't seem like your day. it's their device, their cloud, their decision. Apple doesn't give a shit about your day; they're going cloud. they may be wrong, but i'd guess they're going to have to find that out for themselves. i'd assume they acknowledge this may kill some apps. i don't think they ever promised anyone a business; on the contrary, they seem to remind developers that they are there at their good grace all the time. as someone that built Facebook apps since '07, trust me, i know what this is like. start coding and start calling. best of luck.

rubergly 15y ago

I'm definitely at risk of running into this predicament as a user. I'm more worried about the hundreds of megabytes of podcasts I download when WiFi is available for use throughout the day when WiFi isn't available. To avoid this issue and enjoy the benefits of iOS 5, I'm going to have to clear out a bunch of apps, music, and photos and ensure that I always have a sufficient amount of "buffer" space so the cleaning is never triggered. I cannot be alone in this, and the fact that Apple is making this kind of thinking necessary for end users is kind of ridiculous. I really can't see this behavior lasting for very long, and I'm sure Apple will address it soon; this is the antithesis to the traditional iPhone experience. The only scenario where I could see this being purposeful is if Apple is really trying to hurt offline apps to increase data usage and appease carriers (maybe for pissing them off with iMessage?).

Terry_B 15y ago
Yeah, glad I read this before upgrading. My large collection of podcasts stored in Instacast will probably be lost straight away.
phillmv 15y ago
I would wait until more reports come in. Alternatively, just turn off iCloud back ups.
- rubergly 15y ago
  Will turning off iCloud back ups turn off the auto-cleaning behavior?
  - cschep 15y ago
    It won't matter if the files are saved in the "Documents" folder as that isn't cleaned.
    - rubergly 15y ago
      Well yeah, but the point is developers will start moving it out of the Documents folder.

minga 15y ago

"Prior to iOS 5, the system never deleted the contents of Caches and tmp, so they were safe places for apps to put data that should always be available" Not exactly.

peterclary 15y ago

What about offline web apps which are saved onto the home screen? Are they also "cleaned"? If we can't depend upon an offline web app to be there when offline then that would fundamentally defeat the purpose. Of course, one could argue that the cleaning behaviour fundamentally defeats the purpose of a lot of apps, as already covered above (Instapaper, Offline Maps, etc.). My iMac is packed up for building work, so I can't upgrade my iPad and check this out for myself. Sorry.

qjz 15y ago

Apple - iCloud - Your content. On all your devices. That is the title of Apple's main iCloud page at http://www.apple.com/icloud/ http://www.apple.com/icloud/. iCloud is seamlessly integrated into your apps, so you can access your content on all your devices. This is Apple's definition for the iCloud service. It doesn't matter what the data is, it's your data and Apple is promising to sync it between your devices, to preserve your experience. In the case of Instapaper, the solution is obvious: Put the files in Documents. That is Instapaper's content and part of the experience that users want synced between devices. If Apple penalizes developers and undermines the promise it is making to users because it decides to be miserly about bandwidth, then it has to admit it launched iCloud before it was ready.

deleted 15y ago
[deleted]
ChrisLTD 15y ago
"If Apple penalizes developers and undermines the promise it is making to users because it decides to be miserly about bandwidth, then it has to admit it launched iCloud before it was ready." That's the key right there. The size of the Instapaper database is determined by the user. It will only fill up a user's iCloud space if they decide to save a lot of documents. Plus, they always have the option to buy additional space! Aside from making sure the documents are reasonably compressed, the size of the Instapaper database should not be Marco's concern.
mbateman 15y ago
Their marketing is definitely hyperbolic. Even in terms of Apple's own services. The supposed data syncing in iWork actually doesn't work on the Mac versions. It only works between iOS devices. So if I edit a paper on my Mac, and I want to use it on my iPad, I still have to go through an annoying uploading/downloading phase. There's nothing seamless about it. If you go to the bottom of http://www.apple.com/icloud/features/documents.html http://www.apple.com/icloud/features/documents.html you can see the web interface you have to use. I know this is a tangent to the OP but I was very annoyed to discover this last night, I feel like Apple was pretty misleading here. There are clearly technical difficulties that still need to be surmounted.

zbowling 15y ago

Add .nosync to the file/folder name path and keep it in the home or documents directory. Problem solved. edit: I'm shocked this thread is so long and no body mentioned this. It's been on the apple developer forums for months as a solution.

ColinDabritz 15y ago
Well that could use better publicity. Do you have a citation? Has anyone used this successfully?
mpakes 15y ago
I haven't had a chance to verify this myself, but according to Marco, this does not actually avoid the iCloud sync: https://twitter.com/#!/marcoarment/status/124587437799374848 https://twitter.com/#!/marcoarment/status/124587437799374848
- zbowling 15y ago
  It actually does. I know because I've been testing with our app. It works perfectly. We build up a fairly large 25MB sqlite db and it doesn't show in iCloud anywhere. In fact, if you use UIManagedDocument, it creates a .nosync directory in the document bundle to prevent it from syncing in the framework itself.
- kcbanner 15y ago
  The docs specifically say directory, not file. Maybe that is the problem?
smokey_the_bear 15y ago
Do you have any documentation about this? I see a few scant references in the developer forum and nothing in the docs.
- smokey_the_bear 15y ago
  I moved our large data files from the library cache directory to a data.nosync directory in the documents directory. The data in the nosync folder is being backed up to iCloud.

martinbech 15y ago

I dont get it.. he get furios because he saves files he needs in a folder called /cache , and /cache gets purged when the system runs low on space.... What did you think cache meant? Why dont you just update your app?

MatthewPhillips 15y ago

I think the problem is that Marco sees Instapaper users as his customers. I sympathize with this, but I think iOS is an assertion by Apple that App Store customers are Apple customers. App Store Developers are providing a service to Apple's customers. But everything that happens must point back to Apple's servers, not their own. In the same way Apple doesn't want every developer operating their own independent payment system, they also don't want developers operating their own cloud storage services. If Apple holds the data they can guarantee its security. They likely see these problems as temporary, until developers and customers learn to adjust to the fact that iOS data is controlled by the iCloud service.

tvaughan 15y ago

Or worse: http://blog.foreflight.com/2011/10/14/flying-with-foreflight-and-ios-5/ http://blog.foreflight.com/2011/10/14/flying-with-foreflight...

faizanaziz 15y ago

Apple wants Cache (stuff that can be re downloaded and not core to the experience, just to improve performance) to actually be deletable when space is low. A valid point. Developers want a place to store app information that is critical and sometimes a very important part of the app(example main feature of instapaper is to view offline content). This should not be backed up since its redundant and when the phone needs to be restored(rarely) this data can be re-downloaded. Also backing up will be slow and destroy user experience. Solution make another location which is used for storing the data(not backed up) so then all are happy. Maybe you can have data from hidden directories not backed up inside the Documents directory. Then Apple can delete caches in apps like browsers. User critical data like settings can be backed up and app critical data that need not be backed up can also exist