2 ms·

That's how we designed the original 1Password data format (.agilekeychain). It certainly made syncing with Dropbox much simpler. It does have its drawbacks tho

That's how we designed the original 1Password data format (.agilekeychain). It certainly made syncing with Dropbox much simpler. It does have its drawbacks though. Once you have too many files (and make too many requests), both Dropbox and iCloud will start throttling you. It also might take a while to reload the data, even from the local disk. We had to add a cache file at some point.

fencepost 9y ago
I could see either local caching or (assuming the storage backend provides access to file metadata like date/time stamps that are internally consistent) I can think of several ways to have any instance of the application consolidate those small index files into larger ones such that you'd mostly need to load the consolidated file plus any individual entries modified since its timestamp. If those consolidated files were appropriately named you could even have multiple instances creating them at the same time without causing collisions. Cleanup could be a little trickier, but could likely be done with very little risk as long as a little bit of storage bloat wasn't a big concern. Interesting thought experiment, thanks for mentioning your real-world experiences with it.
- AGKyle 9y ago
  Yea, our current solution, as Roustem mentioned, sort of recreates the Dropbox app on Mac. We replicate the entire keychain file on the device, then perform syncing to keep both up to date. Our newer OPVault format uses band files, each band file is named after the first character in each item's UUID, which means we have far fewer files than AgileKeychain (individual files for each item). They both have their pitfalls, but initial sync with OPVault is infinitely faster as number of items increase. We did a ton of testing when we switched to the new Dropbox API and we had AgileKeychain's with thousands of items in them. They took awhile to sync. The same number of items in OPVault was nearly instant, at least with regard to the replication portion. Sync still took a little while due to the encryption and decryption that had to happen. Sync with 3rd party sync services have always been quite difficult for us. Especially from a troubleshooting standpoint. It's hard to replicate certain issues because the data is incredibly sensitive for users so we can't "get" the data and instead have to try to duplicate the issue with limited information. Every day is certainly a unique challenge :) Kyle AgileBits

fencepost 9y ago

I could see either local caching or (assuming the storage backend provides access to file metadata like date/time stamps that are internally consistent) I can think of several ways to have any instance of the application consolidate those small index files into larger ones such that you'd mostly need to load the consolidated file plus any individual entries modified since its timestamp. If those consolidated files were appropriately named you could even have multiple instances creating them at the same time without causing collisions. Cleanup could be a little trickier, but could likely be done with very little risk as long as a little bit of storage bloat wasn't a big concern. Interesting thought experiment, thanks for mentioning your real-world experiences with it.

AGKyle 9y ago
Yea, our current solution, as Roustem mentioned, sort of recreates the Dropbox app on Mac. We replicate the entire keychain file on the device, then perform syncing to keep both up to date. Our newer OPVault format uses band files, each band file is named after the first character in each item's UUID, which means we have far fewer files than AgileKeychain (individual files for each item). They both have their pitfalls, but initial sync with OPVault is infinitely faster as number of items increase. We did a ton of testing when we switched to the new Dropbox API and we had AgileKeychain's with thousands of items in them. They took awhile to sync. The same number of items in OPVault was nearly instant, at least with regard to the replication portion. Sync still took a little while due to the encryption and decryption that had to happen. Sync with 3rd party sync services have always been quite difficult for us. Especially from a troubleshooting standpoint. It's hard to replicate certain issues because the data is incredibly sensitive for users so we can't "get" the data and instead have to try to duplicate the issue with limited information. Every day is certainly a unique challenge :) Kyle AgileBits