7 ms·

I'm not sure what value that would add here... The customer's issue seems to be that they have a very large number of files, and are trying to sync it with a 3

I'm not sure what value that would add here... The customer's issue seems to be that they have a very large number of files, and are trying to sync it with a 32-bit desktop client, and are hitting memory issues. Dropbox themselves state this on their websites: https://www.dropbox.com/help/space/file-storage-limit https://www.dropbox.com/help/space/file-storage-limit > Note: The performance of the Dropbox application may start to decline when you store above 300,000 files. I've done some comparisons across various companies sync clients, and it is a tricky problem. (Disclaimer: I work for Google).

Dylan16807 9y ago
The question is whether "declines" literally means "declines", or it's code for "breaks".
prirun 9y ago
1. Read list of files present on remote & write to file A 2. Read list of files present locally and write to file B 3. Sort files A & B 4. Read files A & B to update remote and local files This can be done with less than a meg a memory.
- TrainedMonkey 9y ago
  You are forgetting about monitoring for changed files to resync those quickly.
  - Dylan16807 9y ago
    Get notification of file change, binary search the list of files, make adjustments. This can also be done on a small amount of memory, independent of file count. I'm sure it's tricky to update the existing software, but as a generic problem syncing files with O(1) or O(logn) memory is not very hard.
    - Tempest1981 9y ago
      Windows' notification API gets an overflow (ERROR_NOTIFY_ENUM_DIR) if you change many files quickly, like during a copy/paste in explorer. You then have to re-scan the dir manually. While rescanning, the files may continue to change. You may or may not get a subsequent event for a file that changed during your rescan. One example of related issues: https://github.com/libuv/libuv/issues/626 https://github.com/libuv/libuv/issues/626
      - Dylan16807 9y ago
        I'm aware of how awful notification APIs can be, but that's a separate problem entirely from per-file memory use.
    - pjc50 9y ago
      Where's your sync client then? (I've been very slowly designing my own, and the first thing for me was coming up with a list of awkward cases ..)
      - dingo_bat 9y ago
        This seems like a good project. Thanks.
      - Dylan16807 9y ago
        > Where's your sync client then? I guess I was unclear here. It's hard to make a polished user application. The not-hard part is just the core algorithm selection. I have enough minutes in the day to consider algorithms, but not enough tens of hours to write suites of quality software products.
- victorhooi 9y ago
  The problem is unfortunately not as simple as a naive first pass might assume. There's a lot of nuance involved here - and the fact you have to do bi-directional sync, with a reasonable expectation of "real-time" makes it tough as well. Also - the fact that other desktop clients (e.g. Dropbox, Box.net) exhibit similar performance characteristics (e.g. Dropbox advertises a 300,000 file limit - I believe this customer was going on 700,000 files) would also probably indicate it's not quite that simple.
- Tempest1981 9y ago
  At 1 meg, are you not counting storage of the file names and paths? The user has 700000 files. Each is probably >10 char of UTF8. In step 4, you also want to catch things like the user moving 50000 files to a new subdir, or renaming a top level subdir, while the sync app is not running.
dingo_bat 9y ago
I'd understand if you don't want to discuss details but for the life of me I cannot understand how the total number of files can affect anything. What function does a sync app do which depends on total number of files? I can just do the sync file by file. It will take 10x more time to sync 100 files than 10 files. But why the brokenness?

Dylan16807 9y ago

The question is whether "declines" literally means "declines", or it's code for "breaks".

prirun 9y ago

1. Read list of files present on remote & write to file A 2. Read list of files present locally and write to file B 3. Sort files A & B 4. Read files A & B to update remote and local files This can be done with less than a meg a memory.

TrainedMonkey 9y ago
You are forgetting about monitoring for changed files to resync those quickly.
- Dylan16807 9y ago
  Get notification of file change, binary search the list of files, make adjustments. This can also be done on a small amount of memory, independent of file count. I'm sure it's tricky to update the existing software, but as a generic problem syncing files with O(1) or O(logn) memory is not very hard.
  - Tempest1981 9y ago
    Windows' notification API gets an overflow (ERROR_NOTIFY_ENUM_DIR) if you change many files quickly, like during a copy/paste in explorer. You then have to re-scan the dir manually. While rescanning, the files may continue to change. You may or may not get a subsequent event for a file that changed during your rescan. One example of related issues: https://github.com/libuv/libuv/issues/626 https://github.com/libuv/libuv/issues/626
    - Dylan16807 9y ago
      I'm aware of how awful notification APIs can be, but that's a separate problem entirely from per-file memory use.
  - pjc50 9y ago
    Where's your sync client then? (I've been very slowly designing my own, and the first thing for me was coming up with a list of awkward cases ..)
    - dingo_bat 9y ago
      This seems like a good project. Thanks.
    - Dylan16807 9y ago
      > Where's your sync client then? I guess I was unclear here. It's hard to make a polished user application. The not-hard part is just the core algorithm selection. I have enough minutes in the day to consider algorithms, but not enough tens of hours to write suites of quality software products.
victorhooi 9y ago
The problem is unfortunately not as simple as a naive first pass might assume. There's a lot of nuance involved here - and the fact you have to do bi-directional sync, with a reasonable expectation of "real-time" makes it tough as well. Also - the fact that other desktop clients (e.g. Dropbox, Box.net) exhibit similar performance characteristics (e.g. Dropbox advertises a 300,000 file limit - I believe this customer was going on 700,000 files) would also probably indicate it's not quite that simple.
Tempest1981 9y ago
At 1 meg, are you not counting storage of the file names and paths? The user has 700000 files. Each is probably >10 char of UTF8. In step 4, you also want to catch things like the user moving 50000 files to a new subdir, or renaming a top level subdir, while the sync app is not running.

dingo_bat 9y ago

I'd understand if you don't want to discuss details but for the life of me I cannot understand how the total number of files can affect anything. What function does a sync app do which depends on total number of files? I can just do the sync file by file. It will take 10x more time to sync 100 files than 10 files. But why the brokenness?