4 ms·

Hey there - I went down a similar road as you about a year back when I looked for a file backup solution. I started with Dropbox, and attempted a sync of my lo

Hey there - I went down a similar road as you about a year back when I looked for a file backup solution. I started with Dropbox, and attempted a sync of my local files. Within minutes the Windows app crashed. I switched to Google Drive, same thing. Synced a large number of files and the Windows software crashed. Even Tried Insync, still had crashing issues. Researched the issue and found that both clients are limited in the amount of RAM (32bit vs 64bit) and this limit is easily reached when syncing tens-of-thousands of files causing the software to crash. Neither provider offers a 64 bit version, or a version of their client software that handles syncing jobs this large. I finally settled on Backblaze which was affordable ($5/month), and "just worked". The files aren't as accessible as Dropbox or Drive, but I still use those for the files I need/touch often, and use Backblaze for my "entire PC" cloud backup solution.

mcgrath_sh 10y ago
I was sold on Backblaze. Until it would continually time out when backing up my 5TB pseudo-server/HTPC. And then it would restart. Just as it reached 5TB. I think I have finally found my solution with rclone (I moved to a Linux system after Backblaze was a no-go) and Amazon Cloud Drive. All my data is encrypted and it is trivially easy to add fresh backup copies. All of this is mostly for my media. All non-media is in my dropbox account and I back up to a physical drive. I used to be high on BackBlaze (I had issues with CrashPlan taking forever too, BackBlaze was faster) but when it would continually just have my backups magically disappear, I left.
- josefresco 10y ago
  I only backup about 1.5TB (4.4M files) so yeah - it seems every tool has it's limits.
- glenneroo 10y ago
  I've been able to backup close to 1 million files (7.7TB) to CrashPlan, granted I had to manually edit the .ini file multiple times to increase the Java Heap memory allocation to allow that many files. At least the fix is documented on their support page (although I still can't figure out why after however many years this bug exists, they don't just add an option in the software directly). The only problem I have with them is that they have an upload limit, which changes from day to day, but over here in Europe I get between 100-1000 KB/s, which according to support is throttled depending on how many users are currently active. They offered to let me send them a hard-drive... to their US address, haha! It took about 2 years of leaving my PC on 24/7 to finally back everything up, let's hope they don't go the same way as StreamLoad (MediaMax), where I lost ~2TB of time-lapse footage due to being unable to download everything before the service went offline. It's even documented on Wikipedia (https://en.wikipedia.org/wiki/The_Linkup https://en.wikipedia.org/wiki/The_Linkup). p.s. I made this comparison of cloud image backup solutions back in 2012 (https://docs.google.com/spreadsheets/d/1mWx6wbL6NYbhjY1_ISNapAgXbi-JuKJJaF2K5RPAZLk/edit?usp=sharing https://docs.google.com/spreadsheets/d/1mWx6wbL6NYbhjY1_ISNa...), it's crazy how many of these services are now defunct.
  - mcgrath_sh 10y ago
    That is a really long time. I was able to back up most of my media in a little over a week and a half...
    - glenneroo 10y ago
      Yeah it was painful. Thankfully ~50% of that time was at the office, where I don't pay the electricity bill. A lot of the latter half was using a very unreliable DSL connection, which, due to rotting copper cables (and multiple chew-throughs from rats), I only get about half of the promised throughput, so upstream is limited to ~120 KB/s at best. I finally caved in and bought an LTE modem last year which gets me ~1 MB/s (on good days).
  - kalleboo 10y ago
    I can't imagine having the patience to wait 2 years(!!) to upload everything. It took about 24 hours to backup my internal 750 GB drive to Backblaze... At that point I'd just build my own system that uploaded to Glacier or something.
unabst 10y ago
I looked at Backblaze, but they haven't sold themselves as a sync platform. If I just needed backup I'd use them immediately! My main machine has 32GB of RAM, and I bet this helps alleviate the burden of large sync tasks, but here is a link to the issue Google Drive has: https://productforums.google.com/forum/#!topic/drive/mo298Ko1vUY https://productforums.google.com/forum/#!topic/drive/mo298Ko... > If you happen to have a Mac available to do these uploads, this limitation should not exist in the Mac version of the client. Those are the words of a Google employee. Offering a product that is fundamentally flawed is unacceptable -- Google or otherwise. > We've released a new version Didn't work for me. Then I started getting other errors, so I gave up. Pasting below for reference: ------------- Ken Watford said: Error D41D on Windows is a memory error. The Windows version of the product currently has difficulty syncing large numbers of files at once due to the memory constraints on 32-bit applications, even if your machine has a lot of memory. The number of files it can handle at once varies a bit, but it can't reliably do a million at once. The actual size of the files isn't much of a factor. I would recommend moving those files back out of your local Drive folder and starting the app. Once it has settled, move them in a little at a time, preferably fewer than 100k at once, waiting for the sync to complete before the next batch. The limitation is not on the number of files synced, but the number of files actively syncing. If you later need to sync all these files down to another Windows machine, you will probably need to use the Selective Sync option to only sync down a few folders at a time, adding more later. If you happen to have a Mac available to do these uploads, this limitation should not exist in the Mac version of the client.
- josefresco 10y ago
  "preferably fewer than 100k at once" Lulz - this is false/ won't work. I tried 50K, 10K, 5K, 1K - still crashed. Manually syncing my 4.4M files <1K at-a-time is insanity.

mcgrath_sh 10y ago

I was sold on Backblaze. Until it would continually time out when backing up my 5TB pseudo-server/HTPC. And then it would restart. Just as it reached 5TB. I think I have finally found my solution with rclone (I moved to a Linux system after Backblaze was a no-go) and Amazon Cloud Drive. All my data is encrypted and it is trivially easy to add fresh backup copies. All of this is mostly for my media. All non-media is in my dropbox account and I back up to a physical drive. I used to be high on BackBlaze (I had issues with CrashPlan taking forever too, BackBlaze was faster) but when it would continually just have my backups magically disappear, I left.

josefresco 10y ago
I only backup about 1.5TB (4.4M files) so yeah - it seems every tool has it's limits.
glenneroo 10y ago
I've been able to backup close to 1 million files (7.7TB) to CrashPlan, granted I had to manually edit the .ini file multiple times to increase the Java Heap memory allocation to allow that many files. At least the fix is documented on their support page (although I still can't figure out why after however many years this bug exists, they don't just add an option in the software directly). The only problem I have with them is that they have an upload limit, which changes from day to day, but over here in Europe I get between 100-1000 KB/s, which according to support is throttled depending on how many users are currently active. They offered to let me send them a hard-drive... to their US address, haha! It took about 2 years of leaving my PC on 24/7 to finally back everything up, let's hope they don't go the same way as StreamLoad (MediaMax), where I lost ~2TB of time-lapse footage due to being unable to download everything before the service went offline. It's even documented on Wikipedia (https://en.wikipedia.org/wiki/The_Linkup https://en.wikipedia.org/wiki/The_Linkup). p.s. I made this comparison of cloud image backup solutions back in 2012 (https://docs.google.com/spreadsheets/d/1mWx6wbL6NYbhjY1_ISNapAgXbi-JuKJJaF2K5RPAZLk/edit?usp=sharing https://docs.google.com/spreadsheets/d/1mWx6wbL6NYbhjY1_ISNa...), it's crazy how many of these services are now defunct.
- mcgrath_sh 10y ago
  That is a really long time. I was able to back up most of my media in a little over a week and a half...
  - glenneroo 10y ago
    Yeah it was painful. Thankfully ~50% of that time was at the office, where I don't pay the electricity bill. A lot of the latter half was using a very unreliable DSL connection, which, due to rotting copper cables (and multiple chew-throughs from rats), I only get about half of the promised throughput, so upstream is limited to ~120 KB/s at best. I finally caved in and bought an LTE modem last year which gets me ~1 MB/s (on good days).
- kalleboo 10y ago
  I can't imagine having the patience to wait 2 years(!!) to upload everything. It took about 24 hours to backup my internal 750 GB drive to Backblaze... At that point I'd just build my own system that uploaded to Glacier or something.

unabst 10y ago

I looked at Backblaze, but they haven't sold themselves as a sync platform. If I just needed backup I'd use them immediately! My main machine has 32GB of RAM, and I bet this helps alleviate the burden of large sync tasks, but here is a link to the issue Google Drive has: https://productforums.google.com/forum/#!topic/drive/mo298Ko1vUY https://productforums.google.com/forum/#!topic/drive/mo298Ko... > If you happen to have a Mac available to do these uploads, this limitation should not exist in the Mac version of the client. Those are the words of a Google employee. Offering a product that is fundamentally flawed is unacceptable -- Google or otherwise. > We've released a new version Didn't work for me. Then I started getting other errors, so I gave up. Pasting below for reference: ------------- Ken Watford said: Error D41D on Windows is a memory error. The Windows version of the product currently has difficulty syncing large numbers of files at once due to the memory constraints on 32-bit applications, even if your machine has a lot of memory. The number of files it can handle at once varies a bit, but it can't reliably do a million at once. The actual size of the files isn't much of a factor. I would recommend moving those files back out of your local Drive folder and starting the app. Once it has settled, move them in a little at a time, preferably fewer than 100k at once, waiting for the sync to complete before the next batch. The limitation is not on the number of files synced, but the number of files actively syncing. If you later need to sync all these files down to another Windows machine, you will probably need to use the Selective Sync option to only sync down a few folders at a time, adding more later. If you happen to have a Mac available to do these uploads, this limitation should not exist in the Mac version of the client.

josefresco 10y ago
"preferably fewer than 100k at once" Lulz - this is false/ won't work. I tried 50K, 10K, 5K, 1K - still crashed. Manually syncing my 4.4M files <1K at-a-time is insanity.