7 ms·

Not strictly on topic so I apologise if this is unwanted but I thought I'd share my experience with SpiderOak in case anyone here was thinking of purchasing one

by Veratyr 11y ago

Not strictly on topic so I apologise if this is unwanted but I thought I'd share my experience with SpiderOak in case anyone here was thinking of purchasing one of their plans. In February SpiderOak dropped its pricing to $12/month for 1TB of data. Having several hundred gigabytes of photos to backup I took advantage and bought a year long subscription ($129). I had access to a symmetric gigabit fibre connection so I connected, set up the SpiderOak client and started uploading. However I noticed something odd. According to my Mac's activity monitor, SpiderOak was only uploading in short bursts [0] of ~2MB/s. I did some test uploads to other services (Google Drive, Amazon) to verify that things were fine with my connection (they were) and then contacted support (Feb 10). What followed was nearly __6 months__ of "support", first claiming that it might be a server side issue and moving me "to a new host" (Feb 17) then when that didn't resolve my issue, they ignored me for a couple of months then handed me over to an engineer (Apr 28) who told me: "we may have your uploads running at the maximum speed we can offer you at the moment. Additional changes to storage network configuration will not improve the situation much. There is an overhead limitation when the client encrypts, deduplicates, and compresses the files you are uploading" At this point I ran a basic test (cat /dev/urandom | gzip -c | openssl enc -aes-256-cbc -pass pass:spideroak | pv | shasum -a 256 > /dev/zero) that showed my laptop was easily capable of hashing and encrypting the data much faster than SpiderOak was handling it (Apr 30) after which I was simply ignored for a full month until I opened another ticket asking for a refund (Jul 9). I really love the idea of secure, private storage but SpiderOak's client is barely functional and their customer support is rather bad. [0]: http://i.imgur.com/XEvhIop.png http://i.imgur.com/XEvhIop.png

Someone1234 11y ago
Many of these types of services seem to intentionally cap upload speeds to reduce their potential storage liability (since they're likely over-selling storage to be able to offer 1 TB for $12 with the level of redundancy, staffing costs, etc, needed). I wonder if that is happening in this specific case? Although if it were the case the vendor should still be honest about it. Just saying they limit uploads to 2 Mbps is better than giving the run-around.
- scurvy 11y ago
  It's definitely possible to offer that on a monthly basis if you model that each customer stays for 36-39 months. Also, I doubt that they are using replicated storage, but are using erasure coding instead. Also, they dedupe before upload, so more cost savings there.
  - Sir_Substance 11y ago
    Spideroak doesn't and cannot dedupe, since everything you upload is encrypted by a key held only by you.
    - daveguy 11y ago
      Spideroak can and does dedupe client side before uploading. It can't dedupe across multiple clients, but it does dedupe within the client. It also tracks syncs so that data synced between multiple client machines only has to be stored once (with appropriate redundancy).
      - evoloution 11y ago
        That explains why the data are visualized the way they are in the view menu :)
- Joona 11y ago
  Definitely looks like it to me. Took me a good month to back up my (video) files with CrashPlan, as it was using some 10% of my upload. I think it would be a good selling point for a service like this to allow higher upload speeds.
  - BillyParadise 11y ago
    A good upsell, yes. But initial seeding to "affordably priced" online services at full data rate can never be economically viable to the provider. Bandwidth is cheap(er) these days, but routers which can handle big bandwidth are still big bucks. Hold on, this is hacker news. VCs, this is a great idea! No, no of course it's not. Initial seeding is a competitive moat for the first mover. Moving a few hundred gigs to a new backup company just to save a few bucks? I don't think I could be bothered, because I KNOW how long it will take.
theandrewbailey 11y ago
I was going to post a comment about how cloud storage is more of a means to move data around rather than back it up, until I dug a little deeper and saw that SpiderOak actually pitches itself primarily as a backup provider. I agree, it needs to be much faster than that.
bluedino 11y ago
>> my laptop was easily capable of hashing and encrypting the data much faster than the network was capable of handling it You are assuming that you are the only one using that uplink and that server
- Veratyr 11y ago
  Updated my comment: > easily capable of hashing and encrypting the data much faster than SpiderOak was handling it I can believe that there was upstream congestion somewhere outside my network (speeds to Google, Amazon indicated that there were no issues inside) or that their server was overloaded but the engineer who investigated seemed to attribute it to the client: > Additional changes to storage network configuration will not improve the situation much. There is an overhead limitation when the client encrypts, deduplicates, and compresses the files you are uploading"
- deleted 11y ago
  [deleted]
draw_down 11y ago
This comment is ridiculous, and so is the fact that it's at the top. This is supposed to be about Google Analytics, come on.
- y4mi 11y ago
  well, the post is from SpiderOak, so its understandable. but its an ad hominem argument, thats for sure.
- cpach 11y ago
  It is off-topic, yes. For me personally it was very valuable however since I’m in the market for a backup application, and I will definitely take Veratyr’s comment into consideration when choosing between the available offerings.
jasonsync 11y ago
Plug: https://www.sync.com https://www.sync.com
- Veratyr 11y ago
  Love the pricing and features but Win+Mac only and no API largely kills it for me as I need Linux access at the very least.
- newscracker 11y ago
  The only issue I have, which is similar to what I see with some other providers, is that the first non-free plan is a huge jump in storage space and price. If I want a Dropbox replacement, I'd be looking at a 25GB or 50GB plan (just comparing what I have with all kinds of free storage bonuses accumulated over years). Having some more "in-between" plans that are more linear in storage and price would've been an incentive to try this out since I'm not willing to fork $49 a year for 500GB while knowing that my Dropbox usage is less than one-tenth of that.
kbenson 11y ago
Is it possible that they are working on batches, and not doing any hashing/compression in parallel with the uploading? It seems feasible from your screenshot that they are getting ~10GB of data at a time, compressing(?) and hashing, and then uploading, and then starting on the next ~10GB.
mark_l_watson 11y ago
That doesn't sound good. On the other hand, I use SpiderOak with not a lot of cloud storage use, with clients on OS X, Linux, and until this morning Windows 10. The only problem I ever had was more or less my fault - trying to register a new laptop with a previously named setup. BTW, why store photos and videos on encrypted storage? For that I use Office 365's OneDrive: everyone in my family gets a terabyte for $99/year and I really like the web versions of Office 365 because when I am on Linux and someone sends me an EXCEL or WORD file, no problem, and I don't use up local disk space (with SSD drives, something to consider).
- Veratyr 11y ago
  I prefer to store photos and videos on encrypted storage because I want to control who sees them. Storing them on unencrypted storage means I don't have that control, the storage provider does and is kind enough to let me make suggestions. As for OneDrive, I tried it for a while but it didn't work out. Their clients and web interface were terrible and their API was severely lacking. I expect more functionality when I'm sacrificing my privacy. I ended up going with Google Drive in the end, as you can get 1TB for $9/month with an Apps for Work Unlimited account (I actually seem to have Unlimited under that plan, which isn't supposed to happen until 4 users). That of course means sacrificing encryption but I trust Google enough to make the privacy tradeoff in exchange for extra features (OCR, Google Photos etc.).
  - mark_l_watson 11y ago
    I also buy extra storage from Google but I have had some problems downloading large backup files (50 GB, or so) that I have stored on Google Drive, so no system is perfect. A little off topic, but Google really seems to be upping their consumer game lately with Google Music, Youtube Red, Google Movies + TV, etc. I am now less a user of other services like GMail and Search, but Google gets those monthly consumer app payments from me. I have the same kind of praise for Microsoft with Office 365.
ldehaan 11y ago
This has been my experience as well, not to mention how much the client slowed down my machine. It's been really slow going but the client is getting better. I never tried doing the encryption on my side, though, they also do diffs on each file you upload so I imagine that has something to do with the lag. I still use spideroak, they're the only company I'm aware of that encrypts locally and also has done a lot to progress personal security for all of us. So I've gotten used to the slow speeds and buggy software, it keeps getting better so that's a big plus :)
- cpach 11y ago
  There are other backup applications that encrypt locally before sending to server. Two examples are https://www.tarsnap.com/ https://www.tarsnap.com/ and https://www.haystacksoftware.com/ https://www.haystacksoftware.com/
Paul-ish 11y ago
Could the issue be caused by bad peering between your ISPs?
- Veratyr 11y ago
  If that was the case I'd expect the upload to be consistent but slow. Since it was intermittent, I believe it's an app issue.

by Veratyr 11y ago

Someone1234 11y ago

Many of these types of services seem to intentionally cap upload speeds to reduce their potential storage liability (since they're likely over-selling storage to be able to offer 1 TB for $12 with the level of redundancy, staffing costs, etc, needed). I wonder if that is happening in this specific case? Although if it were the case the vendor should still be honest about it. Just saying they limit uploads to 2 Mbps is better than giving the run-around.

scurvy 11y ago
It's definitely possible to offer that on a monthly basis if you model that each customer stays for 36-39 months. Also, I doubt that they are using replicated storage, but are using erasure coding instead. Also, they dedupe before upload, so more cost savings there.
- Sir_Substance 11y ago
  Spideroak doesn't and cannot dedupe, since everything you upload is encrypted by a key held only by you.
  - daveguy 11y ago
    Spideroak can and does dedupe client side before uploading. It can't dedupe across multiple clients, but it does dedupe within the client. It also tracks syncs so that data synced between multiple client machines only has to be stored once (with appropriate redundancy).
    - evoloution 11y ago
      That explains why the data are visualized the way they are in the view menu :)
Joona 11y ago
Definitely looks like it to me. Took me a good month to back up my (video) files with CrashPlan, as it was using some 10% of my upload. I think it would be a good selling point for a service like this to allow higher upload speeds.
- BillyParadise 11y ago
  A good upsell, yes. But initial seeding to "affordably priced" online services at full data rate can never be economically viable to the provider. Bandwidth is cheap(er) these days, but routers which can handle big bandwidth are still big bucks. Hold on, this is hacker news. VCs, this is a great idea! No, no of course it's not. Initial seeding is a competitive moat for the first mover. Moving a few hundred gigs to a new backup company just to save a few bucks? I don't think I could be bothered, because I KNOW how long it will take.

theandrewbailey 11y ago

I was going to post a comment about how cloud storage is more of a means to move data around rather than back it up, until I dug a little deeper and saw that SpiderOak actually pitches itself primarily as a backup provider. I agree, it needs to be much faster than that.

bluedino 11y ago

>> my laptop was easily capable of hashing and encrypting the data much faster than the network was capable of handling it You are assuming that you are the only one using that uplink and that server

Veratyr 11y ago
Updated my comment: > easily capable of hashing and encrypting the data much faster than SpiderOak was handling it I can believe that there was upstream congestion somewhere outside my network (speeds to Google, Amazon indicated that there were no issues inside) or that their server was overloaded but the engineer who investigated seemed to attribute it to the client: > Additional changes to storage network configuration will not improve the situation much. There is an overhead limitation when the client encrypts, deduplicates, and compresses the files you are uploading"
deleted 11y ago
[deleted]

draw_down 11y ago

This comment is ridiculous, and so is the fact that it's at the top. This is supposed to be about Google Analytics, come on.

y4mi 11y ago
well, the post is from SpiderOak, so its understandable. but its an ad hominem argument, thats for sure.
cpach 11y ago
It is off-topic, yes. For me personally it was very valuable however since I’m in the market for a backup application, and I will definitely take Veratyr’s comment into consideration when choosing between the available offerings.

jasonsync 11y ago

Plug: https://www.sync.com https://www.sync.com

Veratyr 11y ago
Love the pricing and features but Win+Mac only and no API largely kills it for me as I need Linux access at the very least.
newscracker 11y ago
The only issue I have, which is similar to what I see with some other providers, is that the first non-free plan is a huge jump in storage space and price. If I want a Dropbox replacement, I'd be looking at a 25GB or 50GB plan (just comparing what I have with all kinds of free storage bonuses accumulated over years). Having some more "in-between" plans that are more linear in storage and price would've been an incentive to try this out since I'm not willing to fork $49 a year for 500GB while knowing that my Dropbox usage is less than one-tenth of that.

kbenson 11y ago

Is it possible that they are working on batches, and not doing any hashing/compression in parallel with the uploading? It seems feasible from your screenshot that they are getting ~10GB of data at a time, compressing(?) and hashing, and then uploading, and then starting on the next ~10GB.

mark_l_watson 11y ago

That doesn't sound good. On the other hand, I use SpiderOak with not a lot of cloud storage use, with clients on OS X, Linux, and until this morning Windows 10. The only problem I ever had was more or less my fault - trying to register a new laptop with a previously named setup. BTW, why store photos and videos on encrypted storage? For that I use Office 365's OneDrive: everyone in my family gets a terabyte for $99/year and I really like the web versions of Office 365 because when I am on Linux and someone sends me an EXCEL or WORD file, no problem, and I don't use up local disk space (with SSD drives, something to consider).

Veratyr 11y ago
I prefer to store photos and videos on encrypted storage because I want to control who sees them. Storing them on unencrypted storage means I don't have that control, the storage provider does and is kind enough to let me make suggestions. As for OneDrive, I tried it for a while but it didn't work out. Their clients and web interface were terrible and their API was severely lacking. I expect more functionality when I'm sacrificing my privacy. I ended up going with Google Drive in the end, as you can get 1TB for $9/month with an Apps for Work Unlimited account (I actually seem to have Unlimited under that plan, which isn't supposed to happen until 4 users). That of course means sacrificing encryption but I trust Google enough to make the privacy tradeoff in exchange for extra features (OCR, Google Photos etc.).
- mark_l_watson 11y ago
  I also buy extra storage from Google but I have had some problems downloading large backup files (50 GB, or so) that I have stored on Google Drive, so no system is perfect. A little off topic, but Google really seems to be upping their consumer game lately with Google Music, Youtube Red, Google Movies + TV, etc. I am now less a user of other services like GMail and Search, but Google gets those monthly consumer app payments from me. I have the same kind of praise for Microsoft with Office 365.

ldehaan 11y ago

This has been my experience as well, not to mention how much the client slowed down my machine. It's been really slow going but the client is getting better. I never tried doing the encryption on my side, though, they also do diffs on each file you upload so I imagine that has something to do with the lag. I still use spideroak, they're the only company I'm aware of that encrypts locally and also has done a lot to progress personal security for all of us. So I've gotten used to the slow speeds and buggy software, it keeps getting better so that's a big plus :)

cpach 11y ago
There are other backup applications that encrypt locally before sending to server. Two examples are https://www.tarsnap.com/ https://www.tarsnap.com/ and https://www.haystacksoftware.com/ https://www.haystacksoftware.com/

Paul-ish 11y ago

Could the issue be caused by bad peering between your ISPs?

Veratyr 11y ago
If that was the case I'd expect the upload to be consistent but slow. Since it was intermittent, I believe it's an app issue.