3 ms·

It doesn't just replicate data off-site, it duplicates and distributes your data to other spacemonkey users. It's like P2P cloud storage.

peterwwillis 14y ago
...Yeah, that's what I said. Still sounds bad, for three reasons: bandwidth, redundancy and availability. Bandwidth is a simple consideration. In America we really haven't caught up to the bandwidth of most other 1st-world nations. Some reports give the average download speed of American broadband users at 7.1Mbps. Netflix reports the highest average broadband speed is with Google Fiber, at 3.45Mbps. Large swaths of the country still operate on 1.5Mbit connections or less, with upload caps being more like 256Kb for those users. Due to the disparity in speeds we can assume most of our transfers from home users would be limited to relatively slow speeds, not to mention (supposedly) using idle bandwidth. If you need your files, how long is it going to take to get them? Assuming something like 2.5Mbps down, and if there were five copies of your data transferred at 256Kbps, that's 160KBps. Downloading at that speed would take 77.6 days to download the entire archive. Then there's redundancy to consider. Disk drives are limited in capacity. Currently 4TB drives are available, with 5TB by the end of the year. To give each customer 1TB of storage while still storing other customer data on the device, you have to decide first how redundant you want the data. If you have a fancy algorithm you can store hundreds of clients' data on the device. Maybe they only have a couple small files, so they can be replicated in lots of places in a small amount of space. But eventually the files will grow in size, so you'll have to remove some redundant copies to make room for maximum capacity. Let's assume they use something like RAID-6, because RAID-5 is shitty for large capacity small arrays. Since we assume a 4TB hard drive, we can have N-2 storage, so for 4TB that's 2TB; make it N-3 storage and you can fit 1TB of user data and 3TB of random customer parity. With maximum capacity use, if three replicated copies of your data go away, your data goes away. How likely is this to happen? This[1] article explains how as drives increase in capacity they aren't increasing in reliability, which results in faster unrecoverable errors. As time goes by it's more likely your data will be lost. This is assuming your data is available. If their software engineers are smart they might develop an algorithm that can take into account actual used space and add or shrink the number of replications across the network based on availability. So even if you have used 1TB of data, as long as other people have extra free space, you can keep replicating copies of your data to more devices. Sounds good! Let's say your data is replicated on five devices. One device holding your data fails and is sent in for repair. Another is taken offline because the owner's internet connection is taken offline by a storm. Yet another is available, but is on a very slow connection. Your data is now on two devices. Sounds fine. In the event that your device is unavailable, you will be able to replicate your 1TB from the two people who hold parity copies of your data. How much data that is, and how long it takes to recover, and whether or not that transfer puts one of their devices over the limit for a URE, is subject several factors. Of course, there's many more factors to consider here, and i'm not an expert in any of them. But before you trust your data to a $10-a-month service, ask yourself: How reliable is the drive, how fast is the connection, and how much do you really need your data? [1] http://www.zdnet.com/blog/storage/why-raid-6-stops-working-in-2019/805 http://www.zdnet.com/blog/storage/why-raid-6-stops-working-i...
- rckclmbr 14y ago
  1. Each node is paired with nodes that are "alike", and bandwidth is considered. You should only be paired with other nodes that have bandwidth similar to you. 2. Restoring backups takes a while. It's just the way it is. But the client should be smart enough to get prioritized data first -- if you need to open a picture, it should fetch that first, so you can open it right away. 3. Files are broken apart and stored in chunks, so "growing size of files" is irrelevant. 4. The probability of your data being lost doesn't increase over time. The node is constantly checking the status of your backups. If a chunk is missing over a long period of time, it will back it up again. Redundancy is only part of the solution -- some missing chunks are recoverable through erasure encoding (like in the article you mentioned). Your node backs up to hundreds of other nodes, reducing risk of failure. Think of having 200 drives in an array instead of 7. 5. The article is somewhat FUD. Making projections for 2019 based on hardware speed where we're at now? Many embedded devices are already replacing ARM processors with dual core intel. As data sizes increase, our processing power will also increase.
  - SteveArmstrong 14y ago
    3. I think he was referring to aggregate file size (total amount stored) not a single file. If that's the case, his point still stands. 4. He was probably saying that, as the average age of the nodes in the network goes up, the chance of them failing goes up. Also, the duplication you say (hundreds of copies) is in direct opposition to usable space (hundreds of copies means you can only use 1/100 of the space on your node. Because of this, I don't think your files will ever be duplicated on the order of "hundreds", probably less than 10. Also, it's important to keep clear: My node will backup to hundreds of other nodes, but each individual file fragment will only be duplicated to 3 (or whatever) nodes. That smaller number is the important one that keeps getting discussed.
    - the_paul 14y ago
      There won't be hundreds of copies. A device may back up data to hundreds of other devices in total, though. Space Monkey data will be resilient to considerably more than 3 nodes failing.

peterwwillis 14y ago

...Yeah, that's what I said. Still sounds bad, for three reasons: bandwidth, redundancy and availability. Bandwidth is a simple consideration. In America we really haven't caught up to the bandwidth of most other 1st-world nations. Some reports give the average download speed of American broadband users at 7.1Mbps. Netflix reports the highest average broadband speed is with Google Fiber, at 3.45Mbps. Large swaths of the country still operate on 1.5Mbit connections or less, with upload caps being more like 256Kb for those users. Due to the disparity in speeds we can assume most of our transfers from home users would be limited to relatively slow speeds, not to mention (supposedly) using idle bandwidth. If you need your files, how long is it going to take to get them? Assuming something like 2.5Mbps down, and if there were five copies of your data transferred at 256Kbps, that's 160KBps. Downloading at that speed would take 77.6 days to download the entire archive. Then there's redundancy to consider. Disk drives are limited in capacity. Currently 4TB drives are available, with 5TB by the end of the year. To give each customer 1TB of storage while still storing other customer data on the device, you have to decide first how redundant you want the data. If you have a fancy algorithm you can store hundreds of clients' data on the device. Maybe they only have a couple small files, so they can be replicated in lots of places in a small amount of space. But eventually the files will grow in size, so you'll have to remove some redundant copies to make room for maximum capacity. Let's assume they use something like RAID-6, because RAID-5 is shitty for large capacity small arrays. Since we assume a 4TB hard drive, we can have N-2 storage, so for 4TB that's 2TB; make it N-3 storage and you can fit 1TB of user data and 3TB of random customer parity. With maximum capacity use, if three replicated copies of your data go away, your data goes away. How likely is this to happen? This[1] article explains how as drives increase in capacity they aren't increasing in reliability, which results in faster unrecoverable errors. As time goes by it's more likely your data will be lost. This is assuming your data is available. If their software engineers are smart they might develop an algorithm that can take into account actual used space and add or shrink the number of replications across the network based on availability. So even if you have used 1TB of data, as long as other people have extra free space, you can keep replicating copies of your data to more devices. Sounds good! Let's say your data is replicated on five devices. One device holding your data fails and is sent in for repair. Another is taken offline because the owner's internet connection is taken offline by a storm. Yet another is available, but is on a very slow connection. Your data is now on two devices. Sounds fine. In the event that your device is unavailable, you will be able to replicate your 1TB from the two people who hold parity copies of your data. How much data that is, and how long it takes to recover, and whether or not that transfer puts one of their devices over the limit for a URE, is subject several factors. Of course, there's many more factors to consider here, and i'm not an expert in any of them. But before you trust your data to a $10-a-month service, ask yourself: How reliable is the drive, how fast is the connection, and how much do you really need your data? [1] http://www.zdnet.com/blog/storage/why-raid-6-stops-working-in-2019/805 http://www.zdnet.com/blog/storage/why-raid-6-stops-working-i...

rckclmbr 14y ago
1. Each node is paired with nodes that are "alike", and bandwidth is considered. You should only be paired with other nodes that have bandwidth similar to you. 2. Restoring backups takes a while. It's just the way it is. But the client should be smart enough to get prioritized data first -- if you need to open a picture, it should fetch that first, so you can open it right away. 3. Files are broken apart and stored in chunks, so "growing size of files" is irrelevant. 4. The probability of your data being lost doesn't increase over time. The node is constantly checking the status of your backups. If a chunk is missing over a long period of time, it will back it up again. Redundancy is only part of the solution -- some missing chunks are recoverable through erasure encoding (like in the article you mentioned). Your node backs up to hundreds of other nodes, reducing risk of failure. Think of having 200 drives in an array instead of 7. 5. The article is somewhat FUD. Making projections for 2019 based on hardware speed where we're at now? Many embedded devices are already replacing ARM processors with dual core intel. As data sizes increase, our processing power will also increase.
- SteveArmstrong 14y ago
  3. I think he was referring to aggregate file size (total amount stored) not a single file. If that's the case, his point still stands. 4. He was probably saying that, as the average age of the nodes in the network goes up, the chance of them failing goes up. Also, the duplication you say (hundreds of copies) is in direct opposition to usable space (hundreds of copies means you can only use 1/100 of the space on your node. Because of this, I don't think your files will ever be duplicated on the order of "hundreds", probably less than 10. Also, it's important to keep clear: My node will backup to hundreds of other nodes, but each individual file fragment will only be duplicated to 3 (or whatever) nodes. That smaller number is the important one that keeps getting discussed.
  - the_paul 14y ago
    There won't be hundreds of copies. A device may back up data to hundreds of other devices in total, though. Space Monkey data will be resilient to considerably more than 3 nodes failing.