6 ms·
If you're cynical you'd think "of course they'd offer a free version and then make you pay for it." But that's basically the cloud storage market in a nutshell.
by klodolph 9y ago
If you're cynical you'd think "of course they'd offer a free version and then make you pay for it." But that's basically the cloud storage market in a nutshell. All the cloud storage providers are in a race to the bottom, and once they have your data they want to upsell you on other cloud technologies. If you just need to store data, that's actually great news, because these providers are working hard on the optimizations that make it possible to give you storage as cheap as they can.
Cheap options are Glacier at $0.004/GB, Backblaze at $0.005/GB, GCS Coldline at $0.007/GB. That's the monthly cost for bare cloud storage with no egress. Anything cheaper than about $50/year for 1TB will fall in one of two categories:
1. Subsidized by the provider, but probably not for long. This has happened so many times I can't even remember them all, Amazon is just the most recent.
2. Strictly DIY, probably with a much higher chance of data loss than you realize.
- cm2187 9y agoIf it is backup, it doesn't need to be super safe. The risk that you lose both your primary data, (optionally your local backup) and your online backup at the same time is pretty insignificant, given that the online backup will be uncorrelated as in a different physical location (primary and local backup very correlated I agree).
- himlion 9y agoThe problem is sometimes you only find out your backup is corrupted when you need it.
- cm2187 9y agoYeah, agree. And I am not talking about enterprise level of reliability. More like personnal / small office. But for anything above a few TB, running your own hardware will be way cheaper if you take say a 5y horizon. Disks are so large today that you don't need a big config. In fact you might even keep two copies to reduce the risk. Of course there is the occasional trip to the datacentre. I made that mistake. I pay more in uber than I saved by picking a datacentre far away.
- w458cmau 9y agoDo you need co-location though? I dropped an HP Microserver at a friend's place, which backs up snapshots of my main backup server at home daily. Seems to work well so far, at quite limited costs. Since my main backup server also backs up his NAS we both win. Since all backups are encrypted there is also little privacy risk from theft (or snooping on one another).
- cm2187 9y agoWith 1Gbps symetrical connections slowly becoming more frequent this would be a much cheaper alternative (assuming you only run it for backups, I also run a mail server and some websites). But in London it is not very practical. I know very few people who have really fast broadband.
- qeternity 9y agoIf you don't test your backups, then they're not backups.
- jjn2009 9y agoyup, the assertion that both failing is highly unlikely is under the assumption that you are checking the status of each copy regularly.
- cm2187 9y agoIn fact for any disk, it is important to have a script that reads all the data at least every couple of months. It will force the bad sectors to be identified or to be notified you have a bad disk before things get worse.
- mongol 9y agoI would argue the opposite. When you reach for your backup, you are in a dire situation. If it fails you then, things will be very gloom.
- klodolph 9y ago> The risk that you lose both your primary data, (optionally your local backup) and your online backup at the same time is pretty insignificant, given that the online backup will be uncorrelated as in a different physical location (primary and local backup very correlated I agree). Actually, that reasoning is exactly what I wanted to talk about. When the primary is lost, there is a much higher chance than you'd expect that a secondary contains unrecoverable errors. This is why RAID 5 arrays fail to rebuild after a disk failure so often—they're supposed to be able to tolerate a single disk failing, but they can't tolerate a disk failing and any other IO error at the same time. Part of this is due to how short the timeout is for failed reads in RAID setups, but I've still seen a lot of RAID 5 arrays fail, and I've seen a few RAID 6 arrays fail too. On top of that, there's the high chance of configuration errors in DIY systems.
- cm2187 9y agoAgree. Having a script doing a full read of all the data every couple of months and sending a report by email (that you would notice if not sent) is a sort of must have.
- w458cmau 9y agoI am still looking for a good way of doing this. Weekly integrity check of all backed up data. E-Mail which informs of result. Web interface which shows overview of results of historical checks. External service which sends E-Mail if integrity check failed to run (e.g. https://deadmanssnitch.com/ https://deadmanssnitch.com/).
- cm2187 9y agoIntegrity is harder because you have to sort of maintain a signature of every file on the side. What I do is to just have a script that reads all data. If some data is unreadable an exception will be thrown and a text message will be sent to me. And a confirmation email is sent otherwise. In .net it's only a few lines of codes. Haven't thought of deadmansnitch but it's a good idea. Would just take one more line of code.
- 9y ago
- jorangreef 9y agoIf it is backup, then it needs to be scrubbed regularly to detect and repair bit rot.
- jo909 9y agoMy backup contains files and old versions of files I no longer have on the primary storage. It is much more than a 1:1 copy, and can not be recreated. I need to feel safe that I can reasonably undo my changes on the primary storage, even if it takes me years to realise what I did. This is why I keep two backups. Until now, the secondary offsite copy was on ACD...
- dsr_ 9y agoThat's not a backup. That's an archive. You need to backup archives the same way you do any other data you care about.
- glenneroo 9y agoSo how would you categorize CrashPlan? They offer unlimited storage for $60/year. I've had around 8TB backed up there for the last 2-3 years. They have existed since 2001 (no idea how long the unlimited feature has existed). On the other hand, supporting your point #1, I used to have several TB on StreamLoad which 10ish years ago used to also offer unlimited... until they suffered a critical hardware failure and lost tons of customer data. They then changed to MediaMax, then suddenly disappeared. Perhaps you know of others, or this list is incomplete, but Wikipedia has a list of file services, of which there aren't many who offer an unlimited plan: https://en.wikipedia.org/wiki/Comparison_of_file_hosting_services https://en.wikipedia.org/wiki/Comparison_of_file_hosting_ser...
- ghshephard 9y agoCrashplan is a backup service. Different category of company.
- klodolph 9y agoCrashPlan is "unlimited" but it's not a storage service, it's a backup service. From what I understand, you have to use their client software, and that software will use RAM proportional to the total size of the data you're storing. In practice this puts a hard limit on the amount of data which is proportional to the amount of RAM you have. So heavy users are not only capped, they're also subsidized by the light users.
- freeone3000 9y agoI've got about 12TB on Crashplan, and the client's never used more than about 2GB. I doubt your "cap" applies in practice.
- nucleardog 9y agoSo about 170mb/TB if this scales linearly for some reason. Some users were apparently bragging about having 1PB+ on ACD. That would mean the RAM usage would actually be a useful upper bound, as I doubt many of these guys have 170GB of RAM in their media server PCs.
- samnwa 9y agoEver hear of Sia? That's a new decentralized option gunning for Amazon. Website is sia.tech.
- klodolph 9y agoIt's very difficult to take Sia seriously. For one thing, Sia only has 25T of data. That's total data, in the entire global network, including parity blocks. And when I checked prices today, it was more expensive than Glacier—but I can only imagine how much the prices would swing if you suddenly bought 2.5T of storage. Sia's decentralization is not unique. Amazon, Google, and Microsoft cloud storage are also decentralized. The idea of reselling unused bytes on disks that you already own is also not unique—again, Amazon, Google, and Microsoft already do this.
- Taek 9y agoSia's decentralization is unique. It's unique not because it spreads data all over the world, but because there's no central player that controls prices or decides to shut things down. Amazon, Google, Microsoft, they can change terms or disable services whenever they want. Sia is a lot more robust, because it's governed via a blockchain, and you put your data on 30+ individual parties, making it extremely unlikely that a sufficient number of them all shutdown simultaneously. This situation with retiring unlimited storage on ACD would not be possible with Sia. Amazon is changing terms and conditions, Sia is a blockchain where no party has the power to do that. ----- To address your pricing comment, there are more than 1000 TB for sale on the Sia network. Adding 2.5 TB to the network would not move the price at all, that's really not how pricing works on Sia. The reason that it's more expensive than it used to be is because prices are set entirely in Siacoin. When the siacoin prices rise, hosts need to manually re-adjust their prices to keep the same USD price. As of writing, there are no tools that will let this happen automatically. The siacoin price has doubled 6 times in 6 months. The result is that a lot of the defaults are now grossly expensive, and hosts need to be very familiar with the pricing mechanisms to respond accordingly. Most have not, though the ones that have are seeing much higher utilization than the ones that haven't. We will be releasing stuff in the next month to help hosts set prices more intelligently, that should move the prices back to the competitive spot that they historically been at. The high prices right now are merely a result of market confusion among the hosts.
- kefka 9y agoBefore you mention "Cheap options" and Amazon Glacier, read this: https://news.ycombinator.com/item?id=10921365 https://news.ycombinator.com/item?id=10921365 "How I ended up paying $150 for a single 60GB download from Amazon Glacier" > If you’re retrieving more than 5% of your stuff, expect to pay a fee, “starting at $0.011 per gigabyte”. Key words here is "starting at". as in, no cap on how much they can and will charge you for your data. Cause you know, 60GB is realllly expensive to transfer :( _______________________________ What Glacier seems to be for, is more like compliance paperwork and other "We are required to save this for X years" kind of paperwork. Because most of those types of documents, you don't care about and only legally have to keep available. And if a court case comes along, the Glacier retrieval price can just be added on top of court fees (which in that case, Glacier is probably a pittance even at the $10k level).
- klodolph 9y agoThere's a lot of articles about these things that seem to have the same summary: "I didn't read the documentation for how pricing works and then I was surprised by how pricing worked." Some people were shocked by Glacier's egress pricing and others were shocked by the cost per file. Both of these are approximately zero, most of the time, for most people, if they know what they're doing. If you don't know what you're doing and haven't read the docs you shouldn't be using Glacier. > Cause you know, 60GB is realllly expensive to transfer :( Network and disk IO capacity planning is hard. You can't just buy GBs of egress at $0.01 and then resell them at $0.01, because you're paying for maximum capacity and turning around and selling average usage. Similarly, when you have a bunch of data shoved into unused sections of disk, you can't just read them back out without affecting whatever else is reading from the same disk. If you want to sell something close to cost, you need to reflect the pricing structure of what you're paying to your customers. So if you upload a bunch of data to a super-cold storage system that's even named after a geological formation that stays frozen for centuries, to remind you how cold it is, and then you make it hot by trying to download it all at once, you'd expect it to be more expensive. Just do your regular cost-benefit analysis and it should be fine, based on how long you plan on storing the data and how likely or often you think you'll need to restore it.
- 9y ago
- prirun 9y agoHave to be careful with Glacier: the retrieval costs can be very high and are extremely hard to calculate. Amazon S3 Infrequent Access has higher retrieval costs, though not as crazy as Glacier, and a delete penalty. Deleting a file after 1 day will cost 30x normal S3 storage fees. Likewise, Coldline sounds great, but they have higher retrieval fees (not so bad) and a delete penalty. The Coldline delete penalty will cost 90x the regular storage cost if a file is deleted the next day. I don't recommend Coldline unless you know you won't be deleting files before 90 days.