12 ms·
Introducing Google Cloud Storage Nearline
- fivedogit 12y agoSome of these Google/Amazon/Apple "Introducing Product X..." stories are starting to strike me as intellectually devoid, creeping into "free advertising" territory. This particular story is about a Google service that is an extremely late entrant into the market (Amazon Glacier, etc) and offers almost nothing new. I'm not saying it's a bad product or not useful to folks, but it's not, by any means, groundbreaking. I'd much rather see the top spot of HN occupied by some startup's new idea or a researcher's new findings. I also don't mean to imply that "big corporate" == bad. Certain products -- self driving cars, Space X automated landings, etc -- are absolutely worthy of our attention and discussion. I just hope that people would think twice before upvoting a story merely because it's from Google.
- vgt 12y agoI suggest you read, for example, this article from TC: http://techcrunch.com/2015/03/11/google-launches-cloud-storage-nearline-a-low-cost-storage-service-for-cold-data/ http://techcrunch.com/2015/03/11/google-launches-cloud-stora... This information should challenge your argument that Nearline is not groundbreaking, at least in the "cloud archival storage" space.
- crb 12y ago1c/GB/mo for data stored, ~3 seconds response times (cf. tape/Glacier at multiple hours), 11 9s durability, same API as Google Cloud Storage online. (edit: add /month)
- thrownaway2424 12y agoThat pricing is per month. (I had to look it up).
- keysheath 12y agoThere's no mention of durability numbers anywhere in the docs. S3 and Glacier mention 11 9s durability in their FAQ though.
- BrandonY 12y agoGoogle Cloud Storage Nearline offers the same durability as the Standard-priced storage offering. See the first paragraph of https://cloud.google.com/storage/docs/nearline-storage https://cloud.google.com/storage/docs/nearline-storage
- deleted 12y ago[deleted]
- nda 12y agofrom the documentation: "You should expect 4 MB/s of throughput per TB of data stored as Nearline Storage. This throughput scales linearly with increased storage consumption."
- blfr 12y agoThere it is. Still, someone like 'cperciva could pool a lot of their customers' data together and offer high speeds even for small amounts of data.
- gearhart 12y agohttp://xkcd.com/1494/ http://xkcd.com/1494/ The unusual thing about this market is that by constantly improving the price point of their product but keeping their profit margins the same, AWS has forced every competitor in it to actually compete at the limits of their capability - so any scheme like this that got any serious traction would presumably be self-undermining. The economists will have considered all of the unused provided capacity in the model before they priced it. Sorry to be miserable ;)
- vidarh 12y agoI don't know why you think so. AWS is not very price competitive. I make good money consulting on setup of more cost-effective alternatives to things like S3. The large cloud providers sell on brand recognition, convenience and trust, not cost. They may compete with each other on cost to try to cannibalise each others markets, but they're not even close to pushing the envelope on cost efficiency in terms of the prices they offer.
- gearhart 12y agoInteresting - I'm speaking from general market sentiment (e.g. http://www.techrepublic.com/article/aws-is-playing-chicken-with-pricing-and-enterprise-incumbents-will-lose/ http://www.techrepublic.com/article/aws-is-playing-chicken-w...) but I'd never heard the "Mountain of Margin" argument (http://www.forbes.com/sites/mikekavis/2014/08/08/how-niche-cloud-providers-compete-with-aws-google-and-microsoft/ http://www.forbes.com/sites/mikekavis/2014/08/08/how-niche-c...) before you called it in to question - thanks for bringing it up. I now don't know what to believe - time to go and do my homework on this one.
- nda 12y agoThe same egress and data transfer charges as for Google Cloud Storage Standard / DRA. Can transfer the data from Standard to Nearline bucket for free until Jun, 11 - https://cloud.google.com/storage/pricing#network-regions https://cloud.google.com/storage/pricing#network-regions
- teraflop 12y agoThat promotion is talking about transferring data between geographical regions, not storage classes.
- SEJeff 12y agoSo is this a blatant dig at Amazon's Glacier? "Fast performance... unlike competitors" As they generally do tit for tat on price wars, it will be interesting to see what Amazon responds with.
- dragonwriter 12y ago> So is this a blatant dig at Amazon's Glacier? Glacier is the other big offering from a comparable cloud vendor in this space, so, yeah. The performance claim is directed largely at Glacier, as is the consistent access one.
- pjc50 12y agoSo how is this implemented? Spun-down disk drives?
- bbrazil 12y agoMy guess would be disk drives that are otherwise used for I/O intensive purposes, and where there's tons of unused/wasted space. 4MB/s is only around 4% of the bandwidth of a modern 1TB hard disk.
- nda 12y agoYou can store 1 Pb and get 4000 MB/s
- mseebach 12y agoPresumably distributed across 1000 times more disks.
- keysheath 12y agoAnd at 3MB/s per TB it will always take over 3 days to retrieve all of your data. The 2-5 second retrieval latency is irrelevant when you're going to be waiting for 3 days...
- jlebar 12y agoFWIW it's nominally 4MB/s/TB: > You should expect 4 MB/s of throughput per TB of data stored as Nearline Storage. But still, you are correct in that it will take about 1TB / 4MB/s = 2.9d to retrieve all your data. If you need the ability to do a restore faster than this, then you need to pay more for storage, is all. For many of us, waiting 3d to recover from a catastrophic failure isn't a big deal.
- IanCal 12y agoIs 3 days particularly slow for restoring a petabyte of data from an offsite backup?
- 12y ago
- omh 12y agoThe detailed pricing is at https://cloud.google.com/storage/pricing#nearline-pricing https://cloud.google.com/storage/pricing#nearline-pricing In short, if I'm understanding correctly: $0.01 per GB per month storage $0.01 per GB retrieval Normal egress fees on top, so additional ~$0.10 per GB if you want to retrieve outside of Google Cloud. Early deletion fee. Effectively just a minimum storage charge of 1 month. It seems this is cheaper than Glacier and quite a bit simpler. The speed restrictions are interesting though.
- fuzzythinker 12y agoUnless Glacier just changed their pricing, how is it cheaper and simpler? Glacier is also $.01/GB storage, but only $.09/GB retrieval, which is cheaper than google's $.12/GB. Glacier also comes with 1st GB retrieval free/mo.
- omh 12y agoGlacier have additional fees depending on how fast you retrieve the data. Honestly I'm not sure what they are because I've never had to use it, but my impression is that these fees could be quite high.
- keysheath 12y agoAt least with Glacier I get the option of retrieving faster. Nearline is going to take more than 3 DAYS to retrieve my data if I only have 1TB stored (3MB/s).
- panarky 12y agoLet's assume that with Glacier you can retrieve your 1000 GB in 4 hours instead of 3 days. You will be surprised by a retrieval fee of 1000 / 4 * 720 * 0.01 = $1,800. If you want to get your 1000 GB for free, you'd have to request no more than 69 MB per hour. It will take you about 20 months to get your data! With Glacier, limit retrievals to less than 5% per month / 30 days / 24 hours = 0.007% of total data stored per hour to avoid these ugly surprises. Source: http://aws.amazon.com/glacier/faqs/#How_much_data_can_I_retrieve_for_free http://aws.amazon.com/glacier/faqs/#How_much_data_can_I_retr...
- tiernano 12y agoslightly off topic here, but when i tried to setup an account, which should get me $300 free, it auto selects my country (Ireland) and then tells me: Tax status Business This service can only be used for business or commercial reasons. You are responsible for assessing and reporting VAT. and then makes me enter a VAT number... well, I will stick with AWS and Azure, since neither of those "REQUIRE" a vat number or business status. PS: i know a business name costs about EUR20 to setup, but the VAT requirement is the pain in the left one...
- deepsun 12y agoWhere did you read about $300 for free?
- sp332 12y agoIsn't the price a little high? $10/TB/mo is the same as Dropbox, and this has a lot fewer features than Dropbox.
- nda 12y agoIt's a bit apples to oranges comparison. Dropbox is a consumer product, like Google Drive. Nearline is targeted to app developers and enterprises
- sp332 12y agoExactly, so why would the lower-level service aimed at big slow transfers be more expensive than a full-featured product?
- nda 12y agoSLAs on availability/durability & guaranteed throughput..
- function_seven 12y agoThe full-featured product is $10 regardless of how much storage you're using. If my requirements are for 1GB, then my price is $0.01 a month.
- manacit 12y agoThere's a few reasons: * Dropbox doesn't guarantee 11 9s like this service does - when I'm backing critical data up, I want to make sure it's _there_. * Dropbox likely wouldn't take kindly to me storing 10PB, whereas that is what this service is designed for. * I've got SLAs and guaranteed speeds with this, Dropbox isn't designed for me to suddenly download 10PB very quickly.
- thrownaway2424 12y agoWhere are you seeing these durability claims? I can't find them. And what would 12 9s durability even mean? You lose one byte out of every TB?
- cornellwright 12y agoGreat to finally see a cloud service competing against Glacier. Hopefully this will lead to some sort of response from AWS.
- gaul 12y agoI added GCS nearline to my object storage comparison: http://gaul.org/object-store-comparison/ http://gaul.org/object-store-comparison/
- ddorian43 12y agoyou are missing runabove edit: also missing constant.com
- pi-rat 12y agoDidn't know they offered object storage, thanks. $0,01/GB/month - for online content - replicated 3 times.. Wow Any experience with them? how is their stability and speed?
- ddorian43 12y agoNo I haven't used them. But their parent-company(ohv) has the ~lowest prices you can find.
- gaul 12y agoI have not previously encountered these providers; could you submit a pull request here: https://github.com/andrewgaul/object-store-comparison https://github.com/andrewgaul/object-store-comparison
- dlisboa 12y agoJust a correction: the city name is Sao Paulo with a 'u'. It has the tilde at São but if you don't want to you can omit it. The thing is that Paolo with an 'o' isn't a Portuguese name (it's Italian). In English it's written like in Portuguese. I see it sometimes so I just got in the habit of reminding people, wasn't calling you out, just trying to prevent future mistakes like that.
- gaul 12y agoCorrected; thanks! Note that GitHub hosts this project for any further issues or corrections: https://github.com/andrewgaul/object-store-comparison https://github.com/andrewgaul/object-store-comparison
- moe 12y agoThis is a nail in the coffin of the SaaS backup industry (Backblaze, RescuePlan, etc.). With storage becoming essentially free the last excuse for not using self-hosted, secure backup tools like Arq[1] disappears. [1] http://www.haystacksoftware.com/arq/ http://www.haystacksoftware.com/arq/
- sp332 12y agoBackblaze is $5/mo per computer. If I have 4TB on a computer, it's still just $5/mo.
- pbsk 12y agoYes, but would you spend weeks uploading your 4TB?
- sp332 12y agoWell... yeah. But it doesn't cost me any money to restore either, so it's much cheaper. I don't think they're going to feel threatened by this.
- dragonwriter 12y agoBackblaze can probably afford to be $5/mo per computer because the average computer using it has less than 4TB; OTOH, if there is a more cost effective option for the low end, it won't make sense to use Backblaze on the low end, which will drive up Backblaze's sustainable per computer price.
- atYevP 12y agoYev from Backblaze here -> Fair point! Our costs keep decreasing as well -> https://www.backblaze.com/blog/150-petabytes-of-cloud-storage/ https://www.backblaze.com/blog/150-petabytes-of-cloud-storag... and we're able to grow quickly (https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect... ), so even if avg. user size goes up, decreasing costs make up for it overall.
- knocte 12y agoDoes anyone know of any online-storage offering that allows paying in bitcoin and allows API access? Thanks
- weitzj 12y agoQuote: "This is a Beta release of Nearline Storage. This feature is not covered by any SLA or deprecation policy and may be subject to backward-incompatible changes." So should I believe in Google's good-will? I would be fine trying out some services, which are in Google Beta. But my valuable data? They should have a SLA right from the start to gain the user's trust.
- fixermark 12y agoIt's probably something that could be deployed in the "nice to have" category: use it as a cache for your offline data to provide quick recovery of things you would previously have needed to go all the way to offline to retrieve. But if something goes wrong, you still have the offline data to recover from.
- deleted 12y ago[deleted]
- libria 12y agoIf we place too many restrictions on how companies should offer preview releases, they'll just stop entirely. Are you suggesting that even those comfortable taking on risk to get a sneak peak should be forced to wait for general availability?
- texthompson 12y agoNobody is placing restrictions, but it's understandable if people are skeptical.
- bernadus_edwin 12y agoIs azure has the product like this? Usually cloud war start from amazon, then google, and then azure is the last one
- joelthelion 12y agoI wonder what the underlying hardware is?
- g8oz 12y agoCan you do rsync style differential backups with this?
- crb 12y agogsutil (the Google Cloud Storage command-line tool) has an rsync option: https://cloud.google.com/storage/docs/gsutil/commands/rsync https://cloud.google.com/storage/docs/gsutil/commands/rsync
- bkruse 12y agoI own a big wholesale telco that does tons of data center business and bandwidth. Of course, margins are super thin so we need great pricing. We are no Google, but we can achieve the same pricing, including the cost of bandwidth (that is 9-9s of durability and no spin-up time, which is where the 3-seconds come from) Maybe I should start my own service to harness all this infrastructure with something like swift!
- wyldfire 12y agoThe transport for this service is HTTP? I imagine most competitors use that as well, right? How does encoding factor in here? I have to transcode my data to base64 in order to put it in or take it out, I assume? I "know" I'd only get billed for the data stored as the original octet/binary encoding. But what about the egress fees? Encoded or decoded data is the input for the billing?
- beolab1700 12y agoThere is no need to base64 encode the data on the way in or way out. I cannot think of any APIs that use base64 encoded requests or responses.
- Guvante 12y agoHTTP can transfer arbitrary binary data just fine. There is no need to encode it. I don't know which they use but HTTP traditionally uses a `Content Length` or an empty chunk to represent the end of data.
- zobzu 12y ago"nearline, still online, but we could make a graph that is supposed to make you believe its not"
- deleted 12y ago[deleted]
- AndyJack1979 12y agoOur story with Amazon Glacier (50TB):, we are a company with 50TB of data on multiple Servers and NAS, and some External drives, most of the data we needed to save/Archive for 10 years at least, it’s starting to become very hard to manage locally, so we saw glacier a great solution for us! Well, not exactly We tried to manually using it, like APIs, Scripts, etc… it’s impossible, it was very had So we tried cloudberry to help us upload, it will upload but also It won’t work for huge data, with glacier you won’t be able to search, list, find any file you want to download easily, also its not practical to manage all the backup and millions of files manually, also we got millions of photos we needed a way to find them easily like thumbs So Glacier had so many restrictions like 3-5 hours restore time, 5% restore quota hard to use even with utilizes like cloudberry, cant list, search, it’s not useable as its! But the price was attractive We considered Seagate Evault , but it was expensive and so many hidden fees and complicated for our case Then we tired another solution called Zoolz ( http://www.zoolz.com http://www.zoolz.com ), Zoolz does not use your own AWS Account, but utilize their own account and from what I heard from them they got 5 Petabyte, so they got massive restore quota, around 5% of the 5 Petabyte, also its simple to use, they offer Zero restore cost, it’s like Mozy or Crashplan but for business and on glacier , like they internally create thumbs for your photos and store it on S3, so you have instant preview, also you will be able to search and browse your data easily and instantly, they got Servers and Polices, and got a reasonable price, all we wanted , we got the 50TB for $12,000 / year, its more expensive than using Glacier by itself, using Glacier will cost around $6000, but its not practical for a company to use it as a standalone storage The only disadvantage is that when/if we needed a file we have to wait 4 hours to get it, which is fair, it was faster than when we used to use Tapes :), awe tried to restore 1 TB with 1.2 million files, it took us around 10 hours to complete, which was okay
- Jolielov 12y agoI still rather use a Cloud Service provider like Zoolz as it would be more efficient to benefit from other plans. Amazing idea though!